Framework introduction – Vendors

This page will introduce the various vendors we have in our simulation framework.

• Vendor
• Reinforcement learning agents
• Rule based vendors

Vendor

The marketplace is a stage for the competition between different vendors. The goal of each vendor is to maximize its profit per episode, which can be influenced through the prices they set during each step of the simulation.

Vendors in our simulation framework are always either a reinforcement learning agent, meaning its policy is driven by an RL-algorithm, or a rule based vendor, meaning its policy is defined through a static set of rules. Both of these approaches are explained in the following sections.

Reinforcement learning agents

Reinforcement learning agents are the main focus of our simulation framework, and the reason it was built. The framework features a large number of RL-algorithms to choose from, all of which have their separate strengths and weaknesses. For a full list of the currently available RL-agents in our framework refer to the next section, for how to configure them, please refer to this page.

Reinforcement learning agents can be trained through the training task of our framework. During their training, they continuously receive feedback from the marketplace, in the form of profits, which they then use to modify and (ideally) improve their policy.

After the agent has been completely trained (the end of the training is not defined by an internal metric but by the number of training episodes defined by the user), its resulting models can be further analyzed using one of our many monitoring tasks (also see this page). Trained models can however only be used on the same combination of market type and environment, whilst the actual competitors may be differed.

Available RL-agent classes

This section includes all current (non-abstract) classes for reinforcement learning agents that are available for use in the framework. Please note that vendors are only compatible with a subset of marketplaces. As this list can change over time, we add a timestamp:

This list was last updated on: 24/07/2022.

• QLearningAgent
• DiscreteActorCriticAgent
• ContinuousActorCriticAgent
• ContinuousActorCriticAgentFixedOneStd
• ContinuousActorCriticAgentEstimatingStd
• StableBaselinesA2C
• StableBaselinesDDPG
• StableBaselinesPPO
• StableBaselinesSAC
• StableBaselinesTD3

Rule based vendors

Different from the abovementioned RL-agent, rule based vendors are not trained, but their policy is defined by a set of pre-determined rules. In our framework, a wide variety of rule based vendors is available, each with their own general strategy. The purpose of these vendors is to play against RL-agents during their training and subsequent monitoring, to challenge them to improve their pricing strategies. The framework can also be used to test the performance of the user's own, self-implemented rule based pricing strategies, by using one of the provided monitoring tools.

Available rule based vendor classes

This section includes all current (non-abstract) classes for rule based vendors that are available for use in the framework. Please note that vendors are only compatible with a subset of marketplaces. As this list can change over time, we add a timestamp:

This list was last updated on: 24/07/2022.

• FixedPriceLEAgent
• LinearRatio1LEAgent
• LERandomAgent
• Just2PlayersLEAgent
• FixedPriceCEAgent
• RuleBasedCEAgent
• FixedPriceCERebuyAgent
• RuleBasedCERebuyAgent
• RuleBasedCERebuyAgentCompetitive
• RuleBasedCERebuyAgentStorageMinimizer