What is SCM and SCML?

The Supply Chain Management (SCM) world simulates a supply chain consisting of multiple factories that buy and sell products from one another. The factories are represented by autonomous agents that act as factory managers. Agents are given some target quantity to either buy or sell and they negotiate with other agents to secure the needed supplies or sales. Their goal is to turn a profit, and the agent with the highest profit (averaged over multiple simulations) wins.

The Supply Chain Management League (SCML) is a part of the International Automated Negotiating Agents Competition (ANAC) since 2019. SCML runs in the SCM world.

The league is built using NegMAS which provides tools for building multiagent systems in which negotiations are embedded within world simulations.

There are two different yet related games in SCML each represented by a NegMAS World class:

SCML2024StdWorld A world representing the complete SCML game described here in which the agent is responsible of defining its own utility function, with who to negotiate, negotiation agendas to propose, production schedule, etc. This form of the SCM world is more representative of real-world trading with a combination of long-term planning, negotiation, and production scheduling.
SCML2024OneShotWorld A simpler form of the SCM world in which agents do not need to consider long-term planning or production scheduling described here. This form was introduced in 2021.

You can use the SCM world simulation in your own development and research as you see fit but it is primarily intended for the SCM league (SCML) of the ANAC competition. SCML has three tracks:

Standard track based on the SCML2024StdWorld full game simulation. In the standard track, a single instance of each agent type exists in the world. This precludes any possibility of collusion between agent instances of the same type and focuses on business operation of a single factory.
OneShot track based on the simpler SCML2024OneShotWorld game simulation. In the OneShot track, a single instance of each agent type exists in the world precluding any possibility of collusion. The focus here is on negotiation with less focus on long-term planning as discussed earlier. This track was introduced for the first time in 2021.
Collusion track based on the SCML2023World full game simulation. In the collusion track, multiple instances of each agent type exist in the world. This opens the possibility for these agents instances to collude in order to corner the market and get higher profits that what each of them can get on its own. In this track, collusion is not considered a form of cheating but is encouraged. This track is discontinued in 2024.

The SCML competition has two versions:

Online Competition Participants to this competition can submit their agents as long as the competition is running (March 15th 2021 to July 1st 2021). The system runs tournaments between the latest version of all submitted agents for each track periodically and maintains a leader-board showing the top agents updated at least weekly. Participants in this competition will get the chance to observe how their agents behave against agents submitted by other competitors and receive feedback about any exceptions or error caused by their agents.
Official Competition The final version of each agent submitted to the online competition (and registered before July 1st, 2021) is entered into the official SCM league which will run as part of the ANAC 2024 competition in conjunction with AAMAS 2024. The winners will be announced during the as part of the official program of IJCAI 2021.

A (not very) brief introduction to NegMAS

The SCM world is built using NegMAS. This section provides a very short introduction of it to help you understand the terms used in this document. Please refer to NegMAS documetnation for full description and details as well as tutorials and other goodies.

This image shows the main components of a simulation in a NegMAS world:

The simulation is run using a World object which defines what happens in every simulation step, provides a BulletinBoard object containing all public information about the game, calls various callbacks defined in the Agent object representing each agent in the environment, takes care of running negotiations and keeps track of agreement signing and the resulting Contracts. The World object also controls logging, event management, serialization, visualization, etc. Refer to the World documentation for more details (you need to do that only if you are implementing new world simulations).

The designer of the game implements a World class by overriding few abstract methods in the base World class. This is how the simulations for scml were created(e.g. SCML2019World, SCML2024World, SCML2024OneShotWorld). As a participant in SCML, you never need to interact directly with the World object.

The logic of an agent is NegMAS is implemented in an Agent object. The designer of the simulation, should provide a base class for its specific world inherited from NegMAS’s Agent class. For SCML, this base class is called OneShotAgent for the OneShot game (to go with the SCML2024OneShotWorld), and StdAgent for the full game (to go with the SCML2024StdWorld). Refer to the Agent documentation for more details about general NegMAS agents. Full information about the OneShotAgent and SCMLStdAgent classes can be found here and here respectively.

<strong>info:</strong> As a participant in SCML, you

only need to develop an Agent class inherited from the appropriate base class depending on the track.

So now we have the World and the Agent objects, and we already said that the agent does not directly interact with the world. How does these two agents interact then?

When the World wants to interact with the Agent, it calls some method in it. For example, to instruct the agent to initialize itself, the world calls the init() method defined by the Agent. To inform the agent that a negotiation it is involved in is concluded with success, the World calls the method on_negotiation_success() defined by the agent.
When the Agent wants to interact with the World, it accesses an interface object called an AgentWorldInterface or AWI for short which provides all the services available to the Agent. For example, to request a negotiation with another agent, the Agent object needs to call request_negotiation() defined in the AWI.

The world designer defines an Agent-World-Interface (AWI) for its world that inherits NegMAS’s AgentWorldInterface class and provides any special services for agents interacting in this world. You can find all the services available to your agent through the AgentWorldInterface here. These methods and properties are still available for your agent in SCML. Nevertheless, in many cases, more convenient ways to access some of the information (e.g. the bulletin board) is provided in the specific AWIs implemented in the SCML package to be described now.

For SCML, there are – as you may already expect – two AWIs. The first is called OneStdAWI and is defined in scml.std to go with SCML2024StdWorld and StdAgent and the second is defined in scml.oneshot to go with SCML2024OneShotWorld and OneShotAgent.

You can find all the services provided by these AWIs here for the OneShot game (OneShot track) and here for the full game (Standard and Collusion tracks).

<strong>Note: </strong>
Earlier versions of the standard AWI are also defined in <i>scml.scml2019</i> and <i>scml2020</i>.
These are <b>discontinued</b> forms of the game used in ANAC 2019 to ANAC 2023 competitions.

Because the standard and oneshot versions of the game are compatible (starting 2024), the two AWIs (OneShotAWI and StdAWI) are actually exactly the same.

Negotiation Management

Now that we know how worlds and agents work and interact, we can look at how negotiation is managed in NegMAS.

A negotiation is controlled by a Mechanism object which implements the negotiation protocol (e.g. the alternating offers protocol). NegMAS provides several mediated and unmediated negotiation protocols (as well as auction mechanisms). The specific Mechanism that is used in SCML is the SAOMechanism which implements the bargaining protocol.

Negotiation strategies are implemented in a Negotiator object which usually inherits some base negotiator-class corresponding to the mechanism(s) it supports. The specific base class of all negotiators in SCML is SAONegotiator.

The interaction between Mechanism and Negotiator objects mirrors the interaction between World and Agent objects. Mechanism objects call methods in Negotiator objects directly but Negotiator objects can only access services provided by the Mechanism object through a NegotiatorMechanismInterface (AMI). Note that it is an AMI not a NMI (for historical reasons). You can find more details about the general NegMAS AMI here.

Each specific Mechanism defines a corresponding specific NegotiatorMechanismInterface class (in the same way that World classes define their own AWI). The SAOMechanism used in SCML defines SAONMI for this purpose (Details are here).

To negotiate effectively, negotiators can employ a UtilityFunction to represent their preferences over different possible Outcomes of the negotiation (where an outcome is a full assignment of values to all negotiated Issues). NegMAS provides an extensive set of utility function, and issue types. Please refer to NegMAS overview and tutorials for more details. NegMAS also provides some basic SAONegotiators for the SAOMechanism (Check the class diagram here). Moreover, you can access almost all Genius using NegMAS’s GeniusNegotiator including all finalists and winners of all past ANAC competitions.

Now we understand how agents interact with worlds through AWIs and negotiators interact with mechanisms through AMIs. We know that the general simulation is controlled by the world while each negotiation is controlled by a mechanism within that world. We need now to connect these two triplets of objects

As the figure above shows: Negotiator objects can be created and controlled by Agent objects for the purpose of negotiating with other Agent objects. The standard flow of operations is something like this:

Agent A uses its AWI to request_negotiation() with Agent B passing a Negotiator to be used in this negotiation. Usually Agent A will also create a UtilityFunction and attach it to the Negotiator it just created (by setting its ufun attribute).
The World calls Agent B’s respond_to_negotiation_request() asking it to provide its own Negotiator to negotiate with Agent A’s Negotiator. It can also just reject the negotiation request by returning no negotiators.
The World will then create a Mechanism and ask both Negotiators to join it. If all goes well, the negotiation starts (at a time defined by the simulation rules) and runs until either an agreement or disagreement is reached.
The World class will then inform Agents A and B about the results of the negotiation using their on_negotiation_success and on_negotiation_failure callbacks.
Successful negotiations lead to Agreements but are still not binding in general until signed by all agents involved (A and B in this case). Agent’s ’sign_all_contracts is used for this.
Signed agreements become Contracts and are executed (as specified in the simulation rules) by the World.

The SCML game (both Standard and OneShot) simplifies things by automatically requesting all negotiations, accepting all requests and signing all contracts.

When negotiations are independent, these are all the objects needed. Nevertheless, in many cases, including the SCML case, negotiations are inter-dependent. This means that what is good in one negotiation depends on other concurrently running negotiations (or on expectations of future negotiations). NegMAS provides two ways to support this case shown in the following figure:

Let Negotiators use UtilityFunctions that depend on some common state. That is what is happening in the left two negotiations. Because you define your own utility functions in the full SCML game (standard and collusion tracks), this is one possible way to orchestrate the behavior of your negotiators.
Have multiple Negotiators be controlled by a single Controller object with its own utility function that depends on what is happening on all the negotiations controlled. This is what is happening in the two negotiations on the right. This is also possible in the full SCML game (standard and collusion tracks) and is the only method available in the OneShot game (OneShot track).

The Negotiators connected to a controller lost their autonomy and just pass control to their owning Controller.

<strong>Just if you are wondering: </strong>SCML agents are actually NegMAS controllers as well.

This concludes our introduction to NegMAS and different objects you need to know about to develop your agent.

In summary, you will need to develop an StdAgent or OneShotAgent depending on the track. This agent uses an StdAWI or OneShotAWI to access sense and act in the world. All negotiations are controlled by SAONegotiators that may optionally be grouped and controlled centrally by Controller objects. The preferences of these negotiators and controllers are defined using UtilityFunction objects.

A brief introduction to SCM

Here we only provide a 10,000-feet birds-eye view of the two games used in SCML 2024’s two tracks. Each section provides pointers to more information.

The Oneshot game (OneShot)

An overview of the SCML-OneShot game is available here and a full description for the details-savy person is available here.

The game runs in a world in which factories can run manufacturing processes to convert products into other products. There are exactly 3 products and 2 processes as shown in the figure. Each agent controls exactly one factory.

The agents that receive the raw material and generate the intermediate product are called \(𝐿_0\) agents (level zero) and the agents. receiving the intermediate product and generate the final product are called \(𝐿_1\) factories (level 1). The product type consumed by the factory controlled by an agent is called its input product and the product it produces is called its output product.

Every day (simulation step), each \(𝐿_0\) factory receives one exogenous contract specifying a quantity and a unit price (supplies) for this day and each \(𝐿_1\) factory receives one exogenous contract specifying a quantity and a unit price of the final product (sales) for the same day. Production and transporation are assumed to take no time.

\(𝐿_0\) and \(𝐿_1\) agents need to negotiate together to secure intermediate product contracts in order to use their supplies (for \(𝐿_0\) agents) or satisfy their sale obligations (for \(𝐿_1\) agents). All products that remain unsold at the end of the day perish (i.e. has no value). Moreover, agents pay a disposal cost for any input products they buy and never sell and pay a shortfall penalty for any sales they cannot satisfy.

Your goal as an agent designer is to maximize your overall profit over the simulation time.

The Full game (Standard Track)

An overview of the SCML-Standard game is available here and a full description for the details-savy person is available here.

The game runs in a world similar to the one-shot game world but with few differences (that make a lot of difference):

Products are not perishable in the Standard game allowing factories to accumulate stock of their input product. This means that profits calculated every day are lower limits as they do not take into account the value of accumulated stock which can be manufactured and sold in the future. This implies that there is no disposal of products at the end of each day, instead there is a small cost associated with carrying stock (storage cost).
Agents can negotiate future contracts instead of only being allowed to negotiate about deliveries on the same day.
Negotiated quantities have a larger range to allow for proactive stockpiling of the input product.
Negotiated prices have a larger range which necessitates reasoning about prices as well as quantity matching.
Production graphs can be deeper which means that some agents will negotiate with both their suppliers and consumers at the same time.
Finally, the fact that agents can have contracts with delivery dates in the future, complicates bankruptcy processing. When an agent goes bankrupt, its future contracts are all nullified/cancelled.

Because the oneshot and standard game have <i>exactly</i> the same API, agents developed for either can run in both in principle. Nevertheless, the differences between the two games is large enough that agents developed for one of them is not <i>expected</i> to run well in the other. For example, agents developed for the oneshot game will not know how to use the opportunity of future contracts, while agents developed for the standard game will assume that stock is carried forward.

Run a session of the SCML world (2024)

The SCML world (Supply Chain Management League) runs on top of NegMAS.

In this tutorial, you will test a run of this world.

Firstly, let’s import everything from the SCML app

Note that the 2024 standard league is based on the oneshot game with appropriate generalizatuions as described in the detailed game description document.

from scml.std import *
from scml.oneshot import *

There are several ways to create an SCML world. One of the simplest is to use the generate method of the SCML2024StdWorld class. This will create a dict that can be passed to the SCML2024StdWorld constructor as keyword arguments.

Here we explicitly set construct_graphs to True which slows the simulation yet allows us to see graph representations of what is happening in the world.

agent_types = [
    SyncRandomStdAgent,
    RandDistOneShotAgent,
    GreedyOneShotAgent,
    RandomStdAgent,
]

world = SCML2024StdWorld(
    **SCML2024StdWorld.generate(agent_types=agent_types, n_steps=50),
    construct_graphs=True,
)

Let’s draw a graph to see what is in this world

_, _ = world.draw()

If you want to just test your installation (and do not care whether you get an accurate indicator of agent performance), you can set the number of steps to a small value (e.g. n_steps=10).

Now you can run this world simulation by just calling run.

world.run_with_progress()  # may take few minutes

Output()

Let’s see what happened in this run. Firstly, how many negotiations were conducted over time

world.plot_stats("n_negotiations")
plt.show()

There are exacty 7 * 8 = 56 negotiations the first step as expected. Smaller number of negotiations indicate that some agents went bankrupt

We can check which ones:

world.plot_stats("bankrupt", ylegend=1.25)

Several other market statistics are available:

world.plot_stats()

You can check all available stats:

print(world.stat_names)

[
    'activity_level',
    'balance',
    'bankrupt',
    'breach_level',
    'disposal_cost',
    'inventory_input',
    'inventory_output',
    'inventory_penalized',
    'n_breaches',
    'n_contracts_cancelled',
    'n_contracts_concluded',
    'n_contracts_dropped',
    'n_contracts_erred',
    'n_contracts_executed',
    'n_contracts_nullified',
    'n_contracts_nullified_now',
    'n_contracts_nullified_price',
    'n_contracts_nullified_quantity',
    'n_contracts_signed',
    'n_negotiations',
    'productivity',
    'score',
    'shortfall_penalty',
    'shortfall_quantity',
    'sold_quantity_0',
    'sold_quantity_1',
    'sold_quantity_2',
    'step_time',
    'storage_cost',
    'total_time',
    'trading_price_0',
    'trading_price_1',
    'trading_price_2',
    'unit_price_0',
    'unit_price_1',
    'unit_price_2'
]

Note that stas that end with a number are about specific products. Some stats (e.g. storage_cost) is actually defined per agent (and per agent type). We will see that later.

Let’s start by seeing how long did each step take (note that stats access the stats as a dict[str, List] but stats_df access the same data as a pandas dataframe.

world.stats_df["step_time"].plot(kind="bar")
plt.show()

You can ge the same information using plot_stats(). You may prefer using stats_df if you are going to do pandas manipulations over the data.

world.plot_stats("step_time")

As we indicated earlier, there are statistics specific for each agent that all have “_{agent_name}“. Lets check what is available for the winner agent:

winner = world.winners[0]
print(list(_ for _ in world.stats.keys() if winner.name in _))

[
    'score_04SRS@0',
    'balance_04SRS@0',
    'bankrupt_04SRS@0',
    'productivity_04SRS@0',
    'shortfall_quantity_04SRS@0',
    'shortfall_penalty_04SRS@0',
    'storage_cost_04SRS@0',
    'disposal_cost_04SRS@0',
    'inventory_penalized_04SRS@0',
    'inventory_input_04SRS@0',
    'inventory_output_04SRS@0'
]

The convention is that agent names has the form {ind}{Type}{process} where ind is a unique index, Type is a shortened version of the agent’s type name, and process is the process the agnet can run. Note that the agent’s input product has the same number as its process and its output product has the next number (i.e. an agent that runs process 1, has input product 1 and output product 2).

We can see that 8 pieces of information are available (for each time-step of the simulation):

bankrupt If true, the agent is bankrupt.
balance The money the agent has in its wallet (account).
inventory_input The number of units of the agent’s input product available in its inventory (by the end of the simulation step).
inventory_output The number of units of the agent’s output product available in its inventory (by the end of the simulation step). This will always be zero in SCML.
inventory_penalized The amount of inventory penalized by paying storage cost or disposal cost.
storage_cost The storage cost paid on the given step. For SCML-OneShot, this will always be zero.
disposal_cost The amout paid to dispose of products at the end of the step (if products are perisahble). This is always zero for SCML-Std
shortfall_quantity The quantity we promised to sell but failt to produce.
shortfall_penalty The penalty we pay for failing to produce the quantity we promised to sell.
productivity The fraction of the agent’s production lines that were active at a given time-step.
score The score of the agent according to the evaluation rule of ANAC SCML 2020

Let’s see how did our agent do

# show the first and last value of each of the agent statistics
print({k: (v[0], v[-1]) for k, v in world.stats.items() if winner.name in k})

{
    'score_04SRS@0': (1.000953440330526, 0.9440109062995361),
    'balance_04SRS@0': (6299.0, 5940.660633342981),
    'bankrupt_04SRS@0': (False, False),
    'productivity_04SRS@0': (0.2, 0.0),
    'shortfall_quantity_04SRS@0': (0, 0),
    'shortfall_penalty_04SRS@0': (0.0, 0.0),
    'storage_cost_04SRS@0': (0.0, 25.646711342564714),
    'disposal_cost_04SRS@0': (0.0, 0.0),
    'inventory_penalized_04SRS@0': (0, 115),
    'inventory_input_04SRS@0': (0, 115),
    'inventory_output_04SRS@0': (0, 0)
}

stats = pd.DataFrame(data=world.stats)
fig, axs = plt.subplots(2, 3)
for ax, key in zip(
    axs.flatten().tolist(),
    ["score", "balance", "productivity", "shortfall_penalty", "storage_cost"],
):
    ax.plot(stats[f"{key}_{winner}"])
    ax.set(ylabel=key)
fig.show()

We can for example check the activity level of this world (defined as the total amount of money transferred due to trade)

world.plot_stats("activity_level")
plt.show()

It is clear that activity goes down over time. This may just be due to the smaller number of negotiations resulting from having bankrupt agents but it can also be due to some end-game effect because agents know the number of simulated days and can change their behavior near the end (e.g. accomulating inventory in the beginning and getting rid of it near the end).

We can see a picture of contracting activity in this world as follows:

world.plot_stats(("n_contracts_signed", "n_contracts_executed"))
plt.show()

Notice that there can be multiple winners

winner_profits = [100 * world.scores()[_.id] - 100 for _ in world.winners]
winner_types = [_.short_type_name for _ in world.winners]
print(f"{world.winners} of type {winner_types} won at {winner_profits}% profit")

[04SRS@0] of type ['SyncRandomStdAgent'] won at [-5.5989093700463854]% profit

Let’s check how did the first winner’s inventory changes over time:

# find the keys in stats for the input and output inventory
in_key = [_ for _ in world.stats.keys() if _.startswith(f"inventory_input_{winner}")][0]
out_key = [_ for _ in world.stats.keys() if _.startswith(f"inventory_output_{winner}")][
    0
]

# find input and output product indices
input_product, output_product = (
    winner.awi.my_input_product,
    winner.awi.my_output_product,
)
# draw
fig, (quantity, value) = plt.subplots(1, 2)
quantity.plot(world.stats[in_key], label="Input Product")
quantity.plot(world.stats[out_key], label="Output Product")
quantity.set(xlabel="Simulation Step", ylabel="Winner's Total Storage (item)")
quantity.legend()
value.plot(
    np.array(world.stats[in_key])
    * np.array(world.stats[f"trading_price_{input_product}"]),
    label="Input Product",
)
value.plot(
    np.array(world.stats[out_key])
    * np.array(world.stats[f"trading_price_{output_product}"]),
    label="Output Product",
)
value.set(xlabel="Simulation Step", ylabel="Winner's Inventory Value ($)")
value.legend()
fig.show()

We can actually check what happens to ALL competitors:

_, axs = plt.subplots(1, 2)
plt.sca(axs[0])
world.plot_stats("balance", ylegend=-0.05)
plt.sca(axs[1])
world.plot_stats("score", legend=False)
plt.show()

fig, (profit, score) = plt.subplots(1, 2)
snames = sorted(world.non_system_agent_names)
for name in snames:
    profit.plot((np.asarray(world.stats[f"balance_{name}"])), label=name)
    score.plot(
        np.asarray(world.stats[f"score_{name}"]) * (world.stats[f"balance_{name}"][0]),
        label=name,
    )
profit.set(xlabel="Simulation Step", ylabel="Player Balance ($)")
profit.legend(loc="lower left")
score.set(xlabel="Simulation Step", ylabel="Player Score Unnormalized ($)")
fig.show()

or just look at the end of the game

fig, (score, profit) = plt.subplots(1, 2)
final_scores = [
    100 * world.stats[f"score_{_}"][-1] for _ in world.non_system_agent_names
]
final_profits = [
    100 * world.stats[f"balance_{_}"][-1] / world.stats[f"balance_{_}"][0] - 100
    for _ in world.non_system_agent_names
]
plt.setp(score.xaxis.get_majorticklabels(), rotation=45)
plt.setp(profit.xaxis.get_majorticklabels(), rotation=45)
score.bar(world.non_system_agent_names, final_scores)
profit.bar(world.non_system_agent_names, final_profits)
score.set(ylabel="Final Score (%)")
profit.set(ylabel="Final Profit (%)")

fig.show()

fig, (score, profit) = plt.subplots(1, 2)
final_scores = [
    world.stats[f"score_{_}"][-1] * (world.stats[f"balance_{_}"][0])
    for _ in world.non_system_agent_names
]
final_profits = [world.stats[f"balance_{_}"][-1] for _ in world.non_system_agent_names]
plt.setp(score.xaxis.get_majorticklabels(), rotation=45)
plt.setp(profit.xaxis.get_majorticklabels(), rotation=45)
score.bar(world.non_system_agent_names, final_scores)
profit.bar(world.non_system_agent_names, final_profits)
score.set(ylabel="Final Unnormalized Score ($)")
profit.set(ylabel="Final Balance  ($)")

fig.show()

You can inspect what happened in the simulation by plotting different output statistics. For example, we can see how did the trading price of different products change over the simulation time.

fig, axs = plt.subplots(2, 2)
for ax, key in zip(
    axs.flatten().tolist(), ["trading_price", "sold_quantity", "unit_price"]
):
    for p in range(world.n_products):
        ax.plot(world.stats[f"{key}_{p}"], marker="x", label=f"Product {p}")
        ax.set_ylabel(key.replace("_", " ").title())
        ax.legend().set_visible(False)
axs[-1, 0].legend(bbox_to_anchor=(1, -0.5), ncol=3)
fig.show()

You can dig futher to understand what happened during this siumulation. For example, let’s see some of the contracts that were signed:

# create a view with only signed contracts
contracts = world.contracts_df
signed = contracts.loc[contracts.signed_at >= 0, :]

fields = [
    "seller_name",
    "buyer_name",
    "delivery_time",
    "quantity",
    "unit_price",
    "signed_at",
    "executed",
    "breached",
    "nullified",
    "erred",
]
signed[fields].sort_values(["quantity", "unit_price"], ascending=False).head(10)

	seller_name	buyer_name	delivery_time	quantity	unit_price	signed_at	executed	breached	nullified	erred
1	02Ra@0	11SRS@1	2	30	18	0	True	False	False	False
58	02Ra@0	08SRS@1	5	30	16	1	True	False	False	False
148	05Ra@0	14SRS@1	4	30	16	3	True	False	False	False
49	05Ra@0	12Ra@1	10	30	15	1	True	False	False	False
97	02Ra@0	08SRS@1	7	30	15	2	True	False	False	False
99	02Ra@0	11SRS@1	6	30	15	2	True	False	False	False
106	05Ra@0	07Ra@1	12	30	15	2	True	False	False	False
303	02Ra@0	11SRS@1	11	30	15	7	True	False	False	False
160	05Ra@0	11SRS@1	12	30	14	3	True	False	False	False
423	02Ra@0	13Ra@1	14	30	14	10	True	False	False	False

Let’s check some of the contracts that were fully executed

signed.loc[signed.executed, fields].sort_values(
    ["quantity", "unit_price"], ascending=False
).head(10)

	seller_name	buyer_name	delivery_time	quantity	unit_price	signed_at	executed	breached	nullified	erred
1	02Ra@0	11SRS@1	2	30	18	0	True	False	False	False
58	02Ra@0	08SRS@1	5	30	16	1	True	False	False	False
148	05Ra@0	14SRS@1	4	30	16	3	True	False	False	False
49	05Ra@0	12Ra@1	10	30	15	1	True	False	False	False
97	02Ra@0	08SRS@1	7	30	15	2	True	False	False	False
99	02Ra@0	11SRS@1	6	30	15	2	True	False	False	False
106	05Ra@0	07Ra@1	12	30	15	2	True	False	False	False
303	02Ra@0	11SRS@1	11	30	15	7	True	False	False	False
160	05Ra@0	11SRS@1	12	30	14	3	True	False	False	False
423	02Ra@0	13Ra@1	14	30	14	10	True	False	False	False

We can now see how does the singning day affect delivery day, product and quantity

fig, ax = plt.subplots(1, 3)
for i, x in enumerate(["delivery_time", "quantity", "product_index"]):
    ax[i].scatter(signed.signed_at, signed[x])
    ax[i].set(ylabel=x.replace("_", " ").title(), xlabel="Signing Day")
fig.show()

fig, ax = plt.subplots(1, 3)
for i, x in enumerate(["delivery_time", "unit_price", "product_index"]):
    ax[i].scatter(signed.quantity, signed[x])
    ax[i].set(ylabel=x.replace("_", " ").title(), xlabel="Quantity")
fig.show()

Did any agents go bankrupt and when?

bankruptcy = {
    a: np.nonzero(stats[f"bankrupt_{a}"].values)[0]
    for a in world.non_system_agent_names
}
print({k: "No" if len(v) < 1 else f"at: {v[0]}" for k, v in bankruptcy.items()})

{
    '00Ra@0': 'No',
    '01Gr@0': 'No',
    '02Ra@0': 'at: 20',
    '03SRS@0': 'No',
    '04SRS@0': 'No',
    '05Ra@0': 'at: 12',
    '06Ra@0': 'No',
    '07Ra@1': 'at: 23',
    '08SRS@1': 'No',
    '09Ra@1': 'No',
    '10Gr@1': 'at: 19',
    '11SRS@1': 'No',
    '12Ra@1': 'at: 24',
    '13Ra@1': 'at: 26',
    '14SRS@1': 'No'
}

You can see what happened during this simulation by drawing graphs at different steps. The meaning of different edge colors can be drawn as follows:

from negmas import show_edge_colors

show_edge_colors()

You can see what happened in this world in a series of graphs using the draw method

world.draw(steps=(0, world.n_steps), together=False, ncols=2, figsize=(20, 20))
plt.show()

You can also run a simple animation to see what happens at every step (you need to download the jupyter notebook and execute it to see the animation) :

world.save_gif("run.gif")

[]

then show the animation

from IPython.display import HTML

HTML('<img src="run.gif">')

Running a tournament

Now that you can run simple world simulations, let’s try to run a complete tournament and see its results. Let’s start by running a standard tournament (in which each agent is represented by a single factory). Running a one-shot tournament (new in 2021) similarily just requires a change from anac2024_std to anac2024_oneshot

Note that in the real competition we use thousands of configurations and longer simulation steps (e.g. 50 \(\le\) n_steps \(\le\) 500).

from scml.utils import anac2024_std

pd.options.display.float_format = "{:,.2f}".format


def shorten_names(results):
    # just make agent types more readable
    results.score_stats.agent_type = results.score_stats.agent_type.str.split(".").str[
        -1
    ]
    results.kstest.a = results.kstest.a.str.split(".").str[-1]
    results.kstest.b = results.kstest.b.str.split(".").str[-1]
    results.total_scores.agent_type = results.total_scores.agent_type.str.split(
        "."
    ).str[-1]
    results.scores.agent_type = results.scores.agent_type.str.split(".").str[-1]
    results.winners = [_.split(".")[-1] for _ in results.winners]
    return results

tournament_types = [GreedyStdAgent, SyncRandomStdAgent, RandDistOneShotAgent]
# may take a long time
results = anac2024_std(
    competitors=tournament_types,
    n_configs=5,  # number of different configurations to generate
    n_runs_per_world=1,  # number of times to repeat every simulation (with agent assignment)
    n_steps=10,  # number of days (simulation steps) per simulation
    print_exceptions=True,
)

results = shorten_names(results)

Who was the winner?

results.winners

['RandDistOneShotAgent']

How many simulations were actually run?

len(results.scores.run_id.unique())

The total number of simulations \(n_{s}\) will be \(n_t \times n_c \times n_r\) where \(n_t\) is the number of competitor agent types, \(n_c\) is the number of configurations, and \(n_r\) is the number of runs per configuration

We can also see the scores that every agent type got

results.score_stats

	agent_type	count	mean	std	min	25%	median	75%	max
0	RandDistOneShotAgent	15.00	0.99	0.09	0.80	0.96	0.99	1.05	1.14
1	GreedyStdAgent	15.00	0.93	0.25	0.23	0.91	1.00	1.06	1.18
2	SyncRandomStdAgent	15.00	0.88	0.24	0.26	0.85	0.95	0.97	1.26

You can also do statistical significance testing using ttest or kstest (with multi-comparison correction)

results.kstest

	a	b	t	p	n_a	n_b	n_effective
0	GreedyStdAgent	SyncRandomStdAgent	0.53	0.03	15	15	15
1	GreedyStdAgent	RandDistOneShotAgent	0.20	0.94	15	15	15
2	SyncRandomStdAgent	RandDistOneShotAgent	0.53	0.03	15	15	15

see the total score

results.total_scores

	agent_type	score
0	RandDistOneShotAgent	0.99
1	GreedyStdAgent	0.93
2	SyncRandomStdAgent	0.85

or the aggregated statistics of the world. For example, let’s draw the activity level for different simulations.

plt.errorbar(
    range(len(results.agg_stats)),
    results.agg_stats.activity_level_mean,
    np.sqrt(results.agg_stats.activity_level_var),
)
plt.xlabel("Simulation Number")
plt.ylabel("Activity Level")
plt.show()

We can even get the scores of every agent belonging to every agent type at every simulation

results.scores.loc[:, ["agent_name", "agent_type", "score"]].head()

	agent_name	agent_type	score
0	00Gr@0	GreedyStdAgent	1.02
1	01SRS@0	SyncRandomStdAgent	0.53
2	04Ra@1	RandDistOneShotAgent	0.99
3	00Ra@0	RandDistOneShotAgent	1.04
4	01SRS@0	SyncRandomStdAgent	0.91

Let’s see how did the location at the production graph affect the score of each type.

results.scores["level"] = results.scores.agent_name.str.split("@", expand=True).loc[
    :, 1
]
results.scores = results.scores.sort_values("level")
sns.lineplot(
    data=results.scores[["agent_type", "level", "score"]],
    x="level",
    y="score",
    hue="agent_type",
    ci=None,
)
plt.plot([0.0] * len(results.scores["level"].unique()), "b--")
plt.show()

Running a One-Shot tournament

In 2021, a new track was introduced to the SCM league called the SCML-OneShot track which simplified the problem in order to focus research efforts on the core many-to-many concurrent negotiation challeng. You can run a tournament for this track with the latest settings in almost the same way as we did with standard/collusion tournaments using anac2024_oneshot

Note that in the real competition we use thousands of configurations and longer simulation steps (e.g. 50 \(\le\) n_steps \(\le\) 500).

from scml.utils import anac2024_oneshot

tournament_types = [
    RandomOneShotAgent,
    SyncRandomOneShotAgent,
    GreedyOneShotAgent,
    GreedySingleAgreementAgent,
]
# may take a long time
results = anac2024_oneshot(
    competitors=tournament_types,
    n_configs=5,  # number of different configurations to generate
    n_runs_per_world=1,  # number of times to repeat every simulation (with agent assignment)
    n_steps=10,  # number of days (simulation steps) per simulation
    print_exceptions=True,
)
results = shorten_names(results)

Who was the winner?

results.winners

['SyncRandomOneShotAgent']

How many simulations were actually run?

len(results.scores.run_id.unique())

The total number of simulations \(n_{s}\) will be \(n_t \times n_c \times n_r\) where \(n_t\) is the number of competitor agent types, \(n_c\) is the number of configurations, and \(n_r\) is the number of runs per configuration

We can also see the scores that every agent type got

results.score_stats

	agent_type	count	mean	std	min	25%	median	75%	max
0	GreedyOneShotAgent	20.00	0.81	0.18	0.43	0.69	0.88	0.96	1.01
1	GreedySingleAgreementAgent	20.00	0.77	0.16	0.58	0.66	0.74	0.88	1.10
2	RandomOneShotAgent	20.00	0.87	0.14	0.55	0.83	0.90	0.96	1.11
3	SyncRandomOneShotAgent	20.00	1.06	0.06	0.94	1.04	1.06	1.08	1.24

You can also do statistical significance testing using ttest or kstest (with multi-comparison correction)

results.kstest

	a	b	t	p	n_a	n_b	n_effective
0	SyncRandomOneShotAgent	GreedySingleAgreementAgent	0.90	0.00	20	20	20
1	SyncRandomOneShotAgent	RandomOneShotAgent	0.85	0.00	20	20	20
2	SyncRandomOneShotAgent	GreedyOneShotAgent	0.90	0.00	20	20	20
3	GreedySingleAgreementAgent	RandomOneShotAgent	0.45	0.03	20	20	20
4	GreedySingleAgreementAgent	GreedyOneShotAgent	0.35	0.17	20	20	20
5	RandomOneShotAgent	GreedyOneShotAgent	0.15	0.98	20	20	20

see the total score

results.total_scores

	agent_type	score
0	SyncRandomOneShotAgent	1.05
1	RandomOneShotAgent	0.87
2	GreedyOneShotAgent	0.81
3	GreedySingleAgreementAgent	0.77

We can even get the scores of every agent belonging to every agent type at every simulation

results.scores.loc[:, ["agent_name", "agent_type", "score"]].head()

	agent_name	agent_type	score
0	00Sy@0	SyncRandomOneShotAgent	1.07
1	01GSA@0	GreedySingleAgreementAgent	0.65
2	02Ra@0	RandomOneShotAgent	0.91
3	06Gr@1	GreedyOneShotAgent	0.43
4	00Ra@0	RandomOneShotAgent	0.97

Let’s see how did the location at the production graph affect the score of each type.

results.scores["level"] = results.scores.agent_name.str.split("@", expand=True).loc[
    :, 1
]
results.scores = results.scores.sort_values("level")
sns.lineplot(
    data=results.scores[["agent_type", "level", "score"]],
    x="level",
    y="score",
    hue="agent_type",
)
# plt.plot([0.0] * len(results.scores["level"].unique()), "b--")
plt.show()

Now that you can run simulations and complete tournaments, let’s see how are we going to develop a new agent for the SCML2023 league \(\rightarrow\)

Running against winners from previous years

You can compare your agent against any agents previously submitted to SCML (same track). To do that, you need to install the scml-agents package from pip:

pip install scml-agents

You can then get agents using the get_agents() function from this package:

from scml_agents import get_agents

winners = [
    get_agents(y, track="oneshot", winners_only=True, as_class=False)[0]
    for y in (2021, 2022, 2023)
]
print(winners)

[
    'scml_agents.scml2021.oneshot.team_86.agent112.Agent112',
    'scml_agents.scml2022.oneshot.team_134.agent119.PatientAgent',
    'scml_agents.scml2023.oneshot.team_poli_usp.quantity_oriented_agent.QuantityOrientedAgent'
]

We can now try tem against each other:

world = SCML2024OneShotWorld(
    **SCML2024OneShotWorld.generate(
        agent_types=winners + [SyncRandomOneShotAgent], n_steps=50
    ),
    construct_graphs=True,
)
world.run()

world.plot_stats(pertype=True)
plt.show()

You can also try them in the new scml-std simulation which has the same API as scml-oneshot:

world = SCML2024StdWorld(
    **SCML2024StdWorld.generate(agent_types=winners + [SyncRandomStdAgent], n_steps=50),
    construct_graphs=True,
)
world.run()

world.plot_stats(pertype=True)

Download Notebook.