scml.runner =========== .. py:module:: scml.runner Classes ------- .. autoapisummary:: scml.runner.WorldRunner Functions --------- .. autoapisummary:: scml.runner.mean scml.runner.median Module Contents --------------- .. py:function:: mean(lst: list[float]) -> float Calculates the mean .. py:function:: median(lst: list[float]) -> float Calculates the median .. py:class:: WorldRunner(generator: scml.oneshot.context.BaseContext, n_configs: int = 10, n_repetitions: int = 1, save_worlds: bool = True, save_common_stats: bool = True, save_agent_stats: bool = True, combiner: Callable[[list[float]], float] = np.mean, control_all_agents: bool = False, shorten_names: bool = True, progress: bool = False) A utility class to run controlled simulations for different agent types. :param generator: The `Context` used to generate world configurations :param n_configs: The number of configurations :param n_repetitions: Number of repetitions of each configuration :param one_offer_per_step: Whether one negotiation step is one offer. Pass as `True` when using RL :param save_worlds: If given worlds will be saved internally. You need this to use worlds_of and the like :param save_common_stats: Save general agent-independent statistics over time :param save_agent_stats: Save agent statistics over time :param combiner: How to combine scores and statistics if the number of agents of the tested type in a world is more than one :param control_all_agents: If given, the agent type(s) passed to __call__ will control the whole market. :param shorten_names: If given, shorter versions of type names will be used when compiling type-names :param progress: If given, a progress bar will be displayed for each world being run .. py:attribute:: n_configs :value: 10 .. py:attribute:: n_repetitions :value: 1 .. py:attribute:: generator .. py:attribute:: configs .. py:attribute:: worlds :type: dict[tuple[str, str], list[WorldInfo]] .. py:attribute:: save_common_stats :value: True .. py:attribute:: save_agent_stats :value: True .. py:attribute:: save_worlds :value: True .. py:attribute:: control_all_agents :value: False .. py:attribute:: _stats .. py:attribute:: _scores .. py:attribute:: _combiner .. py:attribute:: config_names .. py:attribute:: shorten_names :value: True .. py:attribute:: progress :value: False .. py:attribute:: type_names :type: set[str] .. py:method:: from_runner(src: WorldRunner, **kwargs) -> WorldRunner :classmethod: Creates a `WorldRunner` from another one. Remarks: - It simply copies the configurations created by the source runner. - You can override any parameters by passing them as keyword arguments .. py:method:: from_configs(world_type: type[scml.oneshot.world.SCMLBaseWorld], configs: tuple[dict[str, Any], Ellipsis], name: str | None = None, **kwargs) -> WorldRunner :classmethod: Creates a `WorldRunner` from a collection of configs. :param world_type: The world type to use. :param configs: The collection of configs to use. Remarks: - Uses a `RepeatingContext` internally. - You can override any parameters of the new runner by passing them as keyword arguments .. py:property:: existing_config_names :type: list[str] Existing configuration names. .. py:property:: existing_type_names :type: list[str] Existing type names tested so far. .. py:method:: get_type_name(type_: type[AgentType] | str) -> str Used to get a name for the given type. .. py:method:: add(types: tuple[AgentType, Ellipsis] | list[AgentType] | AgentType, params: list[dict[str, Any]] | dict[str, Any] | None = None, name: str | None = None) -> list[WorldInfo] Used to add an agent type collection to the pool of tested agent types. Does not run the worlds :param types: In general, you can pass a collection of types here. Note that this collection will be used in every simulation. This means that the placeholder_types in the context used to create the runner must accept enough such agents. The usual use-case is that the placeholder_types in the context contain a single type and in this case, you can either pass a singelton collection of a single type or just pass the type directly. :param params: Parameters accepted by the constructor of the types passed. :param name: The name to call this type(s). :returns: A list of pairs each containing a world and a list of agents representing the worlds that were run and the agents corresponding to the passed type(s). .. py:method:: runall(progress: bool = False) -> None Runs all worlds .. py:method:: _runall(world_infos: list[WorldInfo], progress: bool = False) -> None Runs all worlds .. py:method:: __call__(types: tuple[AgentType, Ellipsis] | list[AgentType] | AgentType, params: list[dict[str, Any]] | dict[str, Any] | None = None, name: str | None = None, progress: bool = False) -> list[WorldInfo] Used to add an agent type collection to the pool of tested agent types. :param types: In general, you can pass a collection of types here. Note that this collection will be used in every simulation. This means that the placeholder_types in the context used to create the runner must accept enough such agents. The usual use-case is that the placeholder_types in the context contain a single type and in this case, you can either pass a singelton collection of a single type or just pass the type directly. :param params: Parameters accepted by the constructor of the types passed. :param name: The name to call this type(s). :returns: A list of pairs each containing a world and a list of agents representing the worlds that were run and the agents corresponding to the passed type(s). .. py:method:: _types_and_configs(type: AgentType | str | None = None, config: str | None = None) -> tuple[list[str], list[str]] .. py:method:: world_infos_of(type: AgentType | str | None = None, config: str | None = None) -> list[WorldInfo] Returns the information of all worlds of the given type and config. Remarks: Note that you can also pass the name used when running the WorldRunner with the given type. .. py:method:: worlds_of(type: AgentType | str | None = None, config: str | None = None) -> list[scml.oneshot.world.SCMLBaseWorld] Returns the worlds of the given type and config. Remarks: Note that you can also pass the name used when running the WorldRunner with the given type. .. py:method:: agents_per_world_of(type: AgentType | str | None = None, config: str | None = None) -> dict[str, list[AgentType]] Returns the agents representing the type for each world of the given type and config. Remarks: Note that you can also pass the name used when running the WorldRunner with the given type. .. py:method:: agents_of(type: AgentType | str | None = None, config: str | None = None) -> list[AgentType] Returns the agents representing the type and config. Remarks: Note that you can also pass the name used when running the WorldRunner with the given type. .. py:property:: stats :type: pandas.DataFrame The statistics saved (must be constructed with `save_stats`) .. py:property:: all_world_infos :type: list[WorldInfo] Information of all worlds simulated (including the agents evaluated for each world) .. py:property:: all_worlds :type: list[scml.oneshot.world.SCMLBaseWorld] Information of all worlds simulated (including the agents evaluated for each world) .. py:property:: scores :type: pandas.DataFrame The scores of all evaluated agents in all evaluated worlds Remarks: Other than the score, the returned data-frame will contain information about the production level, number of suppliers, number of consumers, number of competitors, etc about each agent. .. py:method:: score_summary(percentiles=None, include=None, exclude=None, by: str | tuple[str, Ellipsis] = 'type', order_by: str | None = 'score', ascending: bool = False) -> pandas.DataFrame A summary of comparative scores of all agent types tested so far. :param percentiles: passed to `groupby` :param include: passed to `groupby` :param exclud: passed to `groupby` :param by: passed to `groupby` :param order_by: The method for sorting resulting scores. Possibilities: score, mean, min, max, 20%, 50%, 75%, median :param ascending: Ascending or descending scores. :returns: A dataframe that describes the scores of all evaluated types. .. py:method:: plot_stats(stats: tuple[str, Ellipsis] | str | None = None, by: tuple[str, Ellipsis] | str = 'type', agg: bool = True, legend: bool = True, ylegend: float = 1.8, legend_ncols=3, title: bool = True, order_by: str | None = 'score', ascending: bool = False, **kwargs) Plots saves statistics (`save_stats` must be given) :param stats: The stats to be displayed. If not given, a default set will be chosen. Any statistic saved by the `World` or `SCMLBaseWorld` can be used. There are over 42 such statistics. :param by: Group the data by the given attribute (used as hue if agg is False and as x if agg is True). :param agg: Whether to aggregate over simulation steps or not. :param legend: whether or not to show a legend (only used if agg is False) :param legend_ncols: How many columns to use in the legend. Pass zero to disable the legend :param ylegend: The y-coordinate of the legend to control where it appears. :param title: Show stat names as title (instead of ylabel) :param order_by: The statistic to order with. Possibilities are score, mean, 50%, max, min, 20%, 75%, median :param ascending: If true, order ascendingly :param \*\*kwargs: Any extra paramters to pass to the underlying seaborn method (lineplot in case agg=False and boxplot in case agg=True) :returns: The figure and axes used. .. py:method:: draw_worlds_of(type: AgentType | str | None, config: str | None = None, what=('contracts-signed', ), n: int | None = 4, randomize: bool = False) Draws the given set of worlds :param type: The type to filter by :param config: The config to filter by :param what: what stat to draw. See `negmas.situated.World.draw_world` for all options available :param n: Number of worlds to draw. If None, all of them will be drawn :param randomize: If given the worlds will be shuffled before display but only if not all worlds will be displayed :returns: figure and axes used. .. py:method:: process(info: WorldInfo)