Compare agents

An agent is your own code that plays the platform's environments. The agent runs on your machine, the environments run on the platform, and every step can be recorded and replayed.

You need the platform's address and a token. In an experiment on a connected machine (see gpu-machines.md), PLATFORM_URL and PLATFORM_TOKEN are already set. Anywhere else, set them yourself.

1. Install the client

The platform serves its client package, so you need no other download:

mkdir -p wheels
(cd wheels && curl -fsSOJ -H "Authorization: Bearer $PLATFORM_TOKEN" "$PLATFORM_URL/v0/wheels/platform-sdk")
pip install wheels/platform_sdk-*.whl

It holds the session client and the agent interface, and needs only numpy, gymnasium, httpx and protobuf. Each version of the platform serves a matching client, so fetch it again after an upgrade.

2. Write the agent

An agent is any Python object with start and act, made by a function that takes your settings as keyword arguments:

# my_agent.py
class MyAgent:
    def __init__(self, seed=0):
        self.seed = seed

    def start(self, env, streams):
        self.space = env.action_space
        self.space.seed(self.seed)

    def act(self, observations, rewards, terminated, truncated, infos):
        return [self.space.sample() for _ in observations]


def make(**settings):
    return MyAgent(**settings)

start(env, streams) is called once, before the first decision.

• env.env: the environment's catalog id.
• env.observation_space, env.action_space: Gymnasium spaces.
• env.metadata: the control rate, action names and similar.
• streams: how many sessions the agent drives.

act(observations, rewards, terminated, truncated, infos) makes one decision per stream and returns one action per stream. Each argument is a list with one entry per stream.

• rewards, terminated and truncated describe that stream's previous action. Before its first action they are None and False.
• When an episode ends, the observation is already the next episode's first, and the info carries final_obs and final_info.

Converting observations into your model's inputs, and its outputs into actions that fit env.action_space, is up to your agent.

These methods are optional:

Set a max_streams attribute if one agent can drive several sessions at once (the default is 1).

3. Run it

From the folder holding my_agent.py:

python -m platform_sdk.agents.run --agent my_agent:make --env cartpole \
  --episodes 20 --record

The log shows one line per episode, then a JSON summary with mean return, success rate, compute time and the recording's id. Add --save agent.ckpt once your agent has a save method.

To drive several sessions at once, or to control the run from your own code, call the loop directly:

import platform_sdk
from platform_sdk.agents.loop import run_episodes

from my_agent import make

agent = make(seed=1)
with platform_sdk.connect("cartpole") as env:
    for record in run_episodes(agent, [env], episodes=10, seeds=range(10)):
        print(record)

Pass one session per stream in the list. platform_sdk.agents.builtin has a seeded RandomAgent and a ScriptedAgent to compare yours against.

Run without episodes

For continual learning, nothing has to end the experience stream:

python -m platform_sdk.agents.run --agent my_agent:make --env cartpole \
  --endless --steps 100000 --window 1000 --save agent.ckpt --save-every 10

Time limits are off, and an environment that still ends (a fall, a game over) restarts in place. That step reports neither terminated nor truncated; its observation is the new start, and its info holds restarted and final_info. The log has one line per --window steps. Agents never see those boundaries. With --record, the recording is split into files of 1,000 steps.

From your own code, pass kwargs={"endless": True} to platform_sdk.connect.

Compare agents

To see which agent copes best with the same surprises, branch them all from one moment:

python -m platform_sdk.agents.compare --env menagerie-franka-emika-panda-reach \
  --agent mine=my_agent:make --agent base=my_agent:baseline \
  --warmup 200 --steps 6000 --window 50 --record \
  --kwargs '{"world_changes": [
    {"at": 2200, "change": "mass", "body": "hand", "scale": 2.0},
    {"at": 4200, "change": "mass", "body": "hand", "scale": 1.0}]}'

The session plays --warmup steps with a seeded random agent and takes a snapshot. Each agent starts fresh from it, or from --load NAME=checkpoint. The timeline is part of the snapshot, so every agent meets each change at the same step, and the agents run one at a time so they do not compete for the machine. The results appear on the Comparisons page, and the command prints the link.

Leave at least one --window of steps before and after each change, or that change is reported as too close to measure. Each agent runs once, so small differences may be noise. The environment must support snapshots, so real hardware cannot be compared this way.

Hold an action

hold repeats one action for as many steps as you say, in one uninterrupted stretch, and returns a JPEG of each step:

held = env.hold(action, 8)
held.rewards, held.frames, held.observation, held.terminated

# Let the world keep moving while the agent decides what to do next:
pending = env.hold(action, 8, wait=False)
next_action = agent.think(observation)
held = pending.result()

It stops early if the episode ends. Up to 256 steps can return frames; pass frames=False to hold for up to 10,000. Anything in agent= is recorded with the first step, where the decision was made. Take no other step until a wait=False hold has finished.

Save the world

A snapshot normally lives only as long as its session. To keep one through a crash or a server restart:

saved = env.save_snapshot("before the swing")

# later, from anywhere, even after a restart:
env = platform_sdk.resume(saved["id"])
observation, info = env.resumed   # already at that moment: step next

The new session starts from the same environment and arguments, and carries on exactly as the original would have. To return a session that is still running, call env.restore_saved(id), which needs the same environment and arguments. Saved snapshots are listed on the Recordings page. Real-time environments cannot be saved. The log of world changes applied so far is not part of a snapshot, though the changes themselves are.

Record a whole life

To keep one continuing record of a long run, name it:

python -m platform_sdk.agents.run --agent my_agent:make --env cartpole \
  --endless --steps 1000000 --record --life robot-1 --max-frames 20000

Recording again under the same name, from any session and after any restart, appends to it; each episode's life_start is its first step counted across the whole life. One session at a time can write a life. Continue it after a crash by combining this with a saved snapshot.

Thinning affects frames only: actions, rewards, observations and agent data are kept for every step. A thinned file adds frame_steps, the step each stored frame belongs to. Because frames is then shorter than the steps, thinned recordings do not load with EpisodeFolder.

What is recorded

With --record, each step keeps the frame, action, reward and episode end, and what readouts() returned. Values appear on the replay page and in the episode files as agent.<name>.

• A value is a finite number, a boolean (stored as 0 or 1), or a list of up to 256 numbers.
• A step can carry up to 32 names, and a recording up to 256. Names use letters, digits, _, . and -.
• A name keeps the shape of its first value. Values that break these rules are left out rather than stopping the run.

Two names have a set meaning: