ForagerRL_step14 - gama-platform/gama GitHub Wiki

14. Training Multiple Agents

By Killian Trouillet

Step 14: Training Multiple Agents

Starting GAMA Headless

Windows

gama-headless.bat -socket 1001

Linux / MacOS

./gama-headless.sh -socket 1001

Port 1000 is reserved for the GUI. Use any other port for headless training.

Parameter-Shared PPO

In Part 2, we trained a single forager with PPO. Here we have two foragers, but they share the same observation/action structure and the same goal. So we use parameter sharing: one single neural network for both agents.

Why parameter sharing?

Criterion Parameter Sharing (ours) Independent PPO MADDPG
# Networks 1 (shared) 1 per agent Complex
Data efficiency Best — 2× data per update Standard Standard
Cooperation Emerges naturally Must emerge independently Explicit
Implementation Simple Simple Complex

Each agent feeds its own observation (15 values, including teammate position) into the same network and gets its own action back. Because both agents contribute trajectory data to the same network, learning is 2× faster.

Using GamaParallelEnv Directly

Unlike Part 2 where we used gym.make(), here we use GamaParallelEnv directly — the PettingZoo Parallel API:

from gama_pettingzoo.gama_parallel_env import GamaParallelEnv

env = GamaParallelEnv(
    gaml_experiment_path="path/to/forager_petz.gaml",
    gaml_experiment_name="petz_env",
    gama_ip_address="localhost",
    gama_port=1001,
)

obs, infos = env.reset()
# {"forager_0": array([...]), "forager_1": array([...])}

Batch Inference

We query the shared network for all agents at once using select_actions_batch():

# Collect observations from all active agents
active = [a for a in AGENT_IDS if a in obs]
obs_list = [np.array(obs[a], dtype=np.float32) for a in active]

# One forward pass through the shared network
actions_np, log_probs, values = agent.select_actions_batch(obs_list)

# Build the actions dict for PettingZoo
actions_dict = {a: actions_np[i] for i, a in enumerate(active)}

This is the same approach used in the Pistonball benchmark.

The Training Loop

agent = PPOAgent(state_dim=15, action_dim=2)
UPDATE_EVERY = 2048

total_steps = 0
agent_buffers = {a: RolloutBuffer() for a in AGENT_IDS}

for ep in range(1, NUM_EPISODES + 1):
    obs, _ = env.reset()
    step = 0
    done = False

    while not done and step < 300:
        active = [a for a in AGENT_IDS if a in obs]
        obs_list = [np.array(obs[a], dtype=np.float32) for a in active]
        actions_np, lps, vals = agent.select_actions_batch(obs_list)

        actions_dict = {}
        for i, a in enumerate(active):
            actions_dict[a] = actions_np[i]
            agent_buffers[a].states.append(torch.FloatTensor(obs_list[i]))
            agent_buffers[a].actions.append(torch.FloatTensor(actions_np[i]))
            agent_buffers[a].logprobs.append(torch.tensor(lps[i]))
            agent_buffers[a].values.append(torch.tensor(vals[i]))

        next_obs, rewards, terms, truncs, _ = env.step(actions_dict)

        for a in active:
            agent_buffers[a].rewards.append(rewards.get(a, 0.0))
            agent_buffers[a].dones.append(terms.get(a, False) or truncs.get(a, False))

        obs = next_obs
        step += 1
        total_steps += len(active)
        # `env.agents` only refreshes on reset(), so the per-agent flags decide the episode
        done = (all(terms.get(a, False) for a in AGENT_IDS)
                or all(truncs.get(a, False) for a in AGENT_IDS))

        # PPO update — all agents' data pooled into one gradient step
        if total_steps >= UPDATE_EVERY:
            agent.update(agent_buffers)
            agent_buffers = {a: RolloutBuffer() for a in AGENT_IDS}
            total_steps = 0

Key differences from Part 2

Aspect Part 2 (Gymnasium) Part 3 (PettingZoo)
Environment gym.make() GamaParallelEnv()
Observations Single array Dict {agent_id: array}
Actions Single array Dict {agent_id: array}
Rollout buffers One buffer One buffer per agent
PPO update agent.update(buffer) agent.update(agent_buffers) — pools all agents' data

What to Expect

  Ep   10/350 | steps 168 | f0 +136.6  f1 +111.1  | success  50%
  Ep   50/350 | steps  96 | f0 +318.1  f1 +356.6  | success  90%
  Ep  100/350 | steps  77 | f0 +315.4  f1 +353.7  | success  95%
  Ep  200/350 | steps  69 | f0 +316.0  f1 +362.1  | success 100%
  Ep  300/350 | steps  60 | f0 +317.7  f1 +361.1  | success 100%
  Ep  350/350 | steps  56 | f0 +316.8  f1 +359.7  | success 100%

Running the Training

cd models/petz
python train_forager_petz.py

The shared model is saved to saved_models/ppo_forager.pth.

Key Files

File Description
models/petz/forager_petz.gaml GAMA model with PetzAgent bridge
models/petz/train_forager_petz.py MARL training script (headless)
models/petz/test_forager_petz.py Testing script (GUI visualization)
⚠️ **GitHub.com Fallback** ⚠️