Benchmarks & Performance

AMBER stores the whole population as a columnar Polars DataFrame and compiles each step into a handful of vectorized expressions. From 0.4.4, the vectorized lane (step_vectorized / legacy step) runs on GPU via model.gpu().run() (GPU backend) with device-resident columns. Published AMBER (GPU) rows may use an explicit approve_fast_path(evidence) label before placement; the label is caller provenance, not a runtime certificate. Batched calibration is separate (GPU batched ensemble).

The full, reproducible suite — correctness checks, raw timings, and per-model tables — lives under benchmarks/ (see benchmarks/README.md). Reproduce with python benchmarks/run_all_frameworks.py.

Large-N multi-framework scaling (1k→10M)

Protocol: NVIDIA RTX 5090, 50 steps, 10 runs (trimmed mean). Ten frameworks: AMBER (GPU / vectorized / loop), mesa-frames, FLAME GPU 2, Agents.jl, SimPy, Melodie, AgentPy, Mesa. AMBER (GPU) uses the native vectorized models under model.gpu().run() (with evidence-labeled approval where a private fast loop is selected). Full table: benchmarks/results/summary_table.md.

Multi-framework scaling from 1k to 10M agents on RTX 5090

At 1M / 10M agents (where each framework still finishes):

Model

AMBER (GPU)

AMBER (vectorized)

FLAME GPU 2

Next best CPU-scale peer

Wealth

3.91 s / 193 s

6.44 s / 214 s

28 ms / 226 ms

Agents.jl 8.53 s @ 1M

Random walk

198 ms / 2.04 s

531 ms / 6.23 s

20 ms / 201 ms

mesa-frames 3.55 s / 20.8 s

Schelling

428 ms / 5.17 s

2.64 s / 59.8 s

2.06 s / 20.8 s

mesa-frames 4.33 s / 86.9 s

SIR (cell-list)

882 ms / 9.39 s

31.3 s / 308 s

108 ms / 3.80 s

—

  • Schelling: AMBER (GPU) is the fastest measured row at 1M and 10M.

  • Wealth / random walk: FLAME GPU 2 leads; AMBER (GPU) still leads other Python-hosted stacks that reach those scales.

  • SIR: AMBER uses cell-list infection. GPU 10M = 9.39 s (~33× vs vectorized 308 s); FLAME still leads at 3.80 s.

  • Reproduce: python benchmarks/run_all_frameworks.py --agents 1000 10000 100000 1000000 10000000 --steps 50 --runs 10 then python benchmarks/plot_scaling_with_gpu_schelling.py.

Calibration throughput

For derivative-free calibration, the GPU ensemble axis batches B simulations of N agents into one device pass (ambr.gpu_ensemble.smac_batch_calibrate()). On a well-mixed SIR recovery task, AMBER’s GPU batched ensemble reaches the best held-out validation loss at roughly 270× the slowest framework and ~3× the fastest CPU competitor, evaluating ~970 candidate parameter sets per second.