← Dev Log
BallOrcs1 min read

A thousand robot playtesters (who never get bored)

We hired playtesters who work for free, never sleep, and file perfect bug reports: little AI bots that live inside the game.

In plain words

We hired playtesters who work for free, never sleep, and file perfect bug reports: little AI bots that live inside the game. One kind rolls straight for the goal like a speedrunner; another mashes buttons at random like a toddler with a gamepad; a third replays a previous run exactly, down to the frame, to prove a bug is really fixed. A camera bot wanders around taking the same screenshots every run, so if the graphics ever change unexpectedly, we see it side-by-side. A swarm of these runs before each weekly build and produces a written report — including the exact recipe to reproduce anything that went wrong.

Under the hood

  • PlaytestDriver is an in-game autoload, completely inert unless the game is launched with -- --playtest. Crucially it plays through synthesized InputEvents via the real InputMap path — the bots press the same virtual buttons a player would, so input handling itself is under test, not bypassed.
  • Three drivers: a goal-seeking route bot steering through playtest_waypoint/playtest_goal node groups, a seeded monkey fuzzer (same seed → same chaos), and tick-exact trace replay — verified bit-identical against the release executable, which is the determinism pipeline paying rent a second time.
  • Per-tick oracles watch for NaN positions, physics explosions, stuck-in-place, and crashes; every run emits trace, path, events, and a summary verdict.
  • Camera mode screenshots deterministic, player-plausible vantages — 216/216 byte-identical across runs — plus 3-second orbit clips for flicker checks. Byte-identical screenshots turn "does it look right?" into a diffable question.
  • tools/playtest/run_swarm.py fans out N headless fixed-fps instances, aggregates verdicts and engine-log errors, diffs shots against a baseline release, and writes report.md with a repro command per failure. Exit code 0/1, so it slots straight into CI ahead of the weekly Monday build.
#testing#tooling#automation