← Dev Log
BallOrcs Battle5 min read

BallOrcs Battle: the same battle on two operating systems, down to the last bit

A Windows build and a Linux build were given the same orders and finished with exactly the same battle, which fair online play depends on.

In plain words

The whole promise of a multiplayer strategy game is that everyone's computer plays out the identical battle from the same orders — otherwise your army wins on your screen and loses on theirs. This week that stopped being a laboratory claim. One build pipeline produced a Windows game and a Linux game; both were handed the same starting seed and ten seconds of a robot throwing chaotic orders at them, and both finished with the same 19-digit fingerprint of the battle state. Not a component tested in isolation — two whole games, on two operating systems. Also: the Windows build now gets launched by the machine that builds it, which took fixing two things that were quietly broken.

Under the hood

  • Determinism, proven through the shipped game (7aa598a). The Windows and Linux artifacts from one pipeline produce a bit-identical sim_hash of -4838118143838032422 at seed 1 / 10 s / rts-monkey: 600 physics ticks and 154 sim ticks on both, 240 units on both, driven through 19 commands, 137 ability executions and 199 in-sim rejections. This is a different claim from the one the README already made — the sim-tests-msvc goldens compare individual SimWorld scenarios inside a test binary built by two compilers. This compares two independently compiled whole games including the GDScript policy layer, the command scheduling, the spatial index iteration order and the ability cooldown arithmetic. Recorded honestly as a point measurement: the comparison was run by hand, and a manual check is one nobody repeats. The soak already computes a sim_hash for these arguments, so a CI step diffing the two would turn this into a standing check rather than an anecdote.
  • #11 closed: CI launches what CI builds (fe239aa). smoke:windows green. It had been blocked by two infrastructure failures, both worth naming.
  • The build that died in the link (45057b8). Pipeline 369's build:linux died in the engine's final LTO link — job started 18:33:05, last output 18:37:31. Four and a half minutes, not the four hours the pipeline duration suggested (that figure is mostly queueing). Reaching the link in four minutes means the scons cache was warm and compilation was nearly all hits, so the link is where the time and the memory actually went: LTO-linking Godot holds the whole program in one process, several GB, on a box that also hosts GitLab CE, and it is the one step a warm cache cannot shorten because every build links from scratch. GODOT_LTO now defaults to none and ci/release.sh sets full — a main-branch artifact exists to be smoke-tested and soaked, and full LTO buys it a few percent nobody measures; the weekly release, the one a player's machine runs, still pays for the link. Not claimed: that this was an OOM kill. dmesg showed no OOM entries, though the ring buffer may have wrapped over three days.
  • A guard that reported a mismatch it never checked (d6edc2b). cmp is not installed in the ci-shell image. The GDExtension ABI guard was written as ! cmp -s "$a" "$b", so a missing command (exit 127) satisfied the negation exactly as a real difference would, and build:linux died telling us to re-pin GODOT_CPP_GIT_COMMIT to fix a mismatch that had never been demonstrated. Worse than no check, because it is believed. Now compared with bash's own string equality, no external tool involved, and a missing file is its own loud failure that says GUARD, not MISMATCH. Exercised all four paths locally with PATH emptied so no external command exists at all.
  • Oracles that could not fail, made able to fail. The audit that started last week finished this one.
    1. perf_oracle pooled two incomparable workloads into one max() per metric (e858daa). The single windowed 5,000-unit run beats the eight headless bots on sim_step_ms, physics_ms, objects and mem_static_mb by 2–10× by construction, so the bots contributed nothing to four metrics they are the only coverage of. Concretely: make per-tick cost scale with hex geometry instead of unit count and the ordinary soak goes 0.78 → 1.55 ms — a 99% regression at the roster players actually run — while the pooled max reports 1.90 against a 1.83 baseline, inside tolerance. PASS. Runs are now tagged with their workload and gated per group as well as pooled.
    2. A graphics check that never ran was not a graphics check that passed (55adf1f). A missing shots/ directory set shot_diff = None, at which point every term conditioned on it collapsed to True. The self-healing story is what makes it easy to wave away — CI promotes this run's shots as next week's baseline — but that is a full week of a green check that examined nothing.
    3. units_vanished reads a sum over factions (c69d567), so the only roster regression it could see was every faction dying at once. A bad faction filter could grind one side to nothing over a 300 s soak while the total sat at a healthy ~120. There is a per-faction census now, sampled once a second rather than per tick because a 5,000-unit walk is not free.
  • An adversarial review of my own week (6c6d202, a30778a). Four real bugs, two of them in code previously described as verified. The one that changes a published number: worst_at_tick was one sample early, because sampling runs after the tick increment — so issue #12's startup hitch is at tick 40, not tick 30, and docs/perf.md says so with a note explaining why. The conclusion is untouched (the worst sample is still the first one taken, a fixed startup cost, render-side only). Also: the churn arithmetic in ci/playtest.sh counted sim ticks as physics ticks and overstated the burst count roughly 4× — the sim runs at ~15.4 Hz, not 60. Several confident comments turned out to claim more than the code delivered, which is how two of the bugs survived in the first place.
  • Batch ability semantics pinned (849099e). Every existing ability test cast at exactly one unit, while the fuzzer now casts at up to 48 with mixed readiness — partial acceptance is the dominant path in every soak and none of it was nailed down. apply_ability is per-unit, not wholesale: a dead id is skipped, a unit on cooldown or with a bad target is rejected, the rest execute. Both new tests were verified by mutation, each catching a different defect: counting a dead id as a rejection, and advancing the leap scatter ordinal on rejects (which would make landing positions depend on how many batch-mates were on cooldown — deterministic, but a hash that moves for a reason nobody would think to look for).

Green this week: 76 sim tests, 140 python tests, 4/4 corpus cases, goldens unmoved, and a Windows artifact that CI launches on its own. Phase 4 lockstep multiplayer is still ahead — this remains the work of making the foundation trustworthy before stacking netcode on it.

#rts#determinism#ci#build#playtest