BallOrcs Battle: we broke our own test on purpose, to see if it would notice

In plain words

Last week we hand-checked that the Windows and Linux versions play out an identical battle. This week a robot does it on every change — and before trusting it, we deliberately sabotaged one side so the two genuinely disagreed, just to watch the alarm go off. It did, and we reverted the sabotage the same hour. That is the theme of the whole week: a check nobody has seen fail is not yet a check. Ten separate places turned out to have the same flaw — they printed "fine" when they had actually been unable to look. One was writing a file of security checksums that was completely empty. Another meant our weekly release had been built without the speed optimisation it asked for, silently, for weeks.

Under the hood

• The determinism gate is standing, not a point measurement (3c269c9, 03dfc29, 6d7bcad, a268678, #18). The cross-platform sim_hash comparison that was run by hand last week is now a pipeline job, and it is known to work in both directions in its real environment: pipeline 376 identical hashes → MATCH, PASS, 7 seconds; pipeline 377 → MISMATCH, FAIL, pipeline red. The failing half came from a throwaway commit (edd28eb, reverted by 453a9d2 and verified byte-identical to its pre-mutation state) that ran the Linux side on seed 2 while smoke:windows stayed on seed 1. The mutation had to live in the script, because the "one definition" rule that makes the gate trustworthy also means both sides read the same variable and would move together. Crucially it failed as MISMATCH, not as CANNOT COMPARE — it sent the reader after a determinism divergence rather than a broken pipeline, and it could only say that because it genuinely compared two hashes.
• Ten instances of one defect: "cannot run" printing as "pass" (debca11, 81c62ea catalogue them). The question that catches all of them: what does this check print when it cannot run at all? The worst three:
  1. An empty SHA256SUMS shipped as a supply-chain claim (5998d95). ( cd "$dir" && ... sha256sum -- * > SHA256SUMS 2>/dev/null || true ), with the success log line printed before the work and unconditionally. Three assertions of success, none checked. An empty checksum file is worse than none: anyone verifying a download finds their artifact absent and must guess between tampering and a broken build. The rewrite refuses, naming the cause, on five paths — and the line-count-equals-file-count check earned its place immediately by catching the fix's own first draft, which wrote its temp file inside the directory being hashed so the file hashed itself.
  2. SOURCE_DATE_EPOCH guessed at 0 (8f15708) — inside setup_determinism_env(), under the comment "Reproducible timestamps", git log -1 --pretty=%ct 2>/dev/null || echo 0. A git that cannot answer yielded 0, and the log then printed SOURCE_DATE_EPOCH=0 in the same shape as a real answer. Same commit, built where git works and where it does not, produced different embedded timestamps. For a project whose central claim is determinism, guessing here is the worst available choice; it now derives or refuses.
  3. Two knobs that never crossed into the build container (8fba724), one of which reaches players. in_container() forwards an explicit -e allow-list, and a variable missing from it reads as unset inside — the build silently does the default thing. GDEXT_ALLOW_CACHE was absent, which meant the documented way to reproduce a known fault was unrunnable by construction (proven on pipeline 395's log). GODOT_LTO was absent too, and ci/release.sh sets it to full for the shipping build — so the weekly release has been building without the LTO it asks for, succeeding every time and merely being slower than intended forever. A test now parses the -e list and fails naming any missing knob.
• A measured price for a cache exclusion (c8a5df0, #13). An accidental control — a cancel that did not take — built the same commit both ways minutes apart: pipeline 396 with the GDExtension scons cache off → smoke:windows PASSED in 22:01; pipeline 397 with it on → 0xC0000005 access violation at GDExtension init, smoke-out never created. Turning the cache on takes build:linux 13 min → 2:52 and build:windows 6:41 → 2:24, so the exclusion costs about 14 minutes per pipeline — now a measured price for an artifact that runs rather than an estimate attached to a hypothesis. Recorded at the exclusion itself in build/common.sh. Causation proven, mechanism not: godot-cpp declares extension_api.json as a binding source, so the cache key ought to discriminate already, and it demonstrably does not.
• The web bundle gets launched now (d04f845, 38cac27). build:web packaged a wasm bundle nothing had ever loaded — the same hole that shipped a faulting Windows build. IMAGE_PLAYTEST gained node and Playwright's Chromium (licences checked, not assumed: Apache-2.0 / BSD-3-Clause / MIT), with Fedora's runtime deps named explicitly and the image build ending by launching Chromium so a browser-less image fails at build time instead of three pipelines later. The first real run then failed for two reasons, both only findable by running it: --modes monkey is the character-controller mode and this is an RTS, so the bot drove nothing for ten seconds and reported "rts": {} with a named failure rather than a pass built on zeroes; and the run produced 301 physics ticks in 10 s (~30/s against a native 60, headless Chromium sampling 1–6 FPS) against a SIM_RESOLVE_GRACE_TICKS of 300 — so the entire smoke was grace, and would have passed or failed on scheduling noise. Default is now 30 s, ~900 ticks.
• Android is checked but still not launched (b1bce06, c7e3ac1, #19). A package check found that three files independently decide which CPU architectures ship, and they disagreed: build/platforms/android.sh sets TARGET_ARCHES="arm64 x86_64", while project/game.gdextension and export_presets.cfg list arm64 only. So every Android build compiles an x86_64 engine template and GDExtension and discards both, and the APK is arm64-only — meaning the x86_64 emulator those artifacts exist "for" cannot run it. Reported as a warning, because it wastes build time but ships nothing incorrect and the fix is a decision. What is an error: an ABI the preset ships with no matching .gdextension entry, where the engine loads and the game runs with no sim. 15 tests over synthetic APKs cover it. #19 stays open until something is actually launched.
• The runner stopped eating the box (7da9c44, bb43540). On 2026-08-09 a nominal 2 h build thrashed for 6 h without finishing, GitLab timed it out server-side at 240 min while the runner measured 5 h 55 m, the runner then panicked during container cleanup, and the orphaned container kept compiling for a further 11.5 hours until the box was rebooted. Two causes: the WSL guest is now capped at 8 GB (~7.8 GB reported, ~1.3 GB into swap at idle with GitLab CE, GlitchTip, postgres and valkey resident), so -j4 on Godot's heavy translation units overcommitted it — now -j2. And gitlab-runner's [runners.docker] memory limit bound the wrong container: measured mid-pipeline on 2026-08-10, ci-shell peaked at 21 MiB against its 4 g cap while the sibling doing the actual compiling sat at 2.4 GiB with memlimit=0. BUILD_MEM_LIMIT now applies --memory/--memory-swap at in_container itself, equal values so an over-budget compile is OOM-killed in minutes rather than thrashing — a legible failure that leaves the runner healthy.

188 tests green across the three suites, 18 jobs, shellcheck clean, and push-to-gate is now ~25 minutes rather than ~60 (web and android moved to build-on-demand so they stop queueing ahead of a 7-second gate). Phase 4 lockstep multiplayer is still ahead; this is still the work of making the foundation trustworthy before stacking netcode on it.