← Dev Log
BallOrcs4 min read

The build that died of silence, and the orc who would not sit still

No new orc hats this week — this was a week of fixing the machine that assembles the game.

In plain words

No new orc hats this week — this was a week of fixing the machine that assembles the game. One build had been dying after nearly two hours with no error message at all: the final assembly step was so hungry for memory that the computer spent all its time shuffling things in and out of storage, went quiet, and the build service assumed it had hung and pulled the plug. The cure was to stop applying an expensive optimisation to a tool that players never receive. Separately, the automatic screenshot check kept flagging three pictures of the orc as "changed" when nothing had changed — because the camera was firing while the ball was still rolling to a stop. Now it waits until the orc is actually still.

Under the hood

Three fixes, none of them gameplay, all of them things that were quietly lying about their own state.

  • A two-hour build killed by a reaper, not a compiler (22e5fad). build:linux #1706 failed after 115 minutes with failure_reason: no_updates_running — GitLab's stuck-job reaper, which drops a running job when the runner emits no trace output for an hour. The log simply stops mid Linking Program bin/godot.linuxbsd.editor.x86_64, with no error and no epilogue, which reads like anything except what it is. Traced through the pinned engine's own source rather than guessed at: SConstruct:674 has production=yes set lto="auto", and linuxbsd/detect.py:212 resolves "auto" to full LTO under GCC, which then links with -flto=<num_jobs> — several parallel threads each holding most of the program image. On a -j4 runner sharing 10 GB with GitLab itself, that link goes to swap, and ld prints nothing while it works, so a slow link is indistinguishable from a hung one. The fix is one word: the host editor is a build tool that runs --export and --script and is never shipped, so it links with lto=none. Everything a player receives comes from the export templates, which keep production=yes and therefore full LTO, untouched. Also: .claude/cowork/ce-job-log.sh now prints failure_reason, queue time and the runner, because inferring "killed" from a truncated log is guesswork when GitLab already knows and the field was merely not displayed.
  • Wait for the orc to stop moving (dc389e6, closes #100). Two playtest runs of the same commit, from the same build artifacts, disagreed on exactly three of 120 anchor screenshots — anchor_spawn_low (rms 3.82), anchor_spawn_yaw000 (7.37) and anchor_spawn_yaw090 (3.24) against a threshold of 3.0. The other 117 reproduced bit for bit, and those three are the only anchors that frame the character closely; every other anchor is level geometry. character_visual.gd derives its lean from the position delta with exponential smoothing, and camera_probe.gd shot after a fixed three frames — at spawn the ball is still settling under physics, so the shot caught the lean mid-smoothing. The probe now waits on the rendered transform, not on velocity: a ball at rest can still be visibly rotating for a fraction of a second while the lean converges, so a velocity test would call it settled while the thing that actually moved was still moving. Origin and basis are both compared, because a rotation is invisible to a position-only test. It settles once before the pose loop (the probe only moves the camera), caps at 300 frames ≈ 5 s, and on timeout warns loudly and shoots anyway — a probe producing no images is worse than one producing suspect images, but a silent give-up would reintroduce this exact flake while looking identical to a working run. This complements rather than replaces last week's flag fix (2c0585c): the evidence quoted above comes from a tree that predates it, so it is not a demonstration that the flags were insufficient. Still unproven either way — playtest does not run on merge-request pipelines, so the first automatic soak on main is the test. Success reads subject settled after N frames and those three names absent from the changed list.
  • A provenance file nothing ever wrote (cdfc245). ci/make-manifest.sh had never been invoked by anything — grepping build/, ci/ and .gitlab-ci.yml found no call site at all. Running it for the first time showed it had never worked: $( ) strips trailing newlines, so files_json+="$(printf ' {...},\n' ...)" lost the newline on every entry, all entries ran onto one line, and the trim that looks for a trailing ",\n" therefore never matched — leaving a trailing comma before the closing bracket. Invalid JSON, every single time. A script nothing runs cannot fail, which makes dead code strictly worse than absent code: it reads as a capability the project has. Also fixed: run_url was built only from GITHUB_SERVER_URL, so it came out empty on every pipeline that could actually run it (this repo moved to GitLab on 2026-07-19), and filenames are now JSON-escaped. Worth being honest about what it buys — dist/<platform>/SHA256SUMS already answers what shipped, and GitLab serves a single artifact file without downloading the archive. What a bare list of hashes cannot carry is provenance: which commit, which pipeline, when. The manifest runs last so it inventories SHA256SUMS too, and it warns rather than failing a build whose artifact is already written and verified.
#build#ci#playtest#screenshots