Failure Forensics¶
Every Foundations story that retried more than once. For each, what failed, why, and what fixed it. The CI logs for the older runs have aged out of GitHub's retention window, so root causes for those are reconstructed from commit subjects + the critic-log artifacts.
Summary¶
| Story | Title | Attempts | Net retry time | Root cause class |
|---|---|---|---|---|
| F-02 | Split cmd/ into operator/agent/webhook/nwctl |
2 | ~6m | Misconfig (Deployment command path) |
| F-15 | e2e-fake.yml workflow |
2 | ~3m | yq syntax — Mike Farah vs python yq |
| F-16 | security-source.yml workflow |
4 | ~52m | GHAS not enabled (CodeQL upload required it) |
| F-30 | internal/os/ OSProvider interface + per-OS stubs |
34 | ~6h | GitHub-hosted runner billing outage |
Total: 42 attempts on 4 stories, ~6h 40m of net retry time. F-30 alone is 97% of the retry-time budget; everything else cleared in under an hour.
F-02 — cmd/ entrypoint split¶
Story. Split the kubebuilder-generated single binary into four entrypoints: operator, agent, webhook, nwctl. Each lives under cmd/<name>/main.go; the operator stays the default container ENTRYPOINT.
| Attempt | Commit | Outcome | What changed |
|---|---|---|---|
| 1 | 97c5b7e |
❌ ci_fail (run 25612047928) | Original split. |
| 2 | 8403b82 → b9e6564 |
✅ ci_pass (run 25612257832) | fix(config): point manager Deployment command at /bin/operator — kustomize's manager.yaml still referenced the original /manager binary path. Updated to /bin/operator. |
Why this slipped past local make. The split renamed the binary from manager to operator. make build produces all four binaries correctly; make manifests regenerates RBAC/CRDs but does not touch the manager Deployment's command: field. The integration test deploys config/default against kind, which fails when the configured command doesn't exist.
Fix prevents regression? Yes — the integration test now exercises the kustomize-rendered manifest, so renaming a binary in cmd/ without updating manager.yaml will fail integration immediately.
F-15 — e2e-fake.yml workflow¶
Story. Add the e2e-fake lane that runs chainsaw against kind on the highest K8s minor from .github/k8s-matrix.yml.
| Attempt | Commit | Outcome | What changed |
|---|---|---|---|
| 1 | 3cd8e21 |
❌ ci_fail (run 25614924803) | Resolve-K8s-matrix step crashed: Error: 1:13: lexer: invalid input text "last". |
| 2 | fd85fcf → 46e6121 |
✅ ci_pass (run 25615053313) | fix yq syntax for Mike Farah (F-15 retry) — workflow used the python yq -y .versions[-1] syntax; the runner has Mike Farah's go-yq, which uses '.versions | last'. |
Why this is permanent. The bulk-CI runner pool ships with Mike Farah's yq baked into the cloud-init; PRD-02 FR-37 forbids using the python yq because it's not part of ubuntu-latest. The workflow code now matches the runner contract.
F-16 — security-source.yml workflow (4 attempts, the F-16 saga)¶
Story. Add the security-source lane: gosec, govulncheck, gitleaks, CodeQL.
| Attempt | Commit | Outcome | What changed |
|---|---|---|---|
| 1 | f945d35 |
❌ ci_fail (run 25615371113) | Original lane. govulncheck and CodeQL both flagged. |
| 2 | dc0ee39 |
❌ ci_fail (run 25615591214) | bump go directive to 1.25.9 — chasing a govulncheck CVE rule that demanded a newer Go. Didn't help. |
| 3 | 1b7f7d3 |
❌ ci_fail (run 25615831218) | bump go directive to 1.26.3 — same chase, no luck. |
| 4 | a41f797 → f0a5300 |
✅ ci_pass (run 25616051577) | skip CodeQL upload when GHAS not enabled — the actual root cause. |
Real root cause. The CodeQL analyze step's upload-sarif action requires GitHub Advanced Security (GHAS) to be enabled on the repo. nctiggy/nodewright is a GitHub Free private repo; GHAS is a paid add-on. Two attempts wasted bumping Go versions (a red-herring govulncheck warning), then attempt 4 wrapped the upload step in if: success() && env.GHAS_ENABLED == 'true' and the lane went green.
Why local make didn't catch it. make security-source runs locally without the upload step (you don't have a SARIF endpoint to push to). Only the CI workflow tries to upload, and only against a repo without GHAS does it 401.
Fix is intentional, not a workaround. PRD-01 FR-12 explicitly says "GHAS optional for v0.1; the security-source lane MUST function without it." So the gate is now: scan locally + emit SARIF + skip upload when GHAS isn't there. v1.0 will revisit if SC turns GHAS on for the repo.
F-30 — internal/os/ OSProvider interface + per-OS stubs (34 attempts)¶
Story. Add the OSProvider abstraction interface and per-OS stub implementations (Kairos for v0.1, Ubuntu for v0.2+, RHEL for v1.0+). Pure code-shape work — no behavior, no tests beyond compile.
The code that landed in attempt 1 was correct. Every failure between attempts 1 and 33 was a CI infrastructure outage, not a code defect. Attempt 34 was the same code on a working CI path.
| Attempt range | Commit(s) | Outcome | Cause |
|---|---|---|---|
| 1 | bfd1bf4 |
❌ ci_fail (run 25619312570) | First push under the GitHub-hosted runner billing/quota outage. All 7 lanes returned 10-second no-log failures with empty runner_name — the textbook signature of a Free-tier minute exhaustion or payment-method issue on the nctiggy account. |
| 2–32 | various retries + interleaved F-31/F-32/F-33/F-34 implementations the inner agent rolled forward | ❌ ci_fail × 32 (in 04:00–06:30 UTC May 10) | Same outage. 56+ CI runs total during this window across all stories committed during the retry — every single one returned the same 10s empty-log failure. |
| 33 | 92ee7ff |
❌ ci_fail (run 25621570591) | Transitional state — workspace cleanup race during the runner cutover. The bulk-CI runner had stale _work/nodewright/nodewright/ state from the in-flight runner restart; actions/checkout@v6 128'd. |
| 34 | (bfd1bf4 re-evaluated by the resumed ralph) |
✅ ci_pass (run 25623136380) | First iteration on the new [self-hosted, linux, x64, ci] pool with the pre-job cleanup hook installed. |
Three runner bugs the saga forced into the open¶
yq: command not found—ubuntu-latestships Mike Farah's yq; the cloud image doesn't. Cloud-init now installs it explicitly. (See docs/runbooks/ci-runner.md.)yamllint: command not found(PATH) —pipx install yamllintlands in~/.local/bin, not on the runner Worker's PATH. Now overridden via/home/runner/actions-runner/.env.fatal: --local can only be used inside a git repository— envtest leavesbin/k8s/<ver>/binaries owned by root; the runner user can'trmthem, so workspace cleanup leaves.githalf-deleted, andactions/checkout@v6trips. Fixed viaACTIONS_RUNNER_HOOK_JOB_STARTEDpre-job hook that wipes the workspace as root via NOPASSWD sudo.
Why this took us 6 hours to diagnose¶
- The first 32 failures were on the GitHub-hosted runners, not under our control. We had to pivot to self-hosted before we could even see the cleanup-hook bug.
- The runner cutover surfaced 3 distinct bugs (yq, PATH, workspace cleanup) sequentially, each requiring a fresh CI cycle to reproduce.
- The
metrics.jsonmis-attributed iters 34–53 to F-30 asno_commit— but real F-31/F-32/F-33/F-34 implementation commits were happening (and failing) under the F-30 attribution. The build journal now surfaces this honestly via the Retry sagas prologue.
Lessons recorded in memory¶
- Self-hosted runner is now the default for the bulk-CI pool. The architecture brief §6 documents the cutover. Runner-pool details:
docs/runbooks/ci-runner.md. - Pre-job cleanup hook is required for any CI lane that uses envtest (
make test, integration). Without it, a test run that creates root-owned files in_workbreaks the next job's checkout. - Cloud image ≠ ubuntu-latest. Tools you assume are there (
yq,~/.local/binon PATH, certain apt packages) are not. Cloud-init must install them, or workflows must install them per-job.