feat(cum-ci): serve the CI store as a binary cache; let kally and quartz keep their nixpkgs #60

Merged
marti merged 3 commits from feature/ci-binary-cache into main 2026-11-12 02:12:41 +01:00
Owner

Summary

CI already builds kally and quartz on every push, into cum-ci's store. cum-media could not use those builds for two reasons: no VM could fetch from cum-ci, and following our nixpkgs made cum-media build different derivations anyway. So every deploy rebuilt both from source, using a lot of RAM.

flowchart LR
    CI[kally / quartz CI job] -->|nix build, shared daemon| Store[(cum-ci /nix/store)]
    Store --> H[harmonia :5000, signed cum-ci-1]
    H -->|substituter, after cache.nixos.org| Media[cum-media deploy]
 flake.nix
   kally  = { url = ...;
-    inputs.nixpkgs.follows = "nixpkgs";
     inputs.trail-code.inputs.nixpkgs.follows = "nixpkgs"; };
   quartz = { url = ...;
-    inputs.nixpkgs.follows = "nixpkgs";
     inputs.trail-code.inputs.nixpkgs.follows = "nixpkgs"; };

 hosts/cum-ci/modules/
+└── binary-cache.nix     # harmonia on the CI store, key from sops
 nixos/system.nix
   substituters
+    http://192.168.1.206:5000   (every VM but cum-ci)
   trusted-public-keys
+    cum-ci-1:dhoG...
  • nix flake lock locked kally's nixpkgs to a fresh nixos-unstable instead of the one kally's own lock pins, so that lock entry is set to kally's pin (f9a75f4) by hand. A later nix flake update kally takes it from kally's lock again.
  • Earlier, Attic was dropped as too much upkeep. Harmonia has no database, no push step and no tokens: it only serves the store cum-ci already has.

Evidence

The derivations cum-media asks for, compared with what each project's CI builds at the locked revision (nix eval ...drvPath):

before (follows ours) after CI at locked rev
kally hplqq87j…-kally-0.2.0 i4mm0bik…-kally-0.2.0 i4mm0bik…-kally-0.2.0
quartz lri0jq5p…-quartz-server-0.1.0 ky4yf93h…-quartz-server-0.1.0 ky4yf93h…-quartz-server-0.1.0
  • nix flake check: all six nixosConfigurations evaluate, and all checks pass.
  • Evaluated config: cum-media's substituters include http://192.168.1.206:5000 and its trusted keys include cum-ci-1. cum-ci's substituters do not include it. cum-ci opens TCP 5000.
  • Not verified: that cum-ci's store actually holds those outputs right now (cum-ci was not reachable from where this was written). Check after deploy with curl http://192.168.1.206:5000/<hash>.narinfo.

Merge Danger

Door: two-way

Reverting the commits restores the old behaviour. The signing key stays in sops either way.

Blast Radius: all-VMs

Every VM gets a new substituter. If cum-ci is down, Nix skips it and falls back to the public caches or to building, as it does today. Deploy cum-ci first: until harmonia runs there, other VMs try an address that refuses connections. Builds still work, with a warning. Harmonia serves anything in cum-ci's store to the LAN, which is the same exposure as the other LAN-only services. cum-media's closure also gains an unstable Node 26, a couple hundred MB.

🤖 Generated with Claude Code

## Summary CI already builds kally and quartz on every push, into cum-ci's store. cum-media could not use those builds for two reasons: no VM could fetch from cum-ci, and following our nixpkgs made cum-media build different derivations anyway. So every deploy rebuilt both from source, using a lot of RAM. ```mermaid flowchart LR CI[kally / quartz CI job] -->|nix build, shared daemon| Store[(cum-ci /nix/store)] Store --> H[harmonia :5000, signed cum-ci-1] H -->|substituter, after cache.nixos.org| Media[cum-media deploy] ``` ```diff flake.nix kally = { url = ...; - inputs.nixpkgs.follows = "nixpkgs"; inputs.trail-code.inputs.nixpkgs.follows = "nixpkgs"; }; quartz = { url = ...; - inputs.nixpkgs.follows = "nixpkgs"; inputs.trail-code.inputs.nixpkgs.follows = "nixpkgs"; }; hosts/cum-ci/modules/ +└── binary-cache.nix # harmonia on the CI store, key from sops nixos/system.nix substituters + http://192.168.1.206:5000 (every VM but cum-ci) trusted-public-keys + cum-ci-1:dhoG... ``` - `nix flake lock` locked kally's nixpkgs to a fresh nixos-unstable instead of the one kally's own lock pins, so that lock entry is set to kally's pin (`f9a75f4`) by hand. A later `nix flake update kally` takes it from kally's lock again. - Earlier, Attic was dropped as too much upkeep. Harmonia has no database, no push step and no tokens: it only serves the store cum-ci already has. ## Evidence The derivations cum-media asks for, compared with what each project's CI builds at the locked revision (`nix eval ...drvPath`): | | before (follows ours) | after | CI at locked rev | |--------|------------------------|------------------------|------------------------| | kally | `hplqq87j…-kally-0.2.0` | `i4mm0bik…-kally-0.2.0` | `i4mm0bik…-kally-0.2.0` | | quartz | `lri0jq5p…-quartz-server-0.1.0` | `ky4yf93h…-quartz-server-0.1.0` | `ky4yf93h…-quartz-server-0.1.0` | - `nix flake check`: all six nixosConfigurations evaluate, and all checks pass. - Evaluated config: cum-media's substituters include `http://192.168.1.206:5000` and its trusted keys include `cum-ci-1`. cum-ci's substituters do not include it. cum-ci opens TCP 5000. - Not verified: that cum-ci's store actually holds those outputs right now (cum-ci was not reachable from where this was written). Check after deploy with `curl http://192.168.1.206:5000/<hash>.narinfo`. ## Merge Danger **Door:** two-way Reverting the commits restores the old behaviour. The signing key stays in sops either way. **Blast Radius:** all-VMs Every VM gets a new substituter. If cum-ci is down, Nix skips it and falls back to the public caches or to building, as it does today. Deploy cum-ci first: until harmonia runs there, other VMs try an address that refuses connections. Builds still work, with a warning. Harmonia serves anything in cum-ci's store to the LAN, which is the same exposure as the other LAN-only services. cum-media's closure also gains an unstable Node 26, a couple hundred MB. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI already builds kally and quartz on every push, into cum-ci's store,
but no host could fetch from it, so each deploy rebuilt them from source
with a lot of RAM. harmonia now serves that store, signed with cum-ci-1,
and every other VM lists it as a substituter after the public caches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With nixpkgs following ours, cum-media built kally and quartz as
derivations their CI never builds, so the cum-ci cache could not serve
them. Each now uses the nixos-unstable its own flake.lock pins, and the
derivations match what CI builds at the locked revisions:
kally i4mm0bik..., quartz ky4yf93h....

nix flake lock resolved kally's nixpkgs to a fresh nixos-unstable
instead of the one kally's own lock pins, so that lock entry is set to
kally's rev (f9a75f4) by hand. A later `nix flake update kally` takes it
from kally's lock again.

The cost is a second nixpkgs to evaluate, and an unstable Node 26 in
cum-media's closure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
marti merged commit e73ee75c72 into main 2026-11-12 02:12:41 +01:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cumsek/nix!60
No description provided.