Root filesystem disk audit: point-in-time attribution and recommendations (v2 supersedes v1)

Report overview

Root filesystem disk audit: point-in-time attribution and recommendations (v2 supersedes v1)

Snapshot (UTC)

2026-10-19T14:27Z to 2026-10-19T14:40Z (v1 pass 14:27-14:30; v2 scan 14:39:39-14:40:47)

Units

GiB = 2^30 bytes; GB = 1e9 bytes; fields ending _b are bytes; *_b_v1 fields come from the v1 pass and are not mixed into v2 GiB figures

Review status

independent review pending

Point-in-time, approximate, read-only analysis. Nothing was deleted, restarted, rotated or executed. Recommendations only. No prices or cost estimates. Generic labels only.

Figures in GiB come from the inventory-v2 scan. Byte fields suffixed _b_v1 come from an earlier v1 pass and are not mixed into GiB figures.

Headline numbers

df size 160921083904 bytes, df available 0 bytes. At the audit snapshot (taken mid-cleanup, ~14:27-14:40Z) the root filesystem was full, with 0 bytes available. A later read-only probe at 16:00Z (different time and metric) showed 16.11 GiB available, 89% used, and falling: down 245 MiB since ~15:19Z.

The v1 byte figures below come from a different pass (14:27-14:30). They are not directly comparable to the v2 GiB figures above.

Allocation inventory (GiB, v2)

Ranked by size

Additivity (centi-GiB integers): parts 148.42 + remainder 0.01 = 148.43 GiB; total_gib 148.43 GiB. Result: additive.

Allocated (dedup) versus apparent size

Two separate series, never merged. Allocated minus apparent (2.68 GiB) is attributed to ext4 block rounding and sparse-file effects; this is INFERRED, not measured. Hardlink duplicates (2.38 GiB) are excluded from allocated_dedup. v1 pass (different snapshot): allocated 161921875968 B, apparent 156491051863 B.

*_b_v1 byte figures come from a different (v1) pass than v2 GiB. v1 bytes: allocated 161921875968 B, apparent 156491051863 B, hardlink 2556145664 B. Not mixed into the GiB figures above.

Section confidence: inferred

df reconciliation (waterfall, GiB)

Waterfall: df used 149.65 GiB − allocated dedup 148.43 GiB = 1.22 GiB residual, unattributed (stated residual 1.22 GiB).

Overlapping rollups

Each row is a separate view of the same bytes. Do not add rows together.

Extensions (ranked by GiB)

Ranked by allocated GiB. Extension is taken from the filename, not from file content.

Modified-time histogram (PROXY ONLY)

PROXY ONLY, not growth, no time series. Last-write, create, copy or touch time; not a growth measure.

Months sum to 148.36 GiB against 148.43 GiB allocated: a 0.07 GiB gap. Cause not measured (unverified).

Largest entries (redacted)

Redacted: leaf names are withheld. Columns are rank, GiB, type and directory group only.

Worktrees

Dependencies

Edges are the recorded dependency intent (configured, policy or observed). Node state is what was last recorded for each node. Nothing here was probed live.

Edge confidence

• verified (solid line)
• inferred (dashed line)
• unverified (dashed line)

Node state

Shutdown impact

Derived only from recorded edges. A node with no recorded dependents is not proof that nothing depends on it.

Installed services: wiring and shutdown impact

Services on this host and how they connect. Confidence: M = measured, C = config-evidenced, I = inferred.

Documentation conflicts

• API origin is named as 138.201.247.255 in AGENTS.md and CLAUDE.md, but as 49.12.12.91 in docs/ops/2026-10-15-old-server-deletion-audit.md:164-171, where that box is recorded as deleted.
• DB host is 46.224.109.170 (public) and 10.88.0.20 (private); docs agree with shared/scaler.py:71-74.
• 188.245.228.253 appears only as a fallback default in shared/scaler.py:74, deploy/worker_start.sh:31, deploy/scaler_start.sh:21 and deploy/provision_dedicated_worker.py:42. No doc names it.
• This host is named nowhere in the documented topology.

Installed vs source

• Installed tree is on main with 8 uncommitted entries; the session branch is clean.
• Four shared files differ between installed tree and source: audio_source.py, routes.py, worker_daemon.py, and one migration.
• The venv python reports 3.14.4 while its config records 3.12.3 (M).

Unverified

• Public 502 on songchop.com and api-origin.songchop.com (inferred from the refused 7860 connection, not curl-verified).
• Whether the daily backup upload to the remote storage box succeeded (journal has no entries for the day).
• Current state of producehits-host-guard (earlier audit recorded failed with timer active).
• Working directories for each unit, except the sampled ones.
• Owner of the true gateway behind the documented API origin addresses.

Writers, triggers, retention and measured growth

Facts

• Free space: 4.1 GiB available at 14:56Z; 16.4 GiB now (+12.3 GiB available). About 12.6 GiB of used space was reclaimed. Attribution is unexplained.
• Audit-time used figure: 149.65 GiB (earlier report, at audit time). df used was 127.4 GiB at ~15:19Z and 127.6 GiB at 16:00Z. These are different times; do not equate them.
• Used-space gap: df used 127.4 GiB against home plus /var at 101.3 GiB leaves 26.1 GiB unattributed by that sum. Reserved blocks (6.1 GiB) affect Avail only and are not part of this gap.
• Deleted-but-open files: 36.7 MiB across 4 processes (fail2ban-server, chrome, bash, filebrowser). Target paths not verified.
• Disk-free log (/tmp/disk-free-log.txt, mtime 14:56): its counters record 0 bytes freed for every itemized step. Manual agent deletions on the audit day are the candidate writer of the observed free-space change (INFERRED, partly unverified; no per-path bytes re-measured; the audit allocated-size metric and df-used are different metrics, so delta attribution is not established).

Gaps (unexplained)

• Writer of /tmp/disk-free-log.txt is not found in the searched trees. Its 100M vacuum counter matches no configured vacuum (scripts use 200M).
• About 6.6 GiB of the 12.6 GiB reclaimed has no candidate after the ~/.cache drop (6.0 GiB, measured, unattributed); candidate is manual agent deletions on the audit day (inferred).
• The ~/.cache shrink has no per-subdirectory baseline; the prior snapshot itemizes about 29 MB of 7.38 GB.
• About 9.7 GiB of the 26.1 GiB used gap remains after a 2.4 GiB ext4 metadata estimate (inode size assumed, not measured). Data hidden under mountpoints and stale superblock counters are not checked.
• Journal: journalctl reports 24M, du reports 145M. The old machine-id dir (113M) is not counted, and no rule removes it.
• Effective journald cap (500M rather than 200M) is inferred from file order and has not been checked with systemd-analyze.
• Lane worktree +0.45 GiB has no established cause. The prior 587 dirs and 120 repos are not reconciled; re-measured as 300 linked plus 1 main checkout over 301 roots.
• host-guard prune functions and reclaim_disk: call sites at :293-294 are commented out. Whether any of them run, and the disk threshold, are unverified.
• producehits-tmp-sweep thresholds are unread, so what it deletes is unknown.
• /var/tmp holds 216.4 MB with 3 recent files and no age rule. Its contents and writer are not checked.
• /tmp/lane-pg liveness is not checked. The writer of the 1G tmpfs under home (~/fill.UDRv/tmp) is unknown.
• node_modules and venv share is unverified (find at depth 3 returned 0).
• Root crontab is empty. The contents of /etc/cron.d (certbot, e2scrub, producehits-*) were not read.
• Merged and pushed status of the largest worktrees is not checked.

Sizes are apparent bytes. The mtime proxy shows recent writes only, not growth.

Migration

RECOMMENDATION ONLY: each item is a proposed change, not an instruction.

Principles

• Cutover: stage each change beside the running system, verify it, then switch over in one documented step.
• Rollback: keep the previous artifact and its config; restore it, re-run the verifier, and record the outcome.
• Bounded scratch quotas with TTL on TEMP, export cache, jobs and projects roots.
• FLAC goes to the archival StorageBox.
• OGG goes to S3-compatible object storage.
• WAV is produced on demand, not stored by default.
• SANCT stays intact: no change to its components or its data paths.
• No processing worker on the gateway; processing runs on workers only.
• No prices appear in this report.
• Nothing executed by this report: every item below needs its own owner decision and execution.

Coding-only relocation checklist and bounded lifecycle plan

Labels: measured = command output in the topology notes; config-evidenced = unit, script or file text; inferred = reasoned from those, not checked.

Part A. Coding-only host: relocation checklist

Dependency order (sequence, not executed)

1. Gate 0: close topology questions T1 to T4 (table below) by written receipts.
2. Gate 1: DNS for api-origin and songchop.com resolves to the one confirmed gateway address (check R1, run from outside this host).
3. Gate 2: the gateway serves the API route to a check from outside (check R2).
4. Gate 3: exactly one scaler process and one autopull timer on the gateway (checks R3, owner-run).
5. Gate 4: a fresh remote backup exists and its run is logged (check R4, owner-run, not from this seat).
6. Gate 5: rollback copies exist: nginx site files and the backup timer unit saved to a dated folder (cutover C1).
7. Cutover C2 to C6 (below). Only after gates 0 to 5 pass in writing.
8. Part B configuration (journald, logrotate, tmpfiles, observability) is independent of the cutover. Apply after gate 0 with owner approval.

Cutover steps (later)

1. C1: copy the nginx site files and the backup timer unit to a dated folder (to be created). Gate 5.
2. C2: remove the songchop.com, www and api-origin proxy blocks from the nginx site files. Keep the static roots. Run nginx -t.
3. C3: reload nginx only after C2 passes nginx -t.
4. C4: disable producehits-backup.timer only if the backup has moved, and only after the new host has completed one successful run.
5. C5: run R6 and the public check from outside. Record the result in the cutover receipt.
6. C6: write the disposition receipts for every item touched.

Rollback steps (prepared before cutover)

1. RB1: restore the saved nginx site files, run nginx -t, then reload nginx.
2. RB2: if C4 was applied, re-enable and start producehits-backup.timer.
3. RB3: if DNS was changed by the owner, revert the records to the values recorded at gate 1. This plan takes no DNS action itself.
4. RB4: remove the Part B drop-in files and restore the prior journald settings. The effect follows the next journald configuration load.

Unresolved topology questions blocking the plan

Part B. Bounded lifecycle: coding host

Part B. Bounded lifecycle: future service VPS

Remote storage (backup box, object storage) is not retention without ownership, limits, TTL and cleanup. It is not counted as capacity for any category above.

Settings for journald, logrotate, tmpfiles and caches

Observability

Worktree rule

Dispositions for dirty or unfinished work

Dispositions: Ready, Continue, Salvage, Parked, Superseded. Receipts go under docs/work-dispositions/. This checkout has a top-level work-dispositions/ folder and no docs/ folder, so the receipt location must be reconciled. Nothing below is Ready.

Not verified in this plan

• The gateway address was not checked from this host (T1).
• Whether any running environment uses the 188.245.228.253 default (T2).
• Whether the backup uploaded (T4). Journal empty today.
• Disk thresholds in producehits-tmp-sweep and producehits-host-guard reclaim_disk; whether the host-guard prune functions run.
• Journald effective cap (inferred from file order, not from systemd output).
• Old journal directory (about 113M) and whether any rule removes it.
• Current inode use and cache sizes. Service host disk size.
• The full bodies of go-live.sh and dev-sync.sh.
• Public requests to songchop.com and api-origin.
• The cloudflared binary path used by the unit.

Cleanup forensic verdict: what was deleted on the audit day

What was deleted, and the evidence for each removal

Did we mess up?

Partly, yes. Most of what was removed was regenerable cache and virtual-environment material, and that class can be restored by reinstalling. The real losses are different. The derived round-8 outputs and wav set (about 1.7 GB) appear gone with no copy found. The scratch service was stopped without checking its owner and is down. The key-corpus restore script now points at directories that no longer exist. Journal history and log content before the trims cannot be recovered. The removals were agent-chosen. For non-cache items there was no pre-deletion backup or bundle, and no per-path byte counts were taken first. The "about 24 GB free" figure in the cleanup record matches no measurement. Measured available space was 4.1 GiB at 14:56Z and 16.11 GiB at 16:00Z, and it is falling again. The freed-space gain cannot be attributed to named deletions. The producehits-app failure at 13:19Z predates the cleanup and is not attributed to it.

Snapshot against the later probes

Figures from different metrics and times are shown side by side for context. They are not a before-and-after comparison of any single deletion.

Limits

• No per-path bytes were measured before deletion, so no deletion can be credited with a specific byte count.
• The later probes are one batch at 16:00Z. Corpus recovery was not exercised and no fixture manifests were opened.
• The "about 24 GB free" figure in the cleanup record was not reproduced by any measurement.

Not done by this report

No cleanup, restore or migration performed by this report. No deletion, service start or restart, credential change or file restore was made. The later-probe figures are read-only observations.

Update: causes, cleanup accounting and preventive changes (evening 2026-10-19)

Current correction: 2026-10-20 recovery and enforcement

The historical audit below is preserved, not a live dashboard. At 13:30 UTC the development disk was again 96% used. About 240 crew checkout directories occupied 52,466 MiB. The replacement services host was only 4% used: moving scheduled services did not move or bound agent checkouts.

Recovery removed 206 clean inactive checkout copies with remotely preserved commits and no ignored files, retaining their branches. Dirty, active, held, parked and unarchived work was preserved. Disk use fell to 64%, with about 53 GB available. Package cache reclaim was separate. This is a current measured recovery, not validation of the older inferred reclaim totals.

Permanent enforcement is now active on the development host: new lane creation invokes retention and admission under the creation lock; default maximum 120 claimed checkout directories and minimum 20 GiB free. A separate installed timer runs safe retention every 15 minutes with a 6-hour age. No session or pane is killed. Ignored files block removal and all safety gates are checked again immediately before removal. Source commit 069ed9d6 is backed up; actual local dispatch uses this source. Fleet publication is pending unrelated intentional edits in the authoritative checkout, not silently claimed complete.

The services-host scheduled backup did fire at 03:53:18 UTC with exit status 0. Both daily analytics jobs have successful target-side output. Weekly search jobs are not yet due. The development backup timer remains disabled. Pack migration completed: 476 directories checked, 409 remotely verified and evicted, zero errors or missing uploads. The 67 differing-manifest directories remain preserved, with no overwrite. Local packs fell to 561 MiB and root ended at 62% used, about 56 GB available.

Causes of continued growth, from paired measurements

Used space rose about 1.25 GB in the 7 minutes between samples A and B and about 0.36 GB in the 6 minutes between B and C. Disk was 87% used with about 19.9 GB free at the last sample. Ranked growers:

Not growing in the window

Reading: the audit-day pressure came from many large worktrees. In this window the dominant growth is still new worktrees, not session logs, packs, uploads or the journal. Three samples over about 13 minutes are a short window, not a trend.

Cleanup accounting since the audit

The 36-45 GB reclaim figure is an inference from free space moving from 16 GiB at 16:00Z while new worktrees kept arriving. There is no byte receipt for it and it is not added to the 12.3 GiB figure as a measurement. The two amounts are separate and not summed here.

Permanent preventive changes: honest status

Status words follow the work-disposition receipts and the main branch history. Landed means present in the remote main history, not deployed or proven in production.

Verified

• Paired disk-usage samples were taken three times and compared directory by directory (measured).
• Worktree directory count, total size and per-checkout size were measured on the host.
• Packs, uploads, journal, backups and deleted-open files were measured flat or small in the window.
• Landed status of the named source commits was checked against the remote main history.

Not verified

• The 284 removed worktrees: count, sizes and that every one was fully merged have no receipt in this report.
• The 36-45 GB reclaimed figure (inferred only).
• Whether the worktree cap and the cold-session archiver exist anywhere beyond the incomplete checkpoint.
• Gateway deployment of the pack and upload fixes, and any production run of the cold mirror.
• The first services-host backup run, due after this update; backup restore parity.
• Who disabled the development host backup timer, and why.
• Growth beyond the 13-minute window; no daily or weekly trend exists.
• Ownership of each remaining worktree directory.
• Independent review of this update.

Duplication and overlap

Hardlink duplicates are excluded from the allocated total. The rollups below overlap each other and the inventory parts. Do not add them up or to the allocated total.

Limits of this evidence

These limits apply to every figure in this report. Read them before acting on any number.

1. residual 1.22 GiB (df used minus allocated dedup) is unattributed
2. worktree counts unreconciled; lane groups are candidates only
3. executor group labels not verified against workload
4. unit states observed in executor logs only
5. root-checkout parity with installed units unverified
6. fill tmpfs origin not inspected
7. DB worker rows and billing not checked
8. gateway-down cause not established
9. top-30 leaf names intentionally withheld
10. point-in-time snapshot with concurrent writers
11. v1 figures superseded by v2: v1 listed 26 lane categories and left 45.87 GiB uncategorised due to truncation; df used is byte-derived 149.65 GiB, not 149.63
12. allocated vs apparent difference attributed to block rounding and sparse files is inferred, not measured
13. extension is a filename property, not a content type
14. independent review pending
15. models claimed cxproxy/claude-haiku-5-5 unverified
16. mtime month map sums to 148.36 GiB vs 148.43 GiB dedup allocated: 0.07 GiB difference unexplained, likely two-decimal rounding of ~300 near-zero buckets (unverified)
17. durable copy limitation: at the audit snapshot (~14:27-14:40Z) the root filesystem was full with 0 bytes available and the Pages deployment plus /tmp artifacts are the only copies; /tmp is volatile. The 16:00Z probe showed 16.11 GiB available, so this is time-scoped, not current
18. update section: 284 merged-worktree removals and the 36-45 GB reclaim are executor-reported or inferred; no byte receipt
19. update section: preventive-change status is from commit and receipt inspection only; deployment and production behaviour not verified
20. update section: three paired samples over about 13 minutes are not a growth trend
21. Exposure (measured, session grep output): a session-level grep printed literal database credentials from a team-access runbook. Values are not repeated in this report. Owner decision required; no credential changes were made by this report.
22. Deletions occurred during and after the snapshot window (Claude session approx. 14:17-14:56Z), so the audit snapshot (14:27-14:40Z) does not describe the post-cleanup disk state.
23. No per-path pre-delete byte counts were captured, so the amount freed by each deletion is not measured.
24. The audit allocated-size figure and the later df-used figure are different metrics at different times; delta attribution of df-used change to the deletions is not established.
25. One generated dataset (derived round data and wav files) is unrecovered and presumed lost.
26. The cleanup record claim of about 24 GB free matches no measurement. The audit snapshot shows df avail 0 B; the post-cleanup probe shows df avail 17,303,072,768 B (about 17.3 GB). Free-space figures from the df free column and the avail column are not interchangeable.

Independent review

Review status: independent review pending. This report has not been independently reviewed. Treat every figure as provisional until that review is recorded.

Claims flagged unverified or inferred

• root-checkout parity with installed units unverified
• allocated vs apparent difference attributed to block rounding and sparse files is inferred, not measured
• models claimed cxproxy/claude-haiku-5-5 unverified
• mtime month map sums to 148.36 GiB vs 148.43 GiB dedup allocated: 0.07 GiB difference unexplained, likely two-decimal rounding of ~300 near-zero buckets (unverified)
• update section: 284 merged-worktree removals and the 36-45 GB reclaim are executor-reported or inferred; no byte receipt

Sources

• inventory-v2 redacted summary (publishable; scan 14:39-14:40; authoritative for all GiB figures)
• v1 redacted summary and v1 reconciliation (14:30; superseded where they disagree with v2)
• research handoff (v1 figures; superseded where they disagree with v2)
• research runtime and retention notes (observed unit states from executor logs; recommendations)
• deleted-open summary (7 inodes, counts only)
• growth measurement summary (three paired samples 22:52Z-23:05Z, read-only)