# PStack Causal Truth Plane **Status:** foundation extension **Scope:** Mode S, CI admission/merge, Fastlane runners, Longhorn recovery, alert delivery, and agent handoffs **Principle:** inspect the live mechanism at the layer where the claim lives ## Why this exists `pstack` replaces a story about a process with the process's actual thread state. The reliability programme repeatedly needs the same move at larger scales: - a green workflow is not proof that its intended transition happened; - a firing alert is not proof that an owner received and acknowledged it; - a Longhorn volume marked healthy is not proof every database page is readable; - low node load is not proof a multi-minute TCP stream can survive; - a clean PR is not proof its finalizer receipts exist or the writer is legally admitted; - a PID is not a process identity unless start time, UID, boot ID, namespace, and cgroup still match. The common defect is **proxy substitution**: observing an adjacent signal and promoting it into a claim about the mechanism. > A process is any live state machine. Read its state directly, bind the observation to an immutable identity, bound the read, and keep inference separate from evidence. The Causal Truth Plane is not another dashboard or database. It is a shared evidence protocol for existing collectors and authorities. ## Five laws ### 1. State before story Capture the mechanism before interpreting it. Logs and summaries are direct evidence only when the claim is about logs or summaries. | Claim | Direct surface | |---|---| | process is blocked | thread state, `wchan`, stack/syscall, scheduler state | | PR is merge-ready | exact gate generation, finalizer receipts, review/hold state, lane, writer admission | | rebuild fails in transport | source/target identity, TCP state, engine terminal | | alert reached an owner | route, receiver result, external message ID, acknowledgement | | database is readable | logical dump or page-integrity proof, not replica robustness alone | ### 2. Identity before state Minimum subject identities: - process: host, boot ID, PID, start ticks, UID; - thread: process identity plus TID; - workflow transaction: repository, PR, exact head/base, run ID, attempt; - network stream: source, destination, protocol, flow ID; - storage transfer: volume, engine, source node, target node, transfer ID; - alert delivery: alert fingerprint, route generation, receiver. Collectors re-read identity around slow surfaces. A mismatch is `identity_race`, never a best-effort attachment to a newer subject. ### 3. Bounded observation before completeness Every potentially blocking surface has a per-surface deadline and every bundle has a whole-collection deadline. Timeout is evidence. For procfs, `/proc//stack`, `wchan`, and `syscall` can block on the very D-state task being diagnosed. A timer around synchronous `open().read()` is not a deadline. Use an isolated helper. If the helper cannot be reaped, emit `timeout_unreaped_helper`, journal its identity, consume a bounded leak budget, and continue with partial evidence. A `complete` bundle contains only successful, untruncated surfaces and a complete declared universe. A `bounded` or `partial` bundle cannot support universal claims. ### 4. Falsifiers before fixes Every hypothesis states what observation would kill it before mutation. Prefer the cheapest safe probe that separates important branches. ### 5. Receipts before closure `SOURCE_FIXED`, `MERGED`, `REALIZED`, `VERIFIED`, `SOAKED`, and `CLOSED` are distinct transitions. One layer's receipt cannot substitute for another. ## Architecture ```text existing collectors and provider readers │ typed surfaces, never prose ▼ PStack Evidence Envelope identity · clock · universe · bounds · surfaces · claims · unknowns · next transition │ ├── canonical seal + SHA-256 digest ├── closed-world validator ├── deterministic before/after diff └── compact agent-frontier handoff ▼ Mode S · Butler/finalizers · watchdogs · storage recovery · alert lifecycle · agents ``` `scripts/ops/pstack/pstack_evidence.py` does not collect data. Existing domain readers remain adapters. The shared layer governs identity, freshness, completeness, claims, unknowns, transitions, sealing, and handoff. ## Evidence envelope Every envelope contains: 1. **Subject** — immutable identity of the observed process or transaction. 2. **Collector** — name, version, and exact source commit. 3. **Clock** — wall time, monotonic observation point, and bundle deadline. 4. **Universe** — selector, expected/observed count, termination reason, and selector digest. 5. **Bounds** — explicit caps. 6. **Surfaces** — authority, typed status, deadline, digest, bytes, truncation, and structured facts. 7. **Claims** — facts, inferences, or hypotheses with evidence bindings. 8. **Unknowns** — answerable questions, blocked transitions, owners, and discriminating probes. 9. **Next transition** — one legal frontier action, success receipt, and abort conditions. 10. **Bundle digest** — canonical SHA-256 over every field except the digest itself. Large or sensitive payloads remain in artifacts referenced by URI and digest. ## Existing foundations to strengthen ### Linux process truth Converge the Mode S process collector and shard-executor survivor classifier around shared identity and typed surfaces. The classifier requires planted `R`, `S`, `D`, and `Z` fixtures. Mapping every state to `R` must fail. Capture identity twice: ```text before slow reads: boot_id, pid, start_ticks, uid, namespace, cgroup slow bounded surfaces before emit: boot_id, pid, start_ticks, uid, namespace, cgroup ``` Any change invalidates attribution. ### Mode S lifecycle A candidate is a distributed process: ```text claim -> build -> execute -> terminal classify -> hold/quarantine/release -> journal -> provider receipt -> consumer acceptance ``` Every edge is `not_started`, `running`, `success`, a typed hold/error, or `unknown`. The observer reports the first missing edge. `ambiguous_failure` must not reuse permanent typed-red quarantine. It is an attribution hold with an executable resolution: ```text ATTRIBUTION_HOLD ├── CONFIRMED_RED -> TYPED_RED_QUARANTINE └── EXONERATED -> RELEASED exactly once ``` ### Admission to merge For each PR, capture the full readiness vector: ```text exact head/base canonical Policy Guards generation policy-admission shadow cardinality and generation app-only admission cardinality and expiry binding review-thread and hold-label state Butler lane classification writer credential eligibility merge commitment B/H/C/tree/actor receipt post-merge sentinel ``` Example frontier: ```text policy_guards success generation X shadow missing app_only_admission blocked_on_shadow writer_admission not_legal next_transition PUBLISH_SHADOW ``` The merger re-reads the same vector immediately before the SHA-fenced write. Observation explains authority; it never grants it. ### Harness liveness Measure service rate by lane: - routine eligible vs routine converted; - controlled eligible vs controlled dispatched/merged; - L3 awaiting owner decision. Twenty controlled PRs and no routine PRs must not trigger a routine-throughput alarm. ### Storage and transport Model a Longhorn rebuild as a storage transfer linked to a network stream: ```text volume/replica and instance-manager identity route, interface, MTU and PMTU TCP bytes, duration, retransmits, RTOs, zero-window and resets RST endpoint NIC/veth drops, conntrack pressure, Cilium verdict engine progress and terminal ``` CPU load is a hypothesis surface, not the stream state. ### Alert delivery Trace the entire process: ```text Prometheus firing -> Alertmanager received -> silence/inhibition -> route -> receiver response -> external message ID -> acknowledgement -> repair -> recovery ``` A firing alert proves only the first edge. ## Agent loop ```text CAPTURE -> BIND IDENTITY -> CLASSIFY UNKNOWN -> CHOOSE FALSIFIER -> MUTATE -> CAPTURE AGAIN -> DIFF -> ISSUE RECEIPT ``` Before mutation require one sealed bundle, typed completeness, facts separated from hypotheses, blocking unknowns with probes, and one next transition. After mutation produce a second bundle for the same subject, deterministic diff, success or typed-failure receipt, updated hypothesis state, and compact handoff. ## Rollout ### P0-A — envelope and validator Land the contract, CLI, hostile tests, architecture, runbook, and work graph. Use an existing CI shard; add no required context. ### P0-B — procfs adapter Extract identity/deadline primitives and typed procfs surfaces, then kill all-`R`, stale-PID, missing-deadline, and helper-leak mutations. ### P0-C — Mode S and admission adapters Emit first-missing-edge vectors for one real candidate and one real PR. Require zero unexplained disagreement with provider behavior before consumers switch. ### P0-D — transport and alert adapters Instrument natural Longhorn/Git/GHCR streams and trace one live critical alert end-to-end. Do not change timeouts, concurrency, routing, or production workloads in this phase. ### P1 — evidence-addressed handoffs Agent Mail carries a compact handoff plus bundle digest. Beads records canonical URI, observation time, expiry, and next transition. ### P2 — consumer enforcement Only after shadow agreement should Mode S reject stale evidence, Butler consume readiness vectors, watchdogs consume lane-specific service rates, and closure tooling require named receipts. ## Acceptance metrics - time to first discriminating observation decreases; - proxy-substitution retractions decrease; - repeated investigation of settled questions decreases; - open unknowns without an owner and probe reach zero; - stale or unbound handoffs reach zero; - transitions per agent-hour increase; - tokens spent narrating status decrease. The adversarial acceptance test gives a new agent only the compact handoff and artifact access. It should identify the first missing transition, challenge the highest-impact hypothesis, and take the next legal action without rereading the programme history.