# PStack Agent Loop Use this runbook before an agent diagnoses, mutates, hands off, or closes a Mode S, CI, runner, storage, identity, or reliability incident. ## Fast path ```bash TOOL=scripts/ops/pstack/pstack_evidence.py CONTRACT=config/pstack-evidence-contract.json python3 "$TOOL" seal \ --contract "$CONTRACT" \ --input var/process-truth/current.raw.json \ --output var/process-truth/current.json python3 "$TOOL" verify \ --contract "$CONTRACT" \ --input var/process-truth/current.json python3 "$TOOL" handoff \ --contract "$CONTRACT" \ --input var/process-truth/current.json \ --output var/process-truth/handoff.json ``` After a mutation: ```bash python3 "$TOOL" seal \ --contract "$CONTRACT" \ --input var/process-truth/after.raw.json \ --output var/process-truth/after.json python3 "$TOOL" diff \ --contract "$CONTRACT" \ --before var/process-truth/current.json \ --after var/process-truth/after.json \ --output var/process-truth/delta.json ``` Never edit a sealed bundle. Change raw input and seal again. ## 1. Choose one stable subject | Work | Subject kind | Required identity | |---|---|---| | process | `process` | host, boot ID, PID, start ticks, UID | | thread | `thread` | process identity plus TID | | PR admission/merge | `workflow_transaction` | repo, PR, exact head/base, run ID/attempt | | Git/GHCR/TCP transfer | `network_stream` | source, destination, protocol, flow ID | | Longhorn rebuild | `storage_transfer` | volume, nodes, engine, transfer ID | | alert routing | `alert_delivery` | alert fingerprint, receiver, route generation | A changed head, start time, boot ID, transfer ID, or alert fingerprint means a new subject. Do not compare it as one continuing process. ## 2. Declare the observation universe Before collecting, record: ```text selector selector SHA-256 expected count or null with reason observed count termination reason coverage-evidence surfaces ``` A complete bundle cannot self-declare completeness without a complete universe. ## 3. Collect direct surfaces Every surface records: ```text stable ID source owning authority typed status duration per-surface deadline artifact URI digest bytes truncation structured facts ``` Typed statuses: ```text ok timeout timeout_unreaped_helper permission_denied not_found io_error unsupported truncated identity_race provider_ambiguous self_error ``` Missing output is never clean output. ## 4. Process-collection safety Read cheap identity surfaces first: ```text /proc/sys/kernel/random/boot_id /proc//stat /proc//status /proc//ns/* /proc//cgroup ``` Run potentially blocking surfaces in isolated helpers: ```text /proc//task//stack /proc//task//wchan /proc//task//syscall ptrace-based collectors ``` If a helper times out and cannot be reaped: 1. emit `timeout_unreaped_helper`; 2. record helper PID/start ticks, target identity, signal, and timestamp; 3. journal the helper under a lock-protected bounded leak budget; 4. refuse unbounded replacement helpers; 5. continue with partial evidence; 6. never report the target as clean. Re-read target identity before sealing. Any mismatch is `identity_race`. ## 5. Write claims honestly ### Fact A fact is direct or deterministically derived. Confidence must be `null`. ### Absence fact An absence requires negative-control evidence proving the observer can detect a known-positive or planted signal. Bad: ```text No EIO entries were found; therefore the disk is healthy. ``` Better: ```text The bounded kernel-log census found no matching EIO signatures, and the same parser detected the planted control signature. This does not prove all database pages are readable. ``` ### Inference or hypothesis It requires confidence and a falsifier. Preserve it as `falsified` when killed; do not erase it. ## 6. Name unknowns One unknown contains: - one answerable question; - blocked transitions; - one owner; - the cheapest safe probe; - expected discriminator; - deadline; - abort conditions. ## 7. Choose one next transition A bundle has exactly one frontier transition, such as: ```text DISCRIMINATE PUBLISH_SHADOW PUBLISH_APP_ONLY_ADMISSION SHA_FENCED_MERGE TRACE_STREAM_FAILURE TRACE_ALERT_DELIVERY RESTORE_REDUNDANCY SOURCE_FIXED REALIZED VERIFIED CLOSED ``` Define the success receipt before work starts. Without the named receipt, the transition did not complete. ## Programme adapters ### Mode S candidate ```text claim -> build -> execute -> terminal classify -> hold/quarantine/release -> journal -> provider receipt -> consumer acceptance ``` Report the first missing edge. Attribution hold and permanent red quarantine are distinct states. ### PR admission and merge Capture exact head/base, Policy Guards generation, shadow row, app-only admission, review threads/holds, classifier lane, writer eligibility, merge receipt, and post-merge sentinel. “Checks green” is never terminal. ### Longhorn and network Link storage-transfer and network-stream bundles. Capture natural transfers while crown jewels are exposed; do not generate synthetic multi-gigabyte load. CPU/load is a hypothesis surface. ### Alert delivery Trace Prometheus through acknowledgement. A firing alert proves only one edge. ## Handoff discipline Send the compact handoff plus sealed bundle URI and digest. A receiver should not have to trust prose. A valid handoff answers: 1. What exact subject was observed? 2. How fresh and complete is the evidence? 3. Which facts are direct? 4. Which hypotheses remain open, and what kills them? 5. Which unknowns block action, and who owns each probe? 6. What is the one next legal transition? 7. What receipt proves it happened? ## Closure discipline | Claim | Required receipt | |---|---| | source fixed | tests/mutation proof at exact source head | | carrier delivered | merged commit on protected main | | runtime realized | live identity at intended revision | | data safe | store-appropriate restore or integrity proof | | conveyor healthy | natural finalizer + writer conversion and sentinel | | alerting effective | external delivery plus acknowledgement | | incident closed | causal condition absent and recurrence control realized | A stale green check, merged PR, replica count, or process heartbeat cannot stand in for another layer’s receipt.