Secret store — operator runbook — thanks, computer. docs

A practical guide for operating the per-tenant secret store. This doc is the day-2 manual: how to bring the feature up, how to manage secrets, what to do when something goes wrong.

TL;DR

Bootstrap is automatic: the chassis mints a master key on first boot at ./chassis/data/secrets/txco-master.key (--secret-master-key to relocate).
Manage secrets via CLI: txco auth tenant secrets {set, generate, list, show, describe, rotate, revoke}. Operator-supplied values come from a TTY prompt; chassis-generated values are printed exactly once.
There is no reveal command. To inspect a value, rotate the secret. Both rotate (with a new operator value) and rotate --generate (chassis mints) show you the value once.
Back up the master-key file separately from the runtime DB. Losing it makes every stored secret permanently unrecoverable.

1. Bootstrap

The secret store auto-bootstraps on first chassis boot — same convention as the runtime DB. No explicit setup step required for the default path. The chassis mints a 32-byte master key at ./chassis/data/secrets/txco-master.key (or wherever --secret-master-key points) the first time it doesn’t find one there. On first mint you’ll see this in the logs:

INFO  secret store: minted new master key — BACK THIS UP; losing it makes every stored secret unrecoverable  path=…/txco-master.key
INFO  secret store enabled  path=…/txco-master.key key_version=1

On every subsequent boot the existing key is loaded and only the second line appears.

Where the file lands

Scenario	Default path
`txco serve` (production)	`./chassis/data/secrets/txco-master.key`
`txco dev` (local dev)	`<workspace>/.txco/dev/secrets/txco-dev-master.key` (gitignored)
Explicit override	`--secret-master-key /your/path` (or `TXCO_SECRET_MASTER_KEY=…`)
Library / embedder opt-out	Set `SecretMasterKeyPath` to empty string

For production, point the flag at an operator-owned root such as /data/secrets/txco-master.key. The auto-mint logic creates any missing parent directories with 0700 perms; the key file itself is 0600.

Explicit init (rare — only when you want a different path before first boot)

# Production: pre-mint at the production path before first chassis
# boot, so the operator chooses the location deliberately.
txco auth secrets init --path /data/secrets/txco-master.key

init is also the verb for forced rotation of an existing key (see §6 — disaster). It refuses to overwrite unless --force is passed, and --force then prompts for an “overwrite” confirmation. init refuses any path that points at a directory — it suggests the canonical filename instead.

If the load fails

If the configured path exists but is malformed (wrong perms, wrong size), you’ll see a WARN and the chassis boots with the secret store off:

WARN  secret store disabled: master key load failed  path=… err=…

The data plane stays up; any op declaring secrets.* in its WITH clause then fails loud with secret_store_unavailable.

Verify

# Generate a probe secret. The value is printed once on stdout;
# anything but a 43-char base64-url string means something's off.
txco auth tenant secrets generate PROBE_KEY --tenant default

# Confirm metadata visible:
txco auth tenant secrets list --tenant default

# Cleanup (the name frees for re-creation):
txco auth tenant secrets revoke PROBE_KEY --tenant default

2. Manage secrets

All operator-facing CRUD is under txco auth tenant secrets. See --help on any verb for full flag reference.

Store a vendor-supplied value (the 99% case)

The operator already has a value (a Stripe key from the Stripe dashboard, an OAuth client secret, etc.). set prompts for it via TTY hidden input — it never appears on the command line, in shell history, or in ps output.

# Hidden TTY prompt; the value is never on the command line.
txco auth tenant secrets set STRIPE_API_KEY \
  --tenant acme \
  --description "Stripe live key, rotated 2026-05-20"

If a secret by that name already exists, set rotates it (writes a new version) rather than failing. The old version row is preserved in tenant_secret_versions for audit history; the resolver only ever sees the latest.

Mint a fresh value (chassis-generated)

When you don’t have a vendor-issued value — e.g. an HMAC signing secret you’re about to share with a webhook — let the chassis mint one. The value is printed once to stdout; capture it immediately or rotate to see it again.

txco auth tenant secrets generate WEBHOOK_HMAC \
  --tenant acme \
  --byte-len 32 \
  --description "Stripe webhook signing"
# stdout: jGWvsjCkKWXq0irVAZlSXNp1-qxQDh_yKpjzkrOKLVk

Format: base64-url no-padding. 32 bytes → 43 chars. Adjust --byte-len for longer keys; max 4096.

Rotate

# Operator-supplied new value (TTY prompt; no shell history):
txco auth tenant secrets rotate STRIPE_API_KEY --tenant acme

# Chassis mints a new random value, prints once:
txco auth tenant secrets rotate WEBHOOK_HMAC --tenant acme --generate

After rotate, the active version of the secret is the new one. Old encrypted versions stay in the DB for audit; the resolver routes all reads to the latest.

Inspect (metadata only — value never shown)

# List active secrets in a tenant:
txco auth tenant secrets list --tenant acme

# Show metadata for one:
txco auth tenant secrets show STRIPE_API_KEY --tenant acme

# Update description without rotating the value:
txco auth tenant secrets describe STRIPE_API_KEY \
  --tenant acme \
  --set "Stripe live key — rotated by alice on 2026-05-20"

There is no reveal command. Per design, the value never leaves the chassis once stored — to inspect it, rotate it (the rotate path shows the new value once).

Revoke

txco auth tenant secrets revoke STRIPE_API_KEY --tenant acme

Soft-delete: the row is marked revoked_at = now; the encrypted versions stay in the DB for audit. The (tenant, stack, name) slot is freed — you can immediately re-create a secret with the same name (it gets version_no = 1 as a fresh identity).

3. Reference secrets from txcl ops

In a txcl rule’s WITH clause, reference a secret by name. The chassis materializes the cleartext into the op handler’s private buffer at execution time; the value never enters op.Input, trace events, mock fixtures, continuations, or logs.

Templated header (the 90% case)

EXEC "https://api.stripe.com/v1/charges"
  WITH secrets.headers.authorization.secret = "STRIPE_API_KEY",
       secrets.headers.authorization.format = "Bearer {}",
       method = "POST"

The format template has exactly one {} placeholder; the materialized cleartext fills it. Bearer tokens, GitHub token <t> legacy, Vendor-API custom prefixes — all fit this shape.

Raw substitution (no format)

EXEC "https://api.vendor.com/things"
  WITH secrets.headers.x-api-key.secret = "VENDOR_KEY"

Body field

EXEC "https://api.partner.com/auth"
  WITH secrets.body.client_secret.secret = "PARTNER_OAUTH_SECRET"

The path under secrets.* mirrors the outbound request — headers.X sets HTTP header X; body.X.y.z overlays the JSON body at that path.

Computed (HMAC, JWT, etc.)

For computed credentials (HMAC over a body, JWT signing, base64(user:pass) for Basic auth), use a separate signing op followed by a normal request:

# Step 1 — compute the HMAC. op.Secrets consumed here only.
EXEC "txco://hmac-sign"
  WITH secrets.key.secret = "STRIPE_WEBHOOK_SECRET",
       algorithm   = "sha256",
       input_path  = "body",
       output_path = "_txc.computed.stripe_sig"

# Step 2 — make the call. The digest is a normal envelope value.
EXEC "https://api.stripe.com/v1/webhook-callback"
  SET @web.req.headers.stripe-signature = @_txc.computed.stripe_sig
  WITH method = "POST"

Two computed-secret ops ship with the chassis:

txco://hmac-sign — HMAC-SHA256/SHA512, hex or base64 digest.
txco://basic-auth-encode — base64(user:password) for HTTP Basic.

Custom signing schemes: register your own op handler that reads op.Secrets via secrets.BagFromContext(ctx).

4. Capabilities

Two capabilities gate secret-store admin endpoints:

Capability	Permits
`secret:*:read`	List secrets, read metadata. Never the value.
`secret:*:write`	Create, generate, rotate, `rotate --generate`, describe, revoke.

Op-time materialization (the data-plane path that fills op.Secrets) is not gated on per-actor capabilities — it’s the chassis acting on the tenant’s behalf inside the tenant’s own request scope.

5. Audit log

Admin actions emit structured logs at info. Op-time materialization logs at debug (per-request frequency makes info too noisy and risks publishing business behavior). For observability, consume the chassis.secret.materialize metric — incremented per secret reference with labels txco.tenant.slug and txco.secret.name.

INFO  secret_action  actor_id=actor_abc tenant_id=tnt_xyz secret_name=STRIPE_API_KEY action=rotate outcome=ok

Grep tokens: secret_action for admin events; secret_materialize for op-time (debug). Value bytes are never logged at any level.

6. Disaster: master-key loss

This is the catastrophic failure mode. If /data/secrets/txco-master.key is lost or corrupted, every secret in the store becomes opaque ciphertext that no one can decrypt. There is no in-chassis recovery path.

Mitigation (preventive)

Back up the master-key file separately from the runtime DB. Different disk. Different access controls. If the runtime DB and the master key are on the same backup volume, you have a single point of compromise — the attacker who steals the backup steals both.
Document the file’s location in your operator-side disaster-recovery doc. The chassis doesn’t self-document where the file lives; only the operator knows.
Encrypt the backup at rest with a different key/passphrase. GPG-encrypted to a hardware-backed key, AWS KMS, etc. The master key is itself a credential that protects credentials; layer accordingly.

Recovery (after loss)

If the master-key file is irretrievably lost:

Accept that the existing store is gone. Encrypted ciphertexts in tenant_secret_versions are now binary noise. There is no way to recover the cleartexts.
Plan vendor-secret rotation. Every stored secret needs to be re-issued by its source: rotate Stripe key in the Stripe dashboard, re-mint OAuth secrets, regenerate webhook signing keys with each vendor, etc. This is the same exercise as any credential-compromise incident.
Mint a new master key with `txco auth secrets init —path `. Restart the chassis.
Truncate the abandoned ciphertexts (or leave them — they’re inert without the old MK; the unique-index WHERE clause excludes revoked rows, so re-creating with the same names just works):
```
DELETE FROM tenant_secret_versions;  -- optional cleanup
DELETE FROM tenant_secrets;           -- (or UPDATE … SET revoked_at)
```
Re-create each secret via txco auth tenant secrets set … with the newly-issued values from step 2.

This is operationally painful by design. The alternative — chassis auto-backup of the master key — would create a second leak surface elsewhere; the explicit responsibility is the better trade.

8. Quick reference

Action	Command
Mint master key (explicit; rarely needed — auto-mints on first boot)	`txco auth secrets init --path /data/secrets/txco-master.key`
Store operator value	`txco auth tenant secrets set NAME --tenant T` (TTY prompt)
Mint random value	`txco auth tenant secrets generate NAME --tenant T --byte-len 32`
List	`txco auth tenant secrets list --tenant T`
Show metadata	`txco auth tenant secrets show NAME --tenant T`
Rotate (operator value)	`txco auth tenant secrets rotate NAME --tenant T`
Rotate (chassis mints)	`txco auth tenant secrets rotate NAME --tenant T --generate`
Update description	`txco auth tenant secrets describe NAME --set "new" --tenant T`
Revoke	`txco auth tenant secrets revoke NAME --tenant T`

Capability	Allowed actions
`secret:*:read`	list, show
`secret:*:write`	set, generate, rotate, `rotate --generate`, describe, revoke