Datasets

A stack can bundle large, read-only lookup data — a book catalog, airport codes, a GeoIP table — as a SQLite file that deploys WITH the code and is queried locally at runtime. No external API, no separate lookup service.

Authoring

Two files under the reserved DATASETS/ subtree, paired by name:

OPS/<stack>/
  DATASETS/
    books.sqlite    the artifact (any schema; FTS5 supported)
    books.yaml      the named queries

The manifest declares every query the runtime may execute. Nothing else — not even a well-formed SELECT — can run against the artifact, and parameters are always bound:

queries:
  search:
    sql: |
      SELECT b.isbn13, b.title, b.author
        FROM books_fts f JOIN books b ON b.rowid = f.rowid
       WHERE books_fts MATCH ?
       ORDER BY rank
    max_rows: 10          # optional; tightens the node cap, never widens
  by_isbn:
    sql: SELECT * FROM books WHERE isbn13 = ?

Querying

WHEN @web.req.url.query.q.0 != ""
  EXEC "txco://dataset"
    WITH dataset = "books",
         query = "search",
         args = &array(@web.req.url.query.q.0)

The result lands at into (default _dataset, private to the flow):

{
  "_dataset": {
    "dataset": "books",
    "query": "search",
    "rows": [
      {
        "isbn13": "…",
        "title": "…"
      }
    ],
    "count": 1,
    "truncated": false
  }
}

Optional WITH params: args (positional binds), limit (tightens the row cap), stack (read another of your tenant’s stacks), into. Errors arrive as dataset.error.{code,message} — handle with WHEN @dataset.error. Codes: txco_dataset_not_found, txco_dataset_unknown_query, txco_dataset_invalid_arg, txco_dataset_missing_artifact, txco_dataset_store, txco_dataset_query.

What apply does

txco apply hashes each artifact by streaming (never in memory), asks the chassis HEAD /blobs/sha256/{hash}, streams the bytes only when missing, and references them from the version as a fingerprint row. Unchanged artifacts cost one HEAD. txco pull streams them back down the same plane, hash-verified.

Activation is the gate: the artifact must be in the content-addressed store, the manifest must parse, and every declared query must PREPARE read-only against the shipped schema. A typo’d column or a sneaky DELETE fails the deploy — with the file and query named — not the request path.

Runtime model

Artifacts are immutable (the reference is their content hash), so each node opens them read-only + immutable: no locks, no journal, shared across requests. Enforcement is layered: only declared queries run; the connection is query_only; and a SQLite authorizer default-denies everything but reads, so even a write that somehow reached preparation dies with “not authorized”.

Nodes fetch an artifact from the content-addressed store on first use and keep it in a local disk cache (fleet nodes prefetch on activation). When the store is the bundled local-disk backend the artifact is opened in place — zero copies at any size.

Limits

knob	default	what it bounds
--dataset-max-file-bytes	4 GiB	artifact size at upload + activation
--dataset-max-rows	200	rows per query (manifest max_rows and WITH limit clamp under it)
--dataset-cache-bytes	4 GiB	node-local materialise cache (LRU)
--dataset-cache-dir	./chassis/data/datasets	where cached artifacts live

Responses are additionally capped at 1 MiB of rows (truncated: true when hit). Queries run under the ordinary per-op timeout (WITH timeout raises it, up to the node max).

Superseded artifact versions currently stay in the content-addressed store (no garbage collection yet); at multi-GB scale, expect storage to grow with every changed-artifact deploy.

A worked example lives at examples/dataset-lookup/.