html-extract

A web page can’t be merged into a flow: an EXEC "https://…" answer must be JSON. This op does the reading for you, with the fetch and the parse both bounded by the chassis. To read the page itself, as Markdown in parts, see html-read: the same fetch and bounds, and they share one tenant allowance.

Read a page

WITH url = ._link,
     selectors = &object(
       "title",       &object("selector", "title", "text", true),
       "description", &object("selector", "meta[name='description']", "attr", "content"),
       "image",       &object("selector", "meta[property='og:image']", "attr", "content"),
       "headings",    &object("selector", "h1, h2", "text", true, "all", true, "limit", 20)),
     into = "_page"
EXEC "txco://html/extract"

The answer lands at into (default _html):

{
  "ok": true,
  "url": "https://example.com/article",
  "final_url": "https://example.com/article",
  "status": 200,
  "content_type": "text/html; charset=utf-8",
  "data": {
    "title": "Example Article",
    "description": "An interesting article",
    "image": "/images/cover.jpg",
    "headings": [
      "Introduction",
      "Background",
      "Conclusion"
    ]
  }
}

• data uses your field names, in the order you named them.
• final_url is where any redirects ended.
• content_type is the header as the server sent it.
• truncated: true appears when a value was cut to 4 KiB.

A failure is an answer too, never a dropped op:

{
  "ok": false,
  "url": "https://example.com/gone",
  "content_type": "text/html",
  "error": {
    "code": "txco_html_http_error",
    "message": "the server answered 404",
    "status": 404
  }
}

content_type is there whenever a response arrived. Branch on ._page.ok in a later scope.

Selectors

selectors is an object of named fields. Each field is {selector, text: true | attr, all?, limit?}:

Key	Default	Meaning
selector	required	A CSS selector, at most 512 bytes.
text	false	The element’s text, with whitespace collapsed. Script, style and template content is left out.
attr	—	One attribute’s value, exactly as written in the page.
all	false	Every match, as an array in document order. Without it, the first match.
limit	10	The most matches all returns, from 1 to 50.

Set exactly one of text: true and attr. You can name at most 20 fields. A field name is letters, digits, _ and -, starting with a letter or _.

When nothing matches:

• a single field is null;
• an all field is [].

An element that matches but lacks the attr is skipped, so the first element that has it answers.

Attribute values are returned as written. A relative href or src stays relative: resolve it yourself against final_url, or against the page’s <base href>, which you can also extract.

What a selector may use:

• type, #id, .class and attribute selectors ([a], [a=v], ~=, |=, ^=, $=, *=, !=, and the i flag);
• *;
• the > and + combinators;
• one descendant space per selector;
• :not(…), :first-child, :last-child and :root;
• comma-separated alternatives.

Refused with txco_html_invalid_selectors:

• :has(), :contains() and :matches();
• :nth-* and :*-of-type;
• the ~ combinator;
• a second descendant space (write main > article h2, not body main article h2);
• [a#=regex].

Each of these costs more than a fixed amount per element on a hostile page, and the chassis can’t stop a match once it has started. An invalid selector is an error, never an empty result.

How the page is fetched

The fetch is narrow on purpose:

• The request: one GET of an http or https URL, on port 80 or 443, with no credentials in it. It carries no cookies, no authentication, no request body and no headers of yours. The User-Agent is txco-extract/<version> (+https://www.thanks.computer).
• Destinations: the host is resolved by the chassis, and every address is checked by the egress policy (--egress-policy; private on a fleet) before it is dialed. That rules out loopback, private networks, link-local and cloud-metadata addresses. The address checked is the address connected, so a DNS answer that changes between lookups can’t slip a private address in.
• Redirects: up to 3, each checked the same way, scheme and port included.
• Size and time: at most 3 MiB of body (--html-extract-max-bytes) and 5 seconds for the whole fetch (--html-extract-timeout; your op’s deadline can only shorten it). A larger body fails; nothing is extracted from a partial page.
• Type: the response must be 2xx, and its media type text/html. The op asks for an uncompressed body; a server that sends gzip anyway is decoded, within the same 3 MiB.
• Encoding: the charset comes from the header, else from the page’s <meta charset>, else UTF-8. The values come back as UTF-8.

Errors

Every code is txco_html_<code>:

Code	Meaning
invalid_url	Not http(s), credentials in it, a port other than 80/443, no host; or a redirect to such a URL
invalid_selectors	A field is malformed, or its selector does not parse or uses something refused (the message names the field)
destination_denied	The host resolves only to addresses the egress policy refuses
dns_failed	The host does not resolve
connection_failed	The connection or TLS handshake failed
timeout	The fetch or the extraction ran out of time
redirect_limit	More than 3 redirects
http_error	A non-2xx answer; error.status holds it
unsupported_content_type	Not text/html, or no Content-Type
unsupported_encoding	A Content-Encoding other than gzip
response_too_large	The body is over the byte cap
document_too_complex	The page exceeds a document limit below
parse_failed	The page could not be decoded or parsed
output_limit_exceeded	The values together are over 64 KiB
rate_limited · busy	This tenant’s limits on this node, below
no_tenant · internal_error	The op ran outside a request, or failed unexpectedly

Messages never contain an address or a connection detail.

Limits

Limit	Value
Fields	20
Selector length	512 bytes
Matches per all field	50
One value	4 KiB, cut at a character boundary, with truncated: true
All values together	64 KiB, else output_limit_exceeded
Page body	3 MiB
Tokens in the page	600,000
Nodes in the parsed page	400,000
Nesting	512
Attributes on one element	256
Parsing and matching	1 second of CPU

The document limits fit a dense 3 MiB page of ordinary markup: about 530k tokens and 380k nodes, about 150 ms and 70 MB on a laptop. A page built to be expensive is refused before it costs that much.

One check applies before the page is parsed. The HTML parsing algorithm recreates every unclosed <b>, <i> and <font> in each new block, so a few kilobytes can grow into millions of nodes. The op estimates the nodes the parser will create, and refuses a page whose estimate is too high.

Per node, across tenants:

Flag	Default	Limit
--html-extract-concurrency	16	Fetches in flight. A call waits for a slot until its deadline.
--html-extract-parse-concurrency	2	Parses at once. Each can hold about 70 MB and a second of CPU.
--html-extract-tenant-concurrency	4	Calls one tenant may have in flight. One more is busy.
--html-extract-rate-per-min	60	Calls one tenant may start a minute. More are rate_limited.

Fuel

Each call pays:

• the EXEC: 25;
• the fetch: 50 more;
• bytes downloaded: 100 per started MiB;
• parsing and matching: 10 per started millisecond, the nano-op compute rate.

A 100 KB page that parses in 5 ms costs about 235; a dense 3 MiB page, about 2,000. See fuel.

What comes back, and what doesn’t

• The values are the page’s words, not yours: treat them as untrusted text. The op writes only to into, never to _txc.
• The URL can carry data, as with EXEC "https://…". A rule that builds the URL from what it has read sends that to whoever runs the host. Which URLs a stack may read is the stack’s decision, as it is for any outbound call.
• The trace records the answer, including url and final_url. If a URL can hold something private, mask it with WITH redact (see trace). The op’s own timeline event, html.completion, records only the host, a short hash of the URL, the status, sizes and times.

Known limits

• No JavaScript. A page that builds itself in the browser has nothing to read in its HTML.
• No pretending to be a crawler. Some sites only send their og:/twitter: tags to the big crawlers’ user agents; for those, the op sees what any browser without JavaScript sees.
• The fetch leaves from the node’s own address. A site that blocks hosting providers’ addresses blocks it.

A link preview

Ask for every source of a title, description and image, and pick in a later step: og: first, then twitter:, then the plain tags. Resolve the image against base or final_url.

WITH url = ._link,
     selectors = &object(
       "title",          &object("selector", "title", "text", true),
       "description",    &object("selector", "meta[name='description']", "attr", "content"),
       "og_title",       &object("selector", "meta[property='og:title']", "attr", "content"),
       "og_description", &object("selector", "meta[property='og:description']", "attr", "content"),
       "og_image",       &object("selector", "meta[property='og:image']", "attr", "content"),
       "og_site_name",   &object("selector", "meta[property='og:site_name']", "attr", "content"),
       "tw_title",       &object("selector", "meta[name='twitter:title']", "attr", "content"),
       "tw_image",       &object("selector", "meta[name='twitter:image']", "attr", "content"),
       "canonical",      &object("selector", "link[rel='canonical']", "attr", "href"),
       "icon",           &object("selector", "link[rel~='icon']", "attr", "href"),
       "base",           &object("selector", "base[href]", "attr", "href")),
     into = "_page"
EXEC "txco://html/extract"

The order of preference and the URL resolution are a few lines in a nano-op. Cache the result in kv with a TTL if the same links come up often.