Document outputs

One canonical model, structure intact

The parsed document is the protobuf message ai.pipestream.document.v1.Document, and it is the same shape whether the source was a born-digital PDF, a scanned page, a spreadsheet, or a slide deck. The body is a tree of groups and references (each text item addressable as something like #/texts/17); the furniture (running headers, footers, page numbers) lives on its own layer so it never contaminates the reading flow; and every item carries its provenance: page number, bounding box, and the source that produced it.

grpc-services/gRParse document.proto, the Document message

message Document {
  DocumentOrigin origin = 4;      // mimetype, binary hash, filename, uri
  GroupItem furniture = 13;       // headers, footers, page numbers
  GroupItem body = 5;             // main structure, refs like #/texts/17
  repeated BaseTextItem texts = 7;
  repeated PictureItem pictures = 8;
  repeated TableItem tables = 9;
  map<int32, PageItem> pages = 12;
}

The origin block alone explains why downstream caching works: the source file's hash is stamped on the document itself, so two parses of the same bytes are recognizably the same document, and a re-ingest of identical content dedupes instead of duplicating.

message ProvenanceItem {
  int32 page_no = 1;
  BoundingBox bbox = 2;        // l / t / r / b plus coord_origin
  IntSpan charspan = 3;        // character span inside the item's own text
  optional TimeSpan time = 4;  // position in a media timeline
}

message BoundingBox {
  double l = 1;  double t = 2;  double r = 3;  double b = 4;
  optional CoordOrigin coord_origin = 5;  // gRParse always emits COORD_ORIGIN_TOPLEFT
}

Provenance is plural on purpose. An item may know its page and box, its span in a media timeline, or its byte range in an archive member, and each claim names its source: a CollectorSource records which collector produced the item (for example grparse or libreoffice), which engine inside it (for example rapidocr or poppler-text), and the engine's confidence when it reports one. Sources never overwrite each other; choosing a winner is left to the consumer.

Same document, many faces

One request names the formats it wants, and the response carries every one of them, rendered from the same merged document. The wire enumerates nine: TEXT, MARKDOWN, HTML, HTML_SPLIT_PAGE, JSON, YAML, DOCTAGS, DOCLANG, and VTT. DOCTAGS and DOCLANG are Docling-compatible formats for tools that speak that dialect; an empty format list keeps the plain-text default. Below, one small field note is rendered four ways. Flip the tabs and notice what each face keeps and what it throws away.

{
  "text": "Impeller vibration stayed inside the normal band for the full run.",
  "label": "DOC_ITEM_LABEL_TEXT",
  "prov": [{
    "pageNo": 1,
    "bbox": { "l": 96, "t": 210, "r": 520, "b": 228,
           "coordOrigin": "COORD_ORIGIN_TOPLEFT" }
  }],
  "source": [{ "collector": {
    "collector": "grparse", "model": "poppler-text"
  } }]
}

<article>
  <h1>Pump station 4: weekly check</h1>
  <p>Impeller vibration stayed inside the normal band
     for the full run. Bearing temperature held at 41 C.</p>
  <ul>
    <li>Intake screen: clear</li>
    <li>Seal flush: no weep</li>
  </ul>
  <table>
    <tr><th>Reading</th><th>Value</th><th>In limit</th></tr>
    <tr><td>Vibration</td><td>2.1 mm/s</td><td>yes</td></tr>
    <tr><td>Bearing temperature</td><td>41 C</td><td>yes</td></tr>
  </table>
</article>

WEBVTT

00:00.500 --> 00:05.000
Pump station 4: weekly check

00:05.500 --> 00:12.000
Impeller vibration stayed inside the normal band
for the full run. Bearing temperature held at 41 C.

00:12.500 --> 00:16.000
Intake screen: clear. Seal flush: no weep.

Offsets that never renumber

Every face above shares one invisible property: the text offsets are stable. The platform's offset contract says that the assembled plain-text form of a document joins its text items with a single newline, in reading order, and that a character position in that string means the same thing on every machine. Offsets are Unicode codepoint positions, not bytes, and they are append-only: when a document streams page by page, a later page never renumbers an earlier one. Item references such as #/texts/17 are stable from the moment they are emitted.

The geometry side is equally fixed. Every provenance box lives in one shared coordinate space: origin at the top-left, y growing downward, integer pixels on the 200 DPI render raster, with the page's own rotation already applied. Each box says so explicitly (coord_origin = COORD_ORIGIN_TOPLEFT), and consumers are expected to assert on it rather than assume. This is what makes search highlights and semantic heatmaps possible: a hit at character range 340 to 355 maps back to a rectangle on page 2, in any rendering.

Chunks you can cache on

Downstream systems rarely want a whole document at once; they want chunks for indexing and embedding. The platform's two chunk operations (hierarchical and hybrid) parse the source exactly the way a conversion does and then split the resulting document. The hierarchical walker makes one chunk per item or list group in body-tree order, carrying the heading trail; the hybrid walker adds peer merging under a token budget and a sentence-wise split, and it requires an explicit max_tokens.

Determinism is the design goal: the same input bytes produce the same chunk bytes on every machine and every run. There is no tokenizer download, no locale dependence, and no defaulted budget; every boundary rule is versioned, and each chunk carries the version it was produced under in rules_digest (for example grparse-hier/1, or grparse-hybrid/1;tok=wordish/1;sent=sentence/1;max_tokens=N;merge_peers=B). Change a rule and the digest changes, which is exactly how a corpus knows it must re-chunk.

grpc-services/gRParse parse_types.proto and the chunking README, one chunk illustrated

{
  "text": "Impeller vibration stayed inside the normal band ...",
  "headings": ["Pump station 4: weekly check"],
  "pageNumbers": [1],
  "startOffset": 42,          // UTF-8 codepoint positions in the
  "endOffset": 108,           // document's concatenated body text
  "rulesDigest": "grparse-hier/1"
}

A chunk reports start_offset and end_offset as codepoint positions in the document's concatenated body text, but only when the parse supplied an offset table for every text item the chunk consumed; otherwise both stay unset rather than guessed. Combined with byte-for-byte determinism, that means a content hash over a chunk is a stable identity: embed it, index it, or store it, and the same content hashes the same way tomorrow, so caches hit and duplicates are detected by comparing hashes instead of re-reading text. This is the same hash-first discipline as the rest of the document lane, where intake derives the document id from the payload's SHA-256 and the repository dedupes on content. Chunks feed the search lane from here: protomolt-search indexes and embeds them, and two indexes agree on their chunk boundaries and vector spaces exactly when their chunking-policy digests agree.

Metadata stays typed, per source family

A PDF, an office file, an XML document, and a web page each declare different things about themselves, and the model keeps those declarations typed instead of flattening everything into string pairs. One DocumentMeta message carries the shared fields (title, authors, creation and modification instants) plus the fields particular to each family: format_version and a structured flag for PDFs, template and editing-cycle counts for office files, namespace bindings and schema locations for XML sources, typed identifiers and classification codes for scholarly files. Dates keep a parsed form and a _raw twin holding the source's own spelling, and the source's embedded XMP packet is kept verbatim when present.

grpc-services/gRParse document.proto, Identifier: one typed metadata field

message Identifier {
  string kind = 1;            // doi, pmid, issn, publisher-id
  string value = 2;
  optional string scope = 3;  // electronic vs print ISSN
}

The rule that keeps this accountable is short: data whose shape the fleet knows gets a typed field, never an entry in the open-vocabulary extra map. When several collectors contribute metadata, FieldSource records which collector's answer each resolved field carries, so a value that lost the resolution is still attributable on the wire.