Repo: the document store

One document, four parts

A stored Document is not one blob. The codec splits it on clean protobuf field boundaries into four parts, and each part is its own object under a shared prefix: core.pb, blobs.pb, chunks/<subKey>.pb, parsed.pb. Every part gets a SHA-256 entry in a manifest. A reader names the parts it wants, and the full document reassembles byte-for-byte from the fragments, merged in manifest order. The round-trip tests compare bytes to prove it.

The split is descriptor-driven, not hardcoded to Document: the codec reads a PartLayout that says which field is the identity, which fields form which part, and which part is chunked and sub-keyed. Any protobuf message type can be decomposed the same way. Click through the four parts below to see what each one holds and when it is fetched.

repo container codec, the canned layout for Document

return PartLayout.builder(Document.getDescriptor())
        .identityField("doc_id")
        .partField(DocumentPart.DOCUMENT_PART_BLOBS, "blob_bag")
        .partField(DocumentPart.DOCUMENT_PART_PARSED, "parser_results")
        .chunkedPart(DocumentPart.DOCUMENT_PART_CHUNKS,
                "search_metadata.semantic_results", "result_id")
        .build();

Two companion mechanics fall out of the split. A partial save writes only the listed parts and carries the rest forward from a source address with a server-side object copy, so the unwritten parts move between addresses without their bytes ever transiting the service. A partial read assembles only the requested parts, and GetDocumentManifest reports locations, hashes, sizes, and states without fetching a single byte. Each manifest entry is also stamped with WriteProvenance (module, node, graph, version); carried-forward parts keep their original stamp, and a blank stamp means unknown, never invented.

Identity: four segments, one deterministic UUID

A document's identity is not a random id and not the raw doc_id string. It is a name-based UUID (the v5 style) derived from four logical segments, together called a NodeAddress: the document id, the graph address id, the owning account id, and the graph id. The graph segment matters because two independent pipelines can give different nodes the same name; including the graph id keeps their stored rows distinct, so one pipeline finishing its own work can never disturb another's in-flight copy.

The raw doc_id never reaches an object key. Keys are built from the deterministic UUID, so re-saving the same logical document lands on the same identity every time. Identity is never random; re-saves are idempotent by construction.

// The storage identity of one stored document state: the four logical
// segments the deterministic node id (UUIDv5) is derived from. This is THE
// way to address a row; node_id strings are derived echoes, never truth.
message NodeAddress {
  string doc_id = 1;
  string graph_address_id = 2;  // datasource id at intake, graph node id in a pipeline
  string account_id = 3;        // owning account (tenant root)
  string graph_id = 4;          // "intake:<accountId>" at intake, owning graph id in a pipeline
}

The ledger enforces the model in SQL. The four segments form a unique identity, blank graphs are unrepresentable, and a row is always in exactly one of three statuses:

repo service, the document identity rules

CONSTRAINT uq_documents_identity UNIQUE (doc_id, graph_address_id, account_id, graph_id),
CONSTRAINT chk_documents_status CHECK (status IN ('AVAILABLE', 'PENDING_PURGE', 'PURGE_FAILED'))

Re-uploading the same bytes costs nothing

When a document is saved again and the split root checksum matches the checksum of the existing AVAILABLE row, the service skips the object writes entirely, marks the row re-processed, and answers deduplicated=true with the existing coordinates. The caller learns the document is already there; no bytes move.

A caller that wants a real rewrite anyway passes force_save, and the write becomes store-verified: the SHA-256 rides the PUT as a checksum trailer, and the object store itself rejects any landed bytes that do not match the digest. A corrupted write cannot be mistaken for a stored document.

repo blob store, the checksum trailer on a verified PUT

// SDK checksum trailer: the store compares the landed bytes against
// this digest and fails the PUT on mismatch (verified write).
b.checksumSHA256(Base64.getEncoder().encodeToString(HexFormat.of().parseHex(spec.sha256Hex())));

The same digest runs through the HTTP upload route, which is the door for big payloads: the request body streams through a SHA-256 DigestInputStream straight into object storage with no in-memory buffering. Content-Length is required (the contract answers 411 without it), and the response is a small JSON receipt. A blank doc_id derives a name-based UUID from the content hash itself.

repo README: the upload receipt

{
  "node_id": "3f8a…",
  "doc_id": "…",
  "deduplicated": false,
  "size_bytes": 8388608,
  "sha256": "…",
  "storage_ref": { "drive_name": "intake", "object_key": "intake/blobs/<accountId>/<blobId>.bin" }
}

Drives: the only door to object storage

A Drive binds four things: a bucket, a key prefix, a region, and a credentials reference. The credentials themselves are never in any message; the reference is resolved at IO time. Every account is provisioned with two drives at creation: an intake drive for staged source material and a pipeline drive for the claim-check bodies passed between hops. A custom drive type exists for anything else.

The house rule is strict: nothing but repo-service speaks S3. Drives are the only thing that resolves to object-storage coordinates, so everyone else in the platform sees drives and FileStorageReferences and nothing more. Swap the backing store (MinIO, LocalStack, SeaweedFS, Ceph, rustfs, AWS itself) and only the drive config changes. A Redis-backed store and a read-through cache decorator implement the same internal BlobStore port, and a dogfood client can even use another repo-service as its byte store over gRPC. The Redis-backed store is a sample-grade implementation.

Objects live under per-account, per-node prefixes: <drive.prefix>/documents/<accountId>/<nodeId>/… for parts, <drive.prefix>/blobs/<accountId>/<blobId>.bin for staged uploads, and <drive.prefix>/blobs/<uuid-of-sha256> for content-addressed puts, where identical puts land idempotently on one object.

Delete is two-phase, and the purge ledger makes it safe

The object write and the ledger row that references it are not committed atomically, and that is a deliberate design choice, not an oversight. The standing rule that makes it safe: an object with no live ledger row that owns it is an orphan, and orphans are reclaimable. Deletes, the purge path, and a set of lifecycle loops are all built around that rule.

A normal delete (purge_storage=false) runs in two phases. Phase A happens synchronously inside the gRPC call; Phase B happens later, asynchronously. Step through it:

flowchart LR
          avail["AVAILABLE"] -- "Phase A: tombstone + enqueue purge record" --> pend["PENDING_PURGE"]
          pend -- "staleness guard: row revived or updated" --> void["VOID: nothing deleted"]
          pend -- "Phase B: keys deleted, row removed" --> purged["PURGED"]
          pend -- "10 failed attempts" --> failed["PURGE_FAILED: operator territory"]

Around the same rule sit three more lifecycle machines, each covered by integration tests. A sweeper rescans for rows stuck in PENDING_PURGE whose purge record never got enqueued (a crashed Phase A) and enqueues one. A storage reconciler walks the bucket listing against what the ledger owns and reports, or deletes, orphaned objects older than an hour (dry-run by default). A coherence probe works the other direction, probing a sample of manifest keys and tombstoning confirmed-missing objects in the manifest while the row stays available. These are library calls today; a slow opt-in loop can run them periodically.

The archive: the same engine as a generic asset store

The ArchiveService is the repository family's general-purpose store, served by the same binary and running on the same drives, blob stores, and Postgres ledger. Its hierarchy is account → archive → entry → version → rendition, where a rendition is any one form of an entry: the raw file, a parsed protobuf, markdown, NDJSON, parquet, anything.

Identity is one deterministic name-based UUID over (account_id, archive, entry_id), which is simultaneously the ledger primary key, the object-key prefix, and the natural partition key for events. Content addressing is entry-local: a rendition object's key is derived from its own content hash, scoped under its entry:

Object layout on the drive: an archive rendition

<drive.prefix>/archive/<accountId>/<archive>/<entryUuid>/<rendition>[<subKey>]/<sha256>

An unchanged rendition across versions is one shared object, so retained history costs only what actually changed, and every deletion question stays bounded to one entry's own manifests. Deliberately, this is not a global content-addressed store, which would need reference-counted garbage collection. Object-key segments admit [A-Za-z0-9._-] only, because signature canonicalization must not diverge on exotic bytes.

Each archive picks a versioning policy. VERSIONING_NONE keeps one retained state, and the version counter doubles as the optimistic-concurrency token. VERSIONING_RETAINED keeps every save as a new immutable version. Renditions have states too: PRESENT, EMPTY, or DELETED, and a deleted rendition keeps its size, its hash, and a named reason. The map still lists the file, the file is gone, and the hash proves what it was, which is what makes redaction and retention auditable.

Every entry carries a classification state: UNCLASSIFIED, DECLARED, IDENTIFIED, VERIFIED, or CONFLICTED, never a silent default. A save can declare a FormatFact, validated against the format's own rules (a .zip filename can never claim tar), and ClassifyEntry re-reads the primary rendition's bytes through the asset family's detection seam. Archive stats are exact and adjusted in the same transaction as the mutations they describe.

There are three upload doors, and the contract publishes no capability the service does not implement: a unary PutEntry for small and medium payloads in memory, a client-streaming UploadRendition (declare the size, optionally the expected SHA-256; with a declared hash the bytes stream straight to their final content-addressed key, without one they stage and settle by server-side copy), and an HTTP POST /v1/archive:upload where the body is the bytes, identity rides query parameters, and a JSON receipt comes back.

Encryption is the caller's choice

There is no encryption code inside repo at all. The service stores and serves opaque bytes; if a caller needs confidentiality, it encrypts before storing and decrypts after reading. The platform itself uses this pattern: the delegation transcripts of the transform module are wrapped in AES-256-GCM before they ever reach the repository, then stored through the plain DocumentService under their own MIME type. Encryption-at-rest is a codec layered on top of the claim-check API, not a repo feature, and a drive's credentials_ref likewise points at external credential resolution, never at secrets in messages. Authentication is optional and off by default: with no token set, repo is a trusted-network service reachable only from inside the node's network, and account_id is a plain request field with no per-account authorization inside the service.