ProtoMolt

Asset: formats and classification

The archive stores opaque bytes on purpose. Downstream parsers, bridges, and catalogs still need to know what those bytes are, and they need "we do not know yet" and "the evidence disagrees" as stored facts, not as missing fields. The asset layer is that vocabulary.

Overview. Asset answers what a blob of bytes is. It uses ordinary archival words: format, format identification, characterization, classification, media type, and fixity. A producer can declare a format at the door; a sniffer can identify one from bytes; and the platform records agreement, disagreement, or ignorance as first-class states instead of inventing a default. 
flowchart LR
        bytes["Opaque bytes on a drive"] --> sniffer["ContentTypeSniffer · one magic-prefix table"]
        sniffer --> fact["FormatFact · closed oneof of 18 formats"]
        decl["Producer declaration at a door"] --> classn["Classification state"]
        fact --> classn
        classn --> archive["Archive · stores, filters, ClassifyEntry"]
        classn --> parse["Parse coordinator · same sniffer for routing"]
Two modules, shared by archive and parse: the contract holds FormatFact and Classification; the engine sniffs bytes and resolves states.

Where it lives

The contract starts in asset/proto (:protomolt-asset-proto): the FormatFact registry, the Classification state machine, and ContentProfile. It depends only on the validation contract. asset/characterize (:protomolt-asset-characterize) is the engine: ContentTypeSniffer, FormatGrammars, the identifier, FormatCompatibility, and Classifications.resolve. It is pure JDK.

Bridging and catalog facets live beside that pair: asset/bridge, asset/bridge-parse, and asset/catalog. The archive in repo/ stores classifications, validates declarations at upload doors, and holds the derived renditions bridges write. The parse coordinator routes work using the same sniffer, so a format conclusion is one platform answer, not two competing guesses.

Formats as messages

Each format is a protobuf message inside a closed oneof on FormatFact. There is no parallel Java model and no separate rulebook: the validate.v1 annotations on the message are the claim contract. Eighteen formats ship today, grouped by kind. Click a kind to see which messages it holds.

Eighteen formats in a closed oneof. The chips filter by kind; every name below is a real FormatFact arm.
  • TarArchiveContainer
  • ZipArchiveContainer
  • GzipFileContainer
  • ParquetDatasetDataset
  • AvroDatasetDataset
  • DelimitedTableDataset
  • NdjsonDatasetDataset
  • JsonDocumentSemi-structured
  • XmlDocumentSemi-structured
  • YamlDocumentSemi-structured
  • PdfDocumentDocument
  • WordDocumentDocument
  • SpreadsheetDocumentDocument
  • PresentationDocumentDocument
  • MarkdownDocumentDocument
  • HtmlDocumentDocument
  • PlainTextDocument
  • RasterImageMedia

Showing all eighteen formats.

The rules are strict on purpose. A TarArchive filename must match its grammar. A GzipFile CEL rule keeps .tar.gz as tar, not gzip. A DelimitedTable needs a one-character delimiter and a HeaderPresence enum (not a bool). A RasterImage media type must be an image/* MIME. Filename fields use ignore_if_zero: an unset name is allowed by the field rule, but a set name is held to grammar. Presence of a name on a declaration is a door rule: the archive refuses nameless declarations.

ObjectStoreOrigin (bucket plus object key) rides beside classification as origin metadata. It is not a format.

Audio and video formats are absent until something in-tree validates or bridges them.

Classification states

CLASSIFICATION_STATE_UNSPECIFIED is never stored. Every stored classification is one of five states. Step through how a typical entry moves, or jump to a state:

Classification is a stored fact. Invalid declarations are refused at the door; valid declarations that contradict bytes land as CONFLICTED, never rewritten in silence.

UNCLASSIFIED

Stored with no format known. Listable and countable as a work item: something is on the shelf, and nobody has said what it is yet.

State Meaning
UNCLASSIFIED Stored, no format known; a listable work item.
DECLARED Producer declared a format and the door validated it.
IDENTIFIED Characterization concluded from bytes with evidence; nothing was declared.
VERIFIED Declared and identified agree.
CONFLICTED Declared and identified disagree; stored loudly; bridging refuses.

Record versus refuse is deliberate. An invalid declaration is refused at the door. A valid declaration that contradicts the bytes lands as CONFLICTED. No declaration leaves the entry UNCLASSIFIED, then IDENTIFIED after characterization. The Classification message carries CEL shape rules so state and fields always match, plus CharacterizationEvidence, Attribution (classified_by), ObjectStoreOrigin, and classified_at.

Identification may generalize without creating a conflict. FormatCompatibility treats CSV as a kind of text, docx as a kind of zip, and a compressed tar as a kind of gzip. A claim that sits inside a generalization is not a conflict; only a ruling-out conclusion conflicts. Classifications.resolve is the single resolution point every consumer calls.

Characterization: one sniffer

ContentTypeSniffer is one platform magic-prefix table. ZIP payloads dispatch on extension; an inconclusive probe returns sniffed=false. The parse coordinator uses the same table, so routing and archival classification cannot drift.

Characterizer.identify(head, filename) reads the first 512 bytes (PREFIX_BYTES). A gzip-plus-tar name becomes compressed tar. JSON gets a probe. Delimited tables are never identified from bytes alone: delimiter and header presence must come from the producer. Names are endorsed only when they match grammar. Every fact the characterizer produces is valid by construction.

Content classes

Beside format sits ContentProfile: what kind of material the bytes carry for downstream quality and routing. The content classes are INFORMATIONAL_TEXT, CONVERSATIONAL_TEXT, TABULAR_DATA, OCR_TEXT, and CODE. A QualityScore carries dimensions; OCR material must carry measured quality.

Archive integration

The archive is where classification becomes durable. EntryInfo.classification rides on every entry. PutEntry and UploadRendition accept a declared format and an origin. RenditionManifestEntry.content_profile carries the content class. The primary rendition is what classifies the entry. Declarations, once made, are never silently withdrawn.

ClassifyEntry re-reads the primary rendition through the same detection seam. ListEntriesRequest can filter by classification_state, and ArchiveStats.classification_states exposes exact per-state counts computed from the archive's live entries.

The archive stores each entry as a v2 document (described on Repo), and its classification rides in the entry's CORE: the encoded Classification under archive.classification and the state name under archive.classification_state, the key the filter matches. A streamed upload of the primary rendition keeps only the 64 KiB head and 128 KiB tail of the bytes as they pass, classifies from them, and sends the result in the upload trailer, which the repository admits again before the one commit. A stock repository host serves these calls on its gRPC port.

Renditions, derived ones included, can be read back as a stream. ReadRendition sends one rendition of one version, whole or as a byte range, in frames of at most 64 KiB paced by the reader. Its header carries the manifest's size and SHA-256 for the whole object; a whole read is checked against that digest and fails with DATA_LOSS on a mismatch, while a range read is checked for length only. The renditions viewer in the repository console lists an entry's renditions with their state, size, hash, and provenance, and previews or downloads them through this call over the repository host's Connect gateway, in binary protobuf, for a signed-in user whose account bindings reach the archive.

Bridging: derive, do not convert

A bridge derives a new rendition beside the original. It never converts bytes in place. Bridging is gated on the classification state machine, provenance-stamped on every output, and refused when classification names no single format: only the three states that name one format (DECLARED, IDENTIFIED, VERIFIED) may bridge. CONFLICTED and UNCLASSIFIED sources are refused rather than guessed at.

Bridges is a routing rule, not a runtime of its own: which bridge kinds apply to which format, which three states name a format, what derived rendition names to write, and each output's schema pin taken from a generated descriptor. BridgeEngine.forKind(kind, format) separates applicable from executable on this host (keyed on both). One schema bridge, for example, reads CSV, NDJSON, and Avro, but not a Parquet footer.

BridgeEntry writes every output in one version beside the untouched original and reports per bridge: PRODUCED, UNCHANGED (idempotence via content addressing, not a ran-already flag), DEFERRED (names what work still needs), or FAILED (verbatim). The bridge-entry verb is the same two calls as a checkpointed, retryable run. There is no new runtime.

Running bridge kinds write derived renditions today. Deferred kinds stay named in the contract with reasons, not as silent gaps. Click a chip to filter.
  • container membersRunning
  • schema for Avro / delimited / NDJSONRunning
  • textRunning
  • ocr-textRunning
  • Parquet schemaDeferred
  • datasetDeferred
  • conversationDeferred

Showing every bridge kind named in the contract.

OCR quality is scored as CEL dimensions on the descriptor (for example "0.62", reviewable in git). Engine confidence rides along unscored: treating silence as zero would punish an engine that declines to guess. Conversation kind still routes from no format yet; extracted text is text/plain with no schema-subject pin.

Facets: catalog that cannot diverge

AssetCatalogRow declares index hints and metric members on its own fields. Projection computes nothing and reads no object, so the catalog cannot diverge from the archive. The path is verified against a real Lucene index and a real metric executor for actual catalog questions.

Build-taught rules

  1. A derived rendition must never become an entry's primary. Bridges write beside the original; the primary stays the classifying bytes.
  2. OCR escalation fires on a zero-byte derivation. An empty text rendition lands as empty, not as a missing row.
  3. Index fields that mean "no answer" must declare null_value. Otherwise the facet silently drops exactly the assets a backlog view exists to find.

A CONFLICTED console surface remains the open follow-on.

Six rules

  1. Format rules are the claim contract. There is no lenient mode.
  2. States are stored facts, never defaults.
  3. Identification carries evidence.
  4. Bridges derive; they never convert in place.
  5. One detector.
  6. Proto-first; no parallel Java model.