Types & Metadata

The descriptor door: decoding a message you have never seen

ProtoMolt's messages are protobuf — compact binary structures defined by schema files. When a pipeline hands a document to a consumer, the payload usually travels inside a google.protobuf.Any envelope: a type URL saying what the message is, plus the raw bytes. That is convenient for transport and awkward for a browser, which has no generated code for the type and would normally stare at opaque bytes.

The registration service — the service that catalogs the modules and types running in a deployment — answers GetTypeDescriptor with everything needed to close that gap. At build time, one snapshot of every schema in the workspace is baked into the service. At runtime it walks the import graph and returns the file that defines the requested type plus every file that file depends on, dependencies first. A browser feeds those bytes straight into a runtime descriptor registry and can then decode the payload — no per-type code, no codegen step.

core-services/pipestream-protos registration module, platform_registration.proto (abridged)

// GetTypeDescriptor: resolve one protobuf message type to a
// self-contained binary FileDescriptorSet ... the frontend feeds
// the bytes to a runtime descriptor registry (e.g. @bufbuild/protobuf
// createFileRegistry) and can then decode ANY google.protobuf.Any
// payload of this type with zero per-type code.

This is not a theoretical hook. The frontend's useAnyDecoder.ts builds exactly such a registry from the served bytes and caches it per type URL; the parsed-metadata viewers and the mapping studio's field trees both render through it. A companion call, ListTypeDescriptors, exists so type pickers can be built without a hardcoded catalog — it lists every type in the workspace with a substring filter.

One detail worth noticing: the version of a descriptor set is a hash of its bytes (SHA-256). Identical content always hashes to the same version, so a client can cache for as long as the version matches. There is no version history to browse — the snapshot is the version.

Layered field metadata: notes that never touch the schema

Knowing the shape of a message is only half of displaying it well. The other half is knowing what to call a field, how to explain it, and how careful to be with it. The proto doc-comments were written for developers reading schema files; a document-inspection screen needs friendlier words, and a pipeline that crawls a private space may need to reclassify a field that is harmless elsewhere.

The type metadata catalog stores those human-facing annotations as overlays — entries keyed to field paths of a message type, kept entirely outside the schema. Four layers stack, and for any given field the highest layer that has an entry wins, replacing the lower entry outright:

• BAKED — the field's proto doc-comment, read from the descriptor snapshot. Free: even a type nobody has annotated serves sensible descriptions.
• GLOBAL — the reviewed, promoted set of annotations, kept in the schema registry. This is the system of record.
• GLOBAL_DRAFT — work in progress. Visible in merged reads, clearly labeled as a draft, until someone promotes it to GLOBAL.
• PIPELINE — scoped to one pipeline graph. Lets one pipeline relabel or reclassify fields for its own context, and is deliberately fenced in: it never promotes and never leaks into another graph's view.

Every field in a merged response carries a provenance tag naming the layer it came from, and the schema itself insists that editors display it — a reader should always be able to tell whether they are looking at a reviewed fact or somebody's draft.

A single annotation entry is small on purpose. It can hold a display name, a description, free-form tags, a sensitivity classification, and a catch-all map for anything the platform does not model explicitly:

core-services/pipestream-protos registration module, type_metadata.proto

message FieldMetadataEntry {
  string display_name = 1;             // Human-friendly name for UI display
  string description = 2;
  repeated string tags = 3;            // Free-form classification tags
  FieldSensitivity sensitivity = 4;    // PUBLIC / INTERNAL / CONFIDENTIAL / PII
  map<string, string> custom = 5;   // Anything the platform does not model explicitly
}

Two guardrails keep the catalog honest. First, every field path is validated against the baked descriptor on save — an annotation that names a field which does not exist is rejected, so overlays can describe the schema but never fork it, and deleting a field from a proto can only orphan an annotation, never silently rebind it to something else. Second, sensitivity is a label, not a lock: nothing in the running platform enforces it today. It exists so that screens and, eventually, policy checks can act on it.

Promotion is the governance step: a reviewed GLOBAL_DRAFT is written into the schema registry as the GLOBAL overlay and goes live to every merged read immediately, and the draft is consumed in the process.

The toolkit: types that say more about themselves

There is a second, younger codebase that shares the ProtoMolt name — a companion toolkit for building on the platform's schemas, still pre-1.0. It takes the opposite approach to the same problem: instead of attaching notes around the schema at runtime, it lets a message declare facts about itself in the schema, as descriptor options written once and read by every subsystem. Nothing in the running platform reads these options yet; they belong to the toolkit's own toolchain.

The option families cover validation (including 36 named string formats — email, URI, date, hostname, language tag, and friends), indexing hints (which fields become text, keyword, date, range, or vector fields, and how), display and sensitivity metadata — a recognizable schema-native cousin of the overlay entry above — plus quality scoring, field provenance, and instructions for language-model extraction. The validation formats are checked by hand-written parsers, not regular expressions: the calendar parser rejects 2026-02-30, a date a pattern would happily accept.

The toolkit also defines a small library of canonical structural types that extend what a plain scalar can say: DateRange, LongRange, DoubleRange (bounded ranges with include/exclude flags on each end), and TreePath for taxonomy paths. The philosophy is wire-plain, server-strict: a range is just two strings and two booleans on the wire, so any client in any language works untouched — and the server enforces the invariants, like "begin must not be after end", with structured rejections. The indexing layer recognizes these range types by name, so they need no extra hints.

A canonical range type with its invariants attached looks like this:

// A calendar-day range. Bounds are strict ISO dates, so the
// whole range stays wire-plain: any proto client can read it.
message DateRange {
  option (ai.pipestream.proto.validate.v1.message) = {
    cel: { id: "date_range.ordered"
           message: "begin must not be after end"
           expression: "!has(this.begin) || !has(this.end) || this.begin <= this.end" }
  };
  optional string begin = 1 [(ai.pipestream.proto.validate.v1.field) = { string: { date: true } }];
  optional string end   = 2 [(ai.pipestream.proto.validate.v1.field) = { string: { date: true } }];
  optional bool include_head = 3;  // Absent = included
  optional bool include_tail = 4;
}

And because the options compose, one field can carry its validation, its human-facing description, and its extraction instructions for a language model in a single declaration:

dev-tools/protomolt llm examples, court/v1/opinion_metadata.proto

string court = 1 [
  (ai.pipestream.proto.validate.v1.field) = { required: true  string: { max_len: 200 } },
  (ai.pipestream.proto.meta.v1.field)     = { description: "The issuing court, as it appears in the caption." },
  (ai.pipestream.proto.llm.v1.field)      = { instruction: "Name the court exactly as it appears in the caption."
                                            safeguards: "Do not abbreviate." }
];