Schema registry, backed by git

Watch a registration become a commit

Every registration runs the same pipeline, in this order: verify that the schema's referenced schemas exist, short-circuit if identical content is already registered, run the compatibility gate for the subject's mode, compile the schema together with its references, and only then commit. Run the three scenarios below. The first adds an optional field and lands as a new commit node. The second changes a field's wire type and dies at the gate with the protocol's 409. The third registers byte-identical text and returns the existing version without touching history.

{"id": <new globalId>}

Two details in that animation carry real weight. First, the gate runs on the subject's effective mode, which is the per-subject setting when one exists and the global mode otherwise, and a mode of NONE skips the gate entirely. Second, compile happens after the gate but before the commit, so what lands in the repository is always something that parses and links. The idempotent path exists so publishers can safely re-register everything they hold: the lookup-by-content endpoint (POST /subjects/{subject}) answers "already there" without writing.

The repository is the storage

GitSchemaRegistryStore treats a git repository as the storage itself, not as an export format. Every register and every setCompatibilityMode is one commit against a non-bare working tree, so the registry's full history is a plain git log and a git pull in the right place is a replication event. The on-disk layout is fixed and human-readable:

schema/registry/core, GitSchemaRegistryStore javadoc: the on-disk layout of the registry repository

registry.json                              global compatibility mode + next globalId counter
subjects/<url-encoded-subject>/v<N>.proto  schema text
subjects/<url-encoded-subject>/v<N>.json   metadata: references, globalId, contentHash
subjects/<url-encoded-subject>/config.json per-subject compatibility mode, when set
workflows/<url-encoded-name>.json          stored workflow definition
workflow-versions/<name>/<version>.pb     immutable promoted version
descriptors/sha256/<fingerprint>.pb       content-addressed descriptor set

Subjects hold ascending 1-based versions, and every registered schema also gets a globalId that is unique and monotonic across the whole store. That id, not the subject or version, is what a serialized message references on the wire, which is why the stale-counter guard matters: on reload the store sets its next id to at least max seen id + 1, so out-of-band commits, say a git pull, can never cause an id to be reused.

Writes are serialized twice over: a shared per-path JVM ReentrantLock plus a registry.lock file lock, so two stores in the same JVM or in different processes cannot corrupt one repository. Reads never take a lock. They run off an in-memory index that is rebuilt on demand, invalidated after every write, and refreshed explicitly to pick up commits made by other tools. There is no delete anywhere in the served protocol: histories are append-only by design.

The Confluent subjects protocol, unchanged

The registry server (protomolt-registry-service) is a JDK HttpServer on virtual threads with no framework dependencies. Its role name is registry and its default port is 8081, the conventional schema registry port. The server itself does no TLS; deployments front it with TLS and the shared-secret token layer. It implements the Confluent subjects REST protocol, content type application/vnd.schemaregistry.v1+json, so stock serializers, the Confluent ecosystem, Apicurio's ccompat facade clients, and Redpanda clients all work as-is:

schema/registry/service, SchemaRegistryServer: the served route table (subjects protocol)

GET     /subjects
GET     /subjects/{subject}/versions
GET     /subjects/{subject}/versions/{version|latest}
POST    /subjects/{subject}/versions        register, returns {"id": …}
POST    /subjects/{subject}                  lookup by content (the idempotency check)
GET     /schemas/ids/{id}
GET|PUT /config                                 global compatibility mode
GET|PUT /config/{subject}                   per-subject compatibility mode

Subject path segments are URL-decoded, so import-path subjects containing slashes round-trip. There is one deliberate protocol quirk to know: PUT bodies and responses say compatibility, while GET responses say compatibilityLevel. That is Confluent's own inconsistency, reproduced so clients do not notice a difference. And PROTOBUF is the only accepted schemaType; anything else is refused as 42201, invalid schema.

Errors use Confluent's envelope shape, with the exact codes the ecosystem expects:

The platform dogfoods this claim as its acceptance test: the round-trip suite publishes a reference-linked source set into the ProtoMolt registry with the stock ConfluentSchemaPublisher and reads it back with ConfluentSchemaRegistryLoader. Third-party conformance runs the same publish/load and compatibility checks against live Apicurio v3, Apicurio's ccompat facade, and Redpanda containers. The rule the tests encode: if those pass against us, we speak the protocol. The registry's live API is HTTP only: there is no registry gRPC service.

The gate and its seven modes

Compatibility modes are kept as opaque, validated strings, exactly the Confluent vocabulary: NONE, BACKWARD, FORWARD, FULL, BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, and FULL_TRANSITIVE. Anything else is refused outright. The global default is BACKWARD, matching Confluent. The built-in gate, CompatibilityWriteGate, is backed by the protomolt-compat library. Non-transitive modes compare the candidate against the latest version; the transitive variants compare it against every historical version.

The checker knows protobuf's real wire-compatibility groups, not just field numbers: varint-interchangeable integers (so an int32 to int64 widening is fine), the zigzag sint* family, fixed-width groups, the asymmetric string to bytes case, and open enums. It also tracks oneof moves, map changes, reserved-number reuse, proto2 required, and gRPC signature and streaming changes. Types match by fully-qualified name across the whole set, so moving a message between files is not a change. Wire-level rules are the default, matching Confluent's protobuf checking; JSON rules (which catch proto3-JSON breakage like field renames) and source rules (which treat generated-code breakage as a violation) are opt-in layers on the same engine. Every violation carries a rule id, a path, and a reason, and over HTTP the refusal is the 409 you saw in the animation.

Standing the store up with the gate attached is a builder call:

var store = GitSchemaRegistryStore.builder()
    .repositoryDir(Path.of("/var/lib/protomolt/registry"))
    .writeGate(new CompatibilityWriteGate())
    .build();

var server = new SchemaRegistryServer(config, store);
server.start();

Federation by git remote

To federate, another mesh's git-backed registry is added as an ordinary git remote of this registry's repository. Sync is strictly pull: fetch, then read straight out of the fetched git objects, never merging into the working tree, then import each version through the normal registration pipeline. Remotes live in git config, which makes them node-local deployment facts rather than registry content, so adding one is not a commit.

flowchart LR
          b["Registry B's repository, in another mesh"] -- "git fetch (pull only)" --> a["Registry A's repository, remote already added"]
          a --> imp["Import each version: namespace subjects, rewrite references, run the gate"]
          imp --> a
          imp -. "diverged history: refused at the divergence point" .-> b

Imported subjects carry their origin. Remote subject s imports as <remote>:s, and a subject federated through two meshes reads as a provenance chain like b:a:s. References between remote subjects are rewritten to the namespaced names, while schema text is left untouched, because its import paths are the reference names. The sync path is always gated by a compatibility check under the target subject's effective local mode, even when the store's own write gate is absent. A version that fails is reported with its violations and stops that subject's import; other subjects continue. Ordering is a fixpoint loop rather than an explicit sort: a version whose rewritten references have not landed yet defers its subject to the next pass, and content-addressed descriptor blobs import verbatim when missing.

Re-running a sync is idempotent and reports versions that were already present. A remote whose history diverged from what was imported is refused at the divergence point, because registry histories are append-only and federation never rewrites what it already imported. Three things are deliberately never synced: the remote's registry.json (global ids and the global mode are local), per-subject compatibility modes (local policy), and workflows (deployment-specific). The whole surface is three contributed verbs with the SCHEMA_WRITE scope: registry-remotes (list, add, remove), registry-sync (fetch and import), and publish-config (typed config write). A designed wire contract for content-addressed descriptor-set sync between peers, DescriptorExchangeService, has no server implementation yet; real sync today is git federation plus the descriptor-set route.

Native extras under /protomolt

Alongside the stock protocol, a configurable single-segment prefix (default /protomolt) serves routes Confluent does not have:

• Descriptor-set download. GET /protomolt/subjects/{subject}/descriptor-set returns the binary FileDescriptorSet of the subject's latest schema plus its transitive references, topologically ordered with dependencies first, so a consumer can link the whole set in one forward pass.
• Parquet schema. GET /protomolt/subjects/{subject}/parquet-schema?message={fqn} derives the Parquet schema of one message as canonical text. It is derived on read, never stored.
• Typed configs. GET /protomolt/configs and GET|PUT /protomolt/configs/{name} store config documents whose type must already be registered. A document is an envelope naming a message type plus canonical proto3 JSON, and the gate runs on write and on read: the type resolves against registered schemas, the JSON parses strictly into a DynamicMessage, and the type's own validate.v1 rules run. The config's version is the git commit id, which is why a hand-edited repository serves a readable refusal rather than an invalid document. The platform's config-registry module reads these documents over this same surface.
• Token auth with scopes. When an API token is configured, every request except GET /health needs the shared secret as an api_token header or a Bearer credential. With a caller resolver mounted, credentials map to principals: reads need the schema-read scope, writes need schema-write, and each contributed verb applies its own scope. Request bodies are capped at 16 MiB (413 beyond that), and a 500 response never carries backend detail: the client gets a correlation id and the stack trace goes to the log.

The actions route (GET /protomolt/actions, POST /protomolt/actions/{name}) is also mounted there, and it is the only HTTP home for the contributed verbs, present only when an action catalog is passed in.