Protobuf power tools

Everything runs on descriptors, never generated classes

When you write a .proto file, the protobuf toolchain turns it into a descriptor: a structured object listing every message, field, and type in the file. Most teams experience descriptors only as a build artifact, an intermediate step on the way to generated classes. ProtoMolt's foundational rule, stated in its README, is that everything operates on descriptors (Descriptor / FileDescriptor), never on those generated classes. The same code path serves a DynamicMessage (a message instance assembled at runtime from a descriptor, with no compiled class anywhere) and a compiled-in type, and no module is coupled to any particular message type.

The platform's own server is the proof. At startup it compiles its service definition with the runtime compiler (the Square Wire schema library, entirely in memory, no protoc binary) and binds one dynamic-message handler per method. Server reflection lists it exactly like a stub-generated service. The tool that manages the format is defined in the format, served through its own machinery, and discoverable by its own reflect verb.

One resilience detail makes reflected schemas first-class rather than read-only: rules and annotations declared as protobuf custom options are reparsed from unknown fields when a descriptor arrives without the extension registry (over reflection, say, or out of a schema registry). A schema pulled from a live server carries its validation and metadata options with it, and they still take effect.

The compile verb is the same machinery exposed directly: send it .proto source texts as JSON and it answers with a base64-encoded binary descriptor set, the canonical compiled form every other tool consumes.

dev-tools/protomolt README and docs/surface/grpc-service.md: the Compile verb compiles sources and returns a descriptor set

$ grpcurl -plaintext -d '{"sources": {"shop/v1/order.proto":
    "syntax = \"proto3\";\npackage shop.v1;\nmessage Order { string id = 1; }"}}' \
    localhost:9090 ai.pipestream.proto.grpc.service.v1.ProtoMoltService/Compile
{
  "ok": true,
  "files": ["shop/v1/order.proto"],
  "descriptor_set_base64": "CjsKE3Nob3AvdjEvb3JkZXIucHJvdG8SB3Nob3Au..."
}

Reflect a live service, then call it, with no stubs

gRPC server reflection is a standard feature: a server can be asked, over gRPC itself, to describe its own schemas. ProtoMolt's reflect verb points at any reflection-enabled host and brings back its descriptor set; the grpc-invoke verb then calls any method named by that descriptor. All four streaming shapes (unary, server-streaming, client-streaming, bidirectional) work, and no stubs are generated at any point: the call is assembled from the descriptor at invocation time.

This is exercised, not theoretical. The serve container's end-to-end test drives a dynamic gRPC call purely by reflection, and the ACP agent (the integration for JetBrains AI chat and Zed) reflects ProtoMolt's own gRPC service and invokes ListTypes using the reflected descriptor set. Registering a live endpoint in the service workspace goes one step further: each unary method becomes a catalog verb named <profile>-<method>, so ListOrders on the billing profile answers to billing-list-orders, with no codegen and no restart. Client-streaming methods get no verb, because a verb takes exactly one request; the grpc-invoke verb can still call them directly.

flowchart LR
          point["point the reflect verb at a host"] --> fd["descriptor set comes back"]
          fd --> art["SHA-256 fingerprint pins the artifact"]
          art --> inv["grpc-invoke binds a dynamic stub per method"]
          inv --> shapes["unary, server, client, and bidi streaming"]

$ grpcurl -plaintext orders.internal:9090 list
shop.v1.CatalogService
shop.v1.OrderService

$ grpcurl -plaintext orders.internal:9090 describe shop.v1.OrderService
service OrderService {
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc ListOrders(ListOrdersRequest) returns (stream Order);
  rpc UploadOrders(stream Order) returns (UploadSummary);
}

{
  "type": "object",
  "properties": {
    "order_id": { "type": "string" }
  },
  "required": ["order_id"]
}

# protoc's python and grpc-python generators, running as WebAssembly
import grpc
from shop.v1 import order_pb2 as shop_dot_v1_dot_order__pb2


class OrderServiceStub(object):
    """Client stub for the reflected shop.v1.OrderService."""

    def __init__(self, channel):
        self.GetOrder = channel.unary_unary(
                '/shop.v1.OrderService/GetOrder',
                request_serializer=shop_dot_v1_dot_order__pb2
                    .GetOrderRequest.SerializeToString,
                response_deserializer=shop_dot_v1_dot_order__pb2
                    .Order.FromString,
                _registered_method=True)

One limit of the derived JSON Schema: it does not model oneof fields and it omits meta.v1 descriptions.

Generate client code without protoc

Codegen usually means installing the protobuf compiler and its plugins on every machine that needs it. ProtoMolt instead ships protoc itself compiled to WebAssembly: one bundled module (protoc-wrapper-v4.wasm, about 2.6 MB, loaded from the classpath) executed on Chicory, a pure-Java WebAssembly runtime. A CodeGeneratorRequest goes in on stdin and a CodeGeneratorResponse comes back on stdout, exactly the protoc plugin protocol, with no native toolchain anywhere. The module and approach come from protobuf4j (Apache-2.0), which compiles upstream protobuf to Wasm; the embedded binary's SHA-256 is verified against a recorded provenance file.

On a regular JVM the module is compiled to JVM bytecode once per process, so repeated invocations are fast. Inside the GraalVM native CLI the same module runs on Chicory's interpreter: same generators, slower per-invocation execution, still no native compiler.

The whole thing is exposed as the generate-stubs verb (scope schema-read), which puts it on every front: gRPC, REST, MCP, ACP, and the CLI. An agent holding only a reflected service's descriptor set can generate a compilable client for it without protoc installed.

Those nine are the targets the bundled WASM module emits. Because every contract is standard protobuf, gRPC clients work in all 13 languages gRPC supports; the WASM generator covers the cases where you want stubs produced for you, and any other language compiles the same .proto with its own toolchain.

Infer a schema from sample JSON

Not every data source has a schema. The infer-schema verb reverse-engineers a proto definition from data-rich JSON, per its own description: objects become nested messages, arrays become repeated fields with element inference, and JSON numbers become int64 when they are integral across every sample and double otherwise. Anything genuinely dynamic (mixed-type values, empty objects, empty or mixed arrays, null-only keys) falls back to google.protobuf.Value rather than guessing. Keys are sanitized to legal field identifiers, and when sanitization changes a key the field carries json_name with the original, so the inferred schema round-trips the very documents it was inferred from. Feeding more samples improves the result: keys union, and the numeric heuristic sees every occurrence. A depth cap of 32 guards against adversarial nesting.

The output is proto source plus a linked descriptor set, like every other shape verb, so an inferred schema is immediately usable by validation, mapping, and the rest of the toolkit. It is one pipe away on the native CLI:

dev-tools/protomolt docs and apps/cli: infer-schema over stdin, no server needed

echo '{"samples": [{"name": "x", "n": 1}]}' \
  | docker run --rm -i ghcr.io/ai-pipestream/protomolt-cli infer-schema

Synthesize shapes and merge schemas

The synthesize-shape verb derives a new message type from existing ones in three shapes: an envelope (one message field per named source, lossless), a projection (a flat message whose field types come from scoped source paths, so the SELECT list becomes the schema), and a tagged union (a protobuf oneof over the source types).

The important part is what the output is: the shape is built as a FileDescriptorProto that depends on the sources' files, linked in-process, and emitted as .proto source with true import paths. A synthesized shape is a real, linked protobuf type, not a runtime convenience object. It registers in the git-backed schema registry with references, history, diffs, and compatibility gates, exactly like a hand-written schema, and when the join definition changes, check-compat says whether downstream consumers survive.

merge-schemas combines whole schemas in three steps. Validate: a field with the same name and type in two sources is a natural join key and is coalesced; the same name with a different type or cardinality is a hard clash that blocks emission. Resolve: each clash is settled by renaming (defaulting to <source>_<field>), preferring one source, or overriding the coalescing. Emit: proto source, a descriptor set, and both relationships made explicit: the defined join (one ruleset reading all sources at once) and the defined union (one ruleset per source, a structural UNION). A report-only mode runs the clash report standalone before you commit to a merge. Map-typed fields are refused for now, with a clear error rather than a silent guess.

Gather .proto sources from anywhere

Schemas live in many places, so acquisition goes through one small interface. A gatherer returns a ProtoSourceSet (source texts keyed by import path) and names its origin; a composite merges several, refusing identical import paths that hold different content. Gathered sources compile in memory through the shared compiler, so consumers get descriptors, never files on disk.

dev-tools/protomolt acquire/gather, ProtoGatherer.java: the whole acquisition contract

public interface ProtoGatherer {
    ProtoSourceSet gather() throws GatherException;
    String origin();   // e.g. git:<repo>@<ref>, jar:<file>, maven:<coordinate>
    default boolean isAvailable() { return true; }
}

Two builders show the shape of it. The git gatherer caches clones under ~/.cache/protomolt/gather/git (overridable), can run offline against a warm cache, and exists for the common case of services that publish their contract in git rather than enabling reflection. The Maven gatherer resolves group:artifact:version[:classifier] against configurable remotes through the local ~/.m2 repository. The Maven gatherer is a library with no verb of its own; the git gatherer ships as the gather-git verb.

var gatherer = GitProtoGatherer.builder()
    .repo("https://github.com/example/schemas.git")
    .ref("main")                      // branch, tag, or commit SHA
    .subdir("proto")                  // default
    .token(System.getenv("GH_TOKEN")) // or username/password
    .build();

var gatherer = MavenProtoGatherer.builder()
    .coordinate("com.example:common-protos:1.4.0")  // g:a:v, optional :classifier
    .repositories(List.of("https://repo1.maven.org/maven2/"))
    .transitive(false)                              // true scans the runtime graph
    .build();

On the registry side, loaders and publishers speak the protocols that already exist. The Confluent-compatible loader works against Confluent itself, Apicurio's compatibility facade, Redpanda, or ProtoMolt's own git-backed registry: it lists subjects, fetches references recursively with cycle detection, and compiles each schema in memory. One bad subject (dangling references, unparseable text) is skipped with a warning so it never poisons the whole load, while registry-level failures abort loudly. Lookups cache for 30 seconds; schema-by-id caches forever, because an id in a Confluent-compatible registry names one exact schema and never names another. The matching publisher registers files in reverse-topological import order so every reference exists before the file that imports it, and reports a per-file outcome: created, updated, unchanged, would-write (dry run), or failed. An Apicurio Registry v3 loader and publisher ship natively as a Quarkus extension that can auto-load descriptors on startup. The publisher supports an API token; the loaders speak anonymous HTTP.

Gather and publish compose into one bridge: pull sources from wherever they live, push them into a registry with compatibility gates in between.

var sources = gatherer.gather();
try (var publisher = new ConfluentSchemaPublisher(URI.create("http://localhost:8081"))) {
    PublishResult result = publisher.publish(sources, PublishOptions.defaults());
    result.throwIfFailed();
}

Moving descriptor sets between peers

Descriptors are also the interchange format between systems. A registered service profile pins the SHA-256 fingerprint of its descriptor set, so reflection and registration send the descriptor once and later inspection and invocation resolve it inside ProtoMolt. The schema registry serves a subject's latest schema as a binary descriptor set, topologically ordered with dependencies first, so a consumer can link the whole set in one forward pass.

For peer-to-peer sync there is a designed wire contract, DescriptorExchangeService: content-addressed descriptor-set transfer where a set's identity is the SHA-256 of its canonical bytes, so registering the same content twice is idempotent. It defines register, get, paged list, and a bidirectional sync stream so two peers can converge on the same inventory without a shared store. It is a designed contract, not yet a running server; today's real sync paths are git federation between registries and the HTTP descriptor-set route.