Mesh: agents working together

What the mesh is (and is not)

A “mesh” here means nodes exchanging protobuf messages as google.protobuf.Any, resolving each message's exact descriptor, applying registered validation and transformation policy, and routing it to local or remote processors. A processor can be a gRPC service, an LLM, a deterministic Java component, an OpenNLP analysis pipeline, or another mesh node. The design corpus states the boundary plainly:

“The mesh is not an LLM task broker. It is a typed entity-processing fabric. LLM-assisted software generation is one processor profile on that fabric.” docs/design/pipestream-protobuf-mesh/README.md

The unifying idea is that typed entities with exact schema identity cross trust boundaries. Everything else (discovery, leases, fencing, the encrypted log) exists to make that safe across machines that share no memory and no clock they fully trust. The profile aligns with an IETF draft, draft-krickert-pipestream-03, without redefining the wire protocol; a gRPC bidirectional transport is the planned first interoperable implementation.

Three Gradle modules make up the mesh:

The contracts commit to wire packages ai.pipestream.proto.mesh.v1, ai.pipestream.proto.mesh.cluster.v1, and ai.pipestream.proto.mesh.cluster.storage.v1, published as Maven artifacts under ai.pipestream:protomolt-*. The contracts module holds no networking, storage, or processor logic; the cluster module is pure and single-threaded, taking time from an injected clock. That discipline is deliberate: contracts first, transports arrive later.

One contract for every hop

Everything that moves through the mesh is an EntityEnvelope: a header, an exact schema identity, and exactly one body. The body is either an inline google.protobuf.Any payload or a ClaimCheck, which is a pointer to a content-addressed artifact for payloads too large or too sensitive to inline. A claim check never carries credentials, pre-signed URLs, or inline bytes, and it retains the payload's type name and descriptor fingerprint so rehydration restores exact identity.

flowchart LR
  hdr["EntityHeader · id, digest, deadline, trace id"]
  sch["SchemaReference · type name plus descriptor fingerprint"]
  env["EntityEnvelope"]
  inl["inline google.protobuf.Any payload"]
  chk["ClaimCheck · content-addressed coordinates, no credentials"]
  hdr --> env
  sch --> env
  env --> inl
  env --> chk

The header carries transport-visible identity only, never credentials: the entity id (a UUID that doubles as the idempotency key under at-least-once delivery), a SHA-256 digest of the exact payload bytes, a deadline, and a W3C trace id. The schema identity is a SchemaReference: the fully qualified type name plus a fingerprint, which is the SHA-256 of the canonical descriptor-set closure (the defining file plus its transitive dependencies, sorted and deterministically serialized). The contract is explicit that a type URL alone is never sufficient identity: a same-named type with drifted bytes fingerprints differently and must be rejected.

The exactly-one-body rule is enforced in the message itself, with CEL (Common Expression Language, protobuf's annotation-driven validation language), so a malformed envelope is rejected before any routing, persistence, or execution sees it:

mesh/proto/src/main/proto/ai/pipestream/proto/mesh/v1/entity.proto: the exactly-one-body rule on EntityEnvelope

option (ai.pipestream.proto.validate.v1.message) = {
  cel: [
    {
      id: "exactly-one-body"
      message: "an entity carries exactly one inline payload or one claim check, never both and never neither"
      expression: "has(this.payload) != has(this.claim_check)"
    },
    ...
  }
};

Note the deliberate choice: the two body fields are plain fields, not a protobuf oneof. A producer bug that sets both is therefore constructible, and the CEL rejects it at the boundary. Failing closed beats making the failure unrepresentable.

The contract gate and exact schema identity

Every boundary a mesh entity crosses (routing, persistence, processor execution) admits it through one method, MeshGate.admit. The gate performs three layers together, so no caller can accidentally run only a subset: structural validation, deadline expiry against the caller's clock, and schema-identity resolution against a descriptor registry. Every failure throws IllegalArgumentException before anything routes, persists, or executes; success returns the exact descriptor the body's bytes deserialize with. Deadline checks are deliberately separate from structure, because schema resolution must still work on expired entities, for example when rehydrating evidence after the fact.

Two details are worth pausing on:

• Fingerprints commit to everything the producer understood. MeshDigest preserves unknown fields instead of stripping them, because an unknown field is schema content the producer understood and the consumer does not, and identity must include it. Payload digests commit to the exact wire bytes as received, with no re-serialization.
• A mismatch names both fingerprints. The schema-identity resolver binds a SchemaReference to a live descriptor: the registry must hold exactly that type name, and the canonical closure fingerprint must equal the declared one. A mismatch error prints both fingerprints side by side, so a drifted contract is diagnosable from the error alone.

Contracts can also declare their own processing semantics with two mesh.v1 custom options on protobuf messages and fields: profiles by reference, recursion bounds, whether LLM processing is allowed, whether a PII scan is required, approval and evidence policies, and nine independent field roles, including remote_disclosure_prohibited, which marks a field that must never leave the trust domain (no remote processor and no LLM provider may receive it). The options describe; they grant no authority. There are no endpoint addresses, no credentials, and no executable policy in a contract:

option (ai.pipestream.proto.mesh.v1.message) = {
  processing_profile: {
    name: "nlp-standard"
    version: "1.2.0"
  }
  result_type: "ai.pipestream.proto.mesh.test.v1.TestResult"
  capabilities: [
    "opennlp-ner",
    "java-build"
  ]
  route_profile: {name: "default-routes"}
  recursion: {
    max_depth: 4
    max_children: 16
  }
  scatter_profile: {name: "line-scatter"}
  rehydration_profile: {name: "ordered-collect"}
  llm_allowed: true
  pii_scan_required: true
  approval_policy: APPROVAL_POLICY_ON_COMPLETION
  evidence_policy: EVIDENCE_POLICY_SUMMARY
};

// A secret the mesh must never send to a remote processor.
string api_token = 4 [
  (ai.pipestream.proto.meta.v1.field).sensitivity = "secret",
  (ai.pipestream.proto.mesh.v1.field) = {remote_disclosure_prohibited: true}
];

The cluster directory: who is here, what can they serve

The part of the mesh that is live today is the cluster directory. Nodes and processors do not get configured into a fleet; they advertise themselves, and the directory is the reduced view of everything currently advertised. Four kinds of advertisement flow in:

flowchart TD
  subgraph ads["What a node publishes"]
    na["NodeAdvertisement · capabilities, schemas served with fingerprints, endpoints, TTL"]
    pa["ProcessorAdvertisement · kind, accepted schemas by fingerprint, lease, provider and model"]
    ca["CapacityAdvertisement · in-flight against limits"]
    np["NodePresence · heartbeat plus TTL · ACTIVE, SUSPECT, GONE, DRAINING"]
  end
  dir["ClusterDirectory · a pure reducer over an event log, deterministic snapshots"]
  log["Durable log · one AES-256-GCM blob per cluster at mesh/<cluster-id>/events.pb.enc, ciphertext only at rest"]
  na --> dir
  pa --> dir
  ca --> dir
  np --> dir
  dir --> log

Four properties keep the directory trustworthy when nodes crash, networks partition, and retries double-deliver:

• Fencing and idempotency. Every identity carries an (epoch, sequence) position. Re-applying an identical record is a no-op and reports UNCHANGED; a changed record needs a strictly newer position; a changed record from a stale position is a conflict and throws. Expired identities keep fencing tombstones, so a delayed frame from a superseded incarnation cannot be admitted after a restart.
• TTL presence. A node is live while its presence is ACTIVE and its expires_at (which CEL pins to last_heartbeat_at + ttl) is in the future. Heartbeats extend the window; a coordinator-side sweeper running every 30 seconds expires silent nodes and cascades their processors. Nodes never remove themselves; they simply stop heartbeating.
• Presence is soft state. Heartbeats and refresh-only re-advertisements emit no event and never touch the repository. Persisting them would buy nothing a restart could not rebuild by waiting, at the cost of the durable write path on the directory's most frequent call.
• Durability is replay, then install. Every durable mutation replays the retained log into a candidate directory, applies and validates the change, saves the log, and only then installs the candidate as the live projection, so failed writes cannot leak memory-only membership. Reads take no lock. Once more than 256 events accumulate, the log folds into a checkpoint; sequences stay monotonic, so watchers are unaffected.

The repository adapter stores the whole event log as one AES-256-GCM-encrypted blob through the repository service's unary blob RPCs, under object key mesh/<cluster-id>/events.pb.enc. The encryption tag authenticates the media type, key reference, drive, and object key, and the envelope records the cluster id, cluster fingerprint, event count, and plaintext SHA-256. The repository, its object store, and any cache only ever see ciphertext. A 15-second deadline on the blob call means an unreachable repository stalls mutations instead of the whole directory, and an 8 MiB plaintext cap bounds the log.

How nodes actually join today

The directory lives in-process inside protomolt-serve on the coordinator host, enabled with --mesh-cluster-id plus --mesh-created-at (or the matching PROTOMOLT_MESH_* environment variables). It mounts six catalog verbs into the shared action catalog, reachable over every ProtoMolt front (MCP streamable HTTP, JSON/REST, and the gRPC actions route), all requiring the worker-coordinate authorization scope: mesh-node-register, mesh-node-heartbeat, mesh-processor-register, mesh-capacity-update, mesh-snapshot, and mesh-sweep. Every mutating verb answers with a DirectoryCommit: an apply outcome (REGISTERED / UPDATED / UNCHANGED) plus the snapshot sequence and fingerprint, so a caller holding both can tell whether a later read reflects its own write.

A real node publishes through these verbs today. A Jetson (called “nano1”) runs a Python systemd service that health-gates its hardware (a live embedding probe against a pinned BAAI/bge-small-en-v1.5 model serving 384-dimensional vectors), then calls mesh-node-register, mesh-node-heartbeat, and mesh-processor-register every 30 seconds with 90-second leases. Its processors include nano1-tei (a gRPC embedding service advertising capabilities like embedding, dimensions-384, and cuda-sm87) and nano1-arm64-builder (a deterministic ARM64 build-capacity processor). If a health gate fails, the lease is simply not renewed, and the processor expires out of the directory:

scripts/nano1-mesh-publisher.py: a deployed node renewing its presence and lease (trimmed)

client.call("mesh-node-register", {"advertisement": node})
client.call("mesh-node-heartbeat", {"presence": {
    "nodeId": "nano1",
    "state": "PRESENCE_STATE_ACTIVE",
    "ttl": "90s",
    "nodeEpoch": str(state.epoch),
    # ...
}})

The publisher keeps its fencing sequence durable locally before any remote mutation becomes visible. A crash may skip a number, which is allowed; it must never reuse one.

Three meshes, one job

Now the story the contracts are built for. Below are three mesh clusters: one whose processors host Kimi agents, one hosting GPT, one hosting a locally served Llama model. A job enters the first mesh, finds no local processor for one step of its schema, and is delegated across meshes. Each hop is the same typed envelope; only the processors change.

Read this as the designed end-to-end flow. The directory half of this story is running today, and the delegation bridge described below is implemented and tested. The general entity router that would move envelopes between meshes is designed and is not implemented yet. The animation exists to teach the contract; the steps underneath it are the real mechanics.

1. A job arrives in the Kimi mesh, wrapped as an EntityEnvelope: a header, the exact schema identity, and exactly one body, either inline bytes or a claim check pointing at stored bytes.
2. The Kimi mesh asks its cluster directory which processor can serve this schema. The directory answers with leased advertisements that match on schema fingerprint, not just type name.
3. One step has no local match, so the envelope crosses to the GPT mesh. It still carries the same schema fingerprint, payload digest, and trace id it started with.
4. That step needs a locally hosted model, so the GPT mesh delegates again, to a Llama processor that advertised itself through an OpenAI-compatible provider.
5. The Llama processor finishes its slice and returns a typed result. The result names the same schema closure, so the receiver can verify the bytes before trusting them.
6. The result travels back through the GPT mesh, where a review step checks the schema fingerprint before admitting it.
7. Every directory mutation answered with a DirectoryCommit, so a retried delivery reports UNCHANGED instead of running the work twice. The entity id is the idempotency key.
8. The Kimi mesh reassembles the pieces and closes the scope as COMPLETED. Three meshes, one typed contract, one trace id end to end.

Agents on the fabric

LLM processors are a first-class processor kind in the mesh vocabulary (ProcessorKind.LLM, alongside gRPC services, deterministic components, OpenNLP pipelines, and other mesh nodes), and a processor advertisement is agent-aware. It names a provider (the contract's own examples are codex and kimi), a model, and a model_version, plus session behavior: whether the processor supports session resume, its disconnect grace, and its maximum active sessions. Eligibility is judged against schema identity, not type names alone, so “an LLM that accepts this exact contract version” is a query the directory can answer.

The agent-host application attaches those providers to the platform's durable delegation protocol over MCP streamable HTTP: it keeps a Codex process, a Kimi process, or a local OpenAI-compatible model attached to the delegation tools. Delegation is the agent-task protocol one level up from the directory: workers say hello (WorkerHello), receive offers, hold leases, write checkpoints, and submit candidates for review, exposed as a family of delegation-* catalog verbs.

The seam between the two is DelegationBridge. It maps a delegation WorkerHello to a leased mesh ProcessorAdvertisement: the processor id is the worker id, the kind is LLM when the hello names a provider (deterministic otherwise), capabilities are deduplicated in declaration order, and the caller supplies the node position and lease. Admission, offers, heartbeats, renewals, and expiry remain the delegation coordinator's business; the derived advertisement is simply how a delegated worker becomes visible to mesh discovery. The bridge is implemented and unit-tested; it does not yet have a production caller wiring the delegation coordinator to the directory.

That is the multi-agent shape: each agent family (Kimi, Codex, an OpenAI-compatible local model, and the platform's own deterministic and pipeline processors) runs as processors in a mesh, and the contracts are how one mesh's job becomes another mesh's work. The directory knows them all the same way: leased, fingerprinted, and fenced.