ProtoMolt
ProtoMolt is an open-source platform that takes documents from where they already live — file shares, S3 buckets, databases, wikis — and turns them into structured, searchable data. Each document travels through a pipeline of processing steps (parsing, chunking, embedding), and the results land in OpenSearch, ready for keyword and semantic search.
flowchart LR src["Sources · files, buckets, databases, wikis"] --> intake["Connector intake"] intake --> engine["Engine"] engine --> modules["Modules · parse, chunk, embed"] modules --> repo["Repository · S3 bodies + Postgres ledger"] repo --> search["OpenSearch · keyword + vector"]
The life of one document
One document, one journey. Press play and watch a single document chip move through the loop — this is the same path every file, database row, or wiki page takes.
- Sources
- Intake
- Engine
- Modules
- Repository
- Search
One document’s path through the platform, left to right.
Try it
All you need is Docker and a machine that can run a multi-container Compose stack. The
whole platform — engine, repository, connectors, modules, Kafka, OpenSearch, and the rest —
ships as one reference Compose project under deploy/compose-stack, built
either from source with ./build-stack.sh or from released images with
./build-stack.sh --released. Underneath, it is an ordinary Compose file:
deploy/compose-stack/README.md — the reference stack
docker compose -f docker-compose.stack.yml up -dA first crawl looks like this: once the stack is healthy, you point a connector at something you own — a filesystem directory, an S3 bucket, a database table — and let it run. Documents start appearing in OpenSearch a few hops later, having been parsed, chunked, and embedded on the way. The ticker above is the journey each one takes.
If you would rather hack on one service than run the fleet: every service is a Quarkus
application, and quarkus dev starts the infrastructure it needs — Postgres,
Kafka, the schema registry, OpenSearch, an S3-compatible store, Redis, Grafana —
automatically, from the same Compose definitions the full stack uses.
The platform, piece by piece
Engine
The traffic controller. It owns each document’s journey, hands work to modules that ask for it — never the other way around — and decides what happens when a module crashes mid-step.
Repository
The document store. Bodies live in S3-compatible object storage, a Postgres ledger tracks where everything is, and deletes are provable — which matters when someone asks to be forgotten.
Modules
The workers. Small processing services — in Java, or in any language behind a proxy — that pull work from the engine over gRPC and hand results back.
Graphs
Pipelines as data. Every pipeline is a versioned snapshot of steps and branches, so you can always see exactly which recipe processed a given document.
Processing
What happens to the text: chunking with stable IDs, embeddings through pluggable inference backends, and quality scoring along the way.
Parsers
The readers. A fleet of standalone services that turn PDFs, Office files, email, ebooks, and more into clean typed data — streamed, never written to disk.
Search
Where results land. One governed writer feeds OpenSearch, keyword and vector search run side by side, and an experimental distributed engine is in the works.
Mapping & validation
The glue logic. Filters, field mappings, and routing rules written as small CEL expressions that travel inside the pipeline definition itself.
Types & metadata
Every field is typed and self-describing: descriptors on demand, layered metadata overlays with sensitivity labels, and extended types like ranges and tree paths.
Connectors
The front door. Connectors for databases, S3, filesystems, Confluence, and SharePoint push documents in through an authenticated intake service; web crawling and Google Drive are on the way.
Output formats
One typed representation of a document, renderable as text, Markdown, HTML, JSON, and more — so downstream tools get the shape they need.
Design positions
Pull, not push. Modules connect to the engine and ask for work; the engine never calls out to a module. Each open stream is one document in flight, so a slow module simply asks less often — backpressure with nothing to configure. Scaling out means opening more streams, and scaling to zero just lets the queue wait. There is no service-discovery system at all; the platform uses static addresses, which works because the engine never needs to find modules — they find it.
Pointers on the bus, bodies in storage. Kafka messages carry a small reference, never the document itself. The bytes live in S3-compatible object storage and are fetched only by the step that needs them, so a huge video file and a two-line email move through the same pipeline without either one clogging it.
Named outcomes instead of silent drops. A module that has nothing to do says so with a typed no-op, and the document moves on unchanged. Stopping a document requires an explicit rejection with a stated reason — a rejection is a recorded decision, not an error. And because the engine only acknowledges work after a valid result comes back, a crashed module means the work is delivered again, not lost; there is no separate retry queue to maintain.
Deterministic identity. Document IDs, work-unit IDs, and Kafka keys are computed from content and coordinates rather than generated at random. A retry of the same work arrives carrying the same identity, so repeats converge on the same state instead of creating duplicates.
Durability is a per-hop choice. Every connection between two steps declares how carefully its handoff is kept — written down for replay, staged cheaply, or held in memory only. A document under a right-to-be-forgotten request can force the in-memory mode on every hop, so its bytes are never written to disk in the first place.
ai.pipestream, buf modules under buf.build/pipestreamai, container
images under docker.io/pipestreamai, and Kafka topics like
pipestream.module.*. Those names are stable and are not changing.