Data Processing

Four modules turn parsed text into embedded, quality-scored documents. module-chunker splits text into deterministic, offset-anchored chunks with per-chunk NLP analytics; pipestream-embedder fills chunk vectors by routing to external inference servers; module-semantic-graph derives centroids and topic boundaries from those vectors; module-quality scores the result with CEL-expression dimensions.

Like every module in the system, these are small stateless gRPC services that pull work from the engine over ai.pipestream.module.work.v1.ModuleWorkService.Work — nothing pushes into a module. The chunker, embedder, quality, and semantic-graph modules listen on ports 19002, 19003, 19006, and 19010 respectively, with HTTP and gRPC sharing one port.

The three-stage semantic pipeline

The semantic pipeline is a strict shape contract over SemanticProcessingResult sets (defined in core-services/pipestream-protos/docs/semantic-pipeline/DESIGN.md):

• Stage 1 — chunker. Emits result sets containing chunks with empty vectors. The empty embedding_config_id / unset vector pair is the explicit "chunked, not embedded" marker, not an omission.
• Stage 2 — embedder. Replaces the Stage-1 results with fan-out copies: vectors filled, everything else copied byte-for-byte.
• Stage 3 — semantic-graph. Appends centroid and semantic-boundary results. Stage-2 results must survive unchanged, and this is enforced by a deep-equal self-check (assertPostSemanticGraph) rather than by convention.

What gets chunked is decided upstream, not by the chunker. Documents arrive carrying vector_set_directives: each VectorDirective pairs a source label and a CEL-ish selector with named chunker configs and named embedder configs. The chunker overrides any sourceField in its config with the directive's source_label, and the identity of the whole directive set is a deterministic key:

modules/module-chunker/docs/architecture.md §6.1

directive_key = sha256b64url(
    source_label + "|" + cel_selector + "|" +
    join(",", sorted(chunker_config_ids)) + "|" +
    join(",", sorted(embedder_config_ids)))

Chunking — module-chunker

The chunker offers five algorithms: token (OpenNLP tokens, the default), sentence, character, inline (the whole field becomes one chunk), and recursive (structural descent: paragraph → line → sentence → whitespace). A sixth, semantic, is declared in the config enum and rejected. The sliding window tokenizes once, takes chunkSize units, and advances chunkSize - chunkOverlap (never less than 1). Config validation allows chunkSize 1–10000 (default 500) and chunkOverlap 0–5000, strictly less than chunkSize (default 50); preserveUrls and cleanText default to true.

Chunk identity is deterministic and offset-anchored, so a re-run of the same text with the same config reproduces the same ids, and any consumer can verify which slice of the source a chunk names:

modules/module-chunker/docs/architecture.md §1

chunk_id = {docHash}:{sourceLabel}:{chunkerConfigId}:{chunkNumber}:{start}:{end}

Every directive also produces a sentence-level sentences_internal result — always, with no opt-out — because it is the substrate for semantic-graph boundary detection. Each chunk carries a ChunkAnalytics record of about twenty fields: POS densities (noun_density, verb_density, lexical_density), vocabulary_density, a potential_heading_score, source offsets, and a SHA-256 content_hash of the sanitised chunk text that enables reprocessing dedup, content-addressed embedder cache keys, and byte-verification of alternative chunker backends (pipeline_core_types.proto:1454). Per-document DocumentAnalytics aggregates the same view at document level.

NLP details

• All NLP is OpenNLP, in-process, with models shipped as Maven resource jars (opennlp-en-ud-ewt-*, langdetect-183.bin).
• Sentence detection is abbreviation-aware (legal and general dictionaries, both on by default). Stock SentenceDetectorME.isAcceptableBreak is quadratic in document length when an abbreviation dictionary is present — minutes on a 4.45 MB document — so the chunker ships AbbreviationAwareSentenceDetector, a bounded-window linear detector, pinned by a corpus test that asserts span-for-span equivalence with stock OpenNLP.
• With chunker.ner.enabled=true, operator-supplied NER models merge entities into single tokens, so "Southern District of New York" cannot be torn across a chunk boundary. OpenNLP publishes no name-finder model as a Maven artifact; models are bring-your-own.
• Glossary merging (bundled legal/finance TSVs), sentence curation (noise and off-language dropping), and optional stemming are available.
• Degradation policy is deliberate: a missing tokenizer model falls back to whitespace, but an enabled NER feature with no loadable model fails at startup — "a pipeline that quietly stopped recognising entities produces indexes that look fine and are wrong."

Embedding — pipestream-embedder

The embedder does no inference itself. It routes per model to a pluggable EmbeddingBackend SPI (CDI-discovered) with three providers:

• djl-serving — DJL Serving over plain HTTP/JSON.
• openvino — OpenVINO Model Server, speaking the standard KServe v2 inference.GRPCInferenceService proto directly, with no Pipestream wrapper.
• static-embeddings — opt-in, in-process model2vec static tables.

Models are registered purely by config — no per-model code. The shipped registry includes minilm, mpnet, e5-small, e5-large, distilroberta, paraphrase-minilm, paraphrase-multilingual, multi-qa, and bge-m3. A model can span several endpoints across priority tiers, with round-robin within a tier and failover that marks a failed endpoint down for a cooldown:

core-services/pipestream-embedder/README.md

embedder.connections.dev.base-url=${EMBEDDER_DJL_SERVING_URL:http://localhost:8080}
embedder.connections.openvino-local.provider=openvino

embedder.models.minilm.connection=dev
embedder.models.minilm.dimensions=384
embedder.models.minilm.serving-name=minilm
embedder.models.minilm.model-identifier=sentence-transformers/all-MiniLM-L6-v2

# multi-endpoint, priority tiers:
embedder.models.minilm.connections=cuda-1:10,cuda-2:10,vino-a:20

Two concurrency gates bound the load: embedder.rpc.max-concurrent (default 64) and embedder.in-flight.max-concurrency (default 32). Everything is blocking Java on virtual threads; there is no reactive framework in the request path. E5-style query:/passage: prefixes are applied per purpose (INDEX/QUERY), and a SemanticEmbedderService.StreamEmbeddings stream serves off-pipeline callers.

Static embeddings, with measured numbers

The in-process static backend exists for throughput-bound indexing, and its trade-off is published rather than implied: 96,710 sentences/sec in isolation versus 3,904 for DJL/MiniLM, a 24.9× end-to-end speedup, at recall@10 of 0.4277 against the transformer teacher — documented in docs/static-embeddings.md alongside an explicit "do not use for final ranking" warning, the Amdahl caveat, and the command to reproduce. The static embedder refuses to emit a vector for all-OOV text (a zero vector "distorts every neighbourhood it lands in") and fails startup if a distilled table emits un-normalised vectors. The offline embedder-model-distiller CLI that distils a teacher into a static safetensors table is kept out of the serving path because it drags in the ONNX runtime.

Semantic graph — module-semantic-graph

The semantic-graph module derives coarser views from the Stage-2 chunk vectors, appending them as new result sets:

• Centroids. Document, section, and paragraph centroids computed as pure-CPU mean pooling (uniform or sqrt_length weighting) plus L2 normalization over existing chunk vectors. parent_result_id makes the document → section → paragraph → chunk hierarchy traversable.
• Topic boundaries. Consecutive-sentence cosine similarity over the sentences_internal vectors, gated by both a percentile and an absolute threshold (boundary_similarity_threshold: 0.5, boundary_percentile_threshold: 20), with group sizes enforced between boundary_min_sentences_per_chunk: 2 and boundary_max_sentences_per_chunk: 30. Each group is then re-embedded via the named boundary_embedding_model_id, which has no "first available" fallback — an absent model is a FAILED_PRECONDITION.

The stage is guarded by hard invariants rather than best effort. assertPostEmbedder runs on entry and assertPostSemanticGraph on exit. The cap max_semantic_chunks_per_doc (default 10000) fails the document rather than silently truncating it. Boundary re-embedding fans out as semaphore-bounded sub-batches on virtual threads with exponential-backoff retries (max_batch_size=32, max_subbatches_per_doc=5, max_retry_attempts=2, retry_backoff_ms=100). Measured on an RTX 4080 SUPER with MiniLM, the boundary pass runs at p50 = 7 ms and p95 = 9 ms against a design gate of ≤ 500 ms ("MET with 55× margin"); on CPU, p50 is 26 ms.

Quality scoring — module-quality

The quality module writes a QualityIndex into SearchMetadata.quality_index: a composite score plus per-dimension records carrying weight snapshots and skip explanations. Five profiles ship in the box — default, scientific-papers, news-articles, technical-docs, web-content — selected per document through a category map (quality-category-map.json), and the composite combines dimensions by weighted_mean, harmonic_mean, geometric_mean, or min.

Dimensions are either CEL expressions — compiled once, cached as ASTs, and coerced/clamped into [0, 1] — or native QualityChecker CDI beans (MinHash duplication, host-reputation authority, cosine topic relevance). Two real dimensions from the default profile:

modules/module-quality/src/main/resources/quality-default-profile.json

{
  "dimension_id": "recency",
  "weight": 0.5,
  "expected_inputs": ["search_metadata"],
  "cel": "hasDate ? exp(-0.693 * ageDays / 365.0) : 0.5"
},
{
  "dimension_id": "readability",
  "weight": 0.75,
  "expected_inputs": ["document_analytics"],
  "cel": "clamp(1.0 - (avgSentenceLength > 20.0 ? (avgSentenceLength - 20.0) / 40.0 : (avgSentenceLength < 8.0 ? (8.0 - avgSentenceLength) / 8.0 : 0.0)), 0.0, 1.0)"
}

The governing rule is "judge only what was witnessed": a dimension whose expected_inputs are absent from the document is skipped, not zeroed — it vanishes from the record with a skip explanation, and the composite is computed over witnessed evidence only. A missing input is therefore never silently treated as a bad score. Confidence is binary today (1.0 scored, 0.0 skipped); graduated confidence is listed as future work.

Related