Inference: embeddings, rerank, LLMs

One contract, pluggable backends

An embedding is a fixed-length list of numbers (a vector) that a model computes from a piece of text, arranged so that texts with similar meaning land near each other. Cosine similarity measures the angle between two vectors: 1.0 means "same direction, same meaning", 0 means "unrelated". A kNN (k-nearest neighbors) query turns a search question into a vector and asks the index for the closest stored vectors, and an HNSW index is the approximate data structure that answers that question quickly over millions of vectors.

Everything model-facing in ProtoMolt follows one pattern: a small contract, a catalog that lists what is available, and a service-provider interface (SPI), which is Java's name for a plug socket discovered automatically from the classpath. For embeddings the contract is three methods:

public interface EmbeddingProvider extends AutoCloseable {
    /** Stable id, e.g. {@code model2vec}. */
    String providerId();

    /** Number of components in every vector this provider produces. */
    int dimension();

    /** Embeds {@code text} into a {@link #dimension()}-component vector. */
    float[] embed(String text);
}

The defaults matter: embedAll loops over embed unless a provider has a real batch API, and close() is a no-op unless the provider holds network resources. Remote providers hold connections, so lookups hand lifecycle to the caller: you get one owned instance, and the ones not selected are closed for you.

Three embedding providers, three deployment shapes

Three providers ship today, each a different answer to "where does the model run":

• tei talks to Hugging Face Text Embeddings Inference, a dedicated model-serving container, over its native gRPC protocol. One TEI process serves one model, calls carry a 30 second deadline, and the provider learns its vector dimension at runtime by embedding a fixed probe text rather than trusting configuration. A server-side truncation toggle is exposed as the PROTOMOLT_TEI_TRUNCATE knob.
• ovms talks to OpenVINO Model Server over the KServe v2 gRPC protocol, the same wire protocol NVIDIA Triton speaks. The provider sends raw strings as one BYTES tensor of shape [N] and reads back an FP32 tensor of shape [N, dim]; the model tokenizes server-side, and each request is capped at a batch size.
• model2vec runs in the same process as the platform: a distilled Model2Vec token-vector table with subword tokenization and mean pooling. There is no neural network forward pass and no server to operate; it is pure Java over an OpenNLP static embedding model, and it is the product default, the standalone CPU fast path with nothing to ship or run.

Discovery never breaks on an unconfigured provider: constructors are lazy, so a TEI or OVMS provider sitting on the classpath unconfigured is simply absent from the list until its endpoint is set. Remote channels are plaintext, meant for behind a trusted network boundary.

One vocabulary word matters here. Some embedding model families, E5 among them, are trained with a marker word glued to the text: query: in front of a question, passage: in front of a document. The same sentence with and without the marker lands in a different region of the vector space. So whether a model expects markers, and which ones, is part of the recipe rather than a footnote, which is exactly why the recipe is declared in the schema and pinned by a digest, as the next sections show. It is also why swapping providers is a certification event, not a config tweak: if one side applies the marker and the other does not, their vectors stop agreeing and the harness below refuses to certify the pair.

A fourth participant rides along: the mapping embedder joins a provider to the index mapping and fills a document's VECTOR field from its TEXT field. Before it embeds anything it passes a sensitivity gate: by default, a TEXT field carrying any meta.v1 sensitivity class is refused, because an embedding leaves the process, is kept by whatever serves the model, and cannot be un-sent. Deployments opt classes in by name. That gate lives on the same descriptor metadata the Types page covers.

Certification: prove two providers agree before you mix them

Here is the problem this page exists to teach. Search workloads mix providers all the time: index the corpus with one server, answer queries from another, or shard documents across both. That is safe only if both servers produce the same vector space for the same model. Two servers both claiming to serve, say, one bge checkpoint can still disagree: different tokenizers, different batching, different normalization, a marker prefix applied on one side and not the other. If they disagree, a query vector computed on one server silently fails to match document vectors computed on the other, and nothing throws an error. The index just gets worse.

ProtoMolt's answer is a certification harness. EmbeddingEquivalence.compare(a, b, texts, threshold) embeds a corpus with both providers and reduces per-text cosine similarities to an EquivalenceReport: minimum cosine, mean cosine, the range of norm ratios, the threshold, and a certified flag. The norm-ratio range is the subtle part: cosine similarity is scale-invariant, so two providers can certify at cosine ~1 while one emits vectors twice the magnitude of the other. That disagreement does not hurt a cosine-scored index, but it breaks an index scored with L2 or dot product, so the report carries it explicitly.

The rerank side gets its own harness with a different statistic, because a reranker's raw scores are not comparable across providers at all: one emits sigmoid probabilities, another emits raw logits, so only the order a provider produces is meaningful. RerankEquivalence.compare therefore scores per-query Kendall tau-b (a rank correlation: 1.0 means identical ordering, accounting for ties) plus top-1 agreement, with a scoreEpsilon noise floor of 1e-3 so that sub-floor jitter on sigmoid-scaled providers counts as "same relevance" rather than a disagreement.

These are not paper checks. TeiOvmsEquivalenceLiveIntegrationTest enforces the 0.995 cosine threshold and its rerank sibling, TeiOvmsRerankEquivalenceLiveIntegrationTest, enforces the 0.9 tau-b bar against live servers, gated behind environment knobs; the OpenSearch integration lane runs against a self-provisioned OpenSearch with two TEI CPU containers. Two providers serving one model must agree before they may be mixed: the rule is in the code, with a number attached.

Where the vector config lives: on the schema

Nothing about the vector lane lives in a separate config file. The chunking policy and the vector recipe are field options in the protobuf schema itself, the same mechanism the Types page describes: declare once, every subsystem reads the descriptor. A text field that should be chunked and embedded at index time carries a chunking_policy option; a field that already holds a vector (say, one a client computed itself) declares a vector hint directly:

dev-tools/protomolt search index SPI, indexing_hints.proto: the policy shape and the vector hint

// A chunking policy: derive chunks and vectors from the annotated source text
// field at index time (chunk + embed on the server). Attach to a string field.
// The policy digest (spi ChunkingPolicy.digest()) identifies the derivation
// pipeline; index consumers pin it the way they pin a schema.
message ChunkingPolicy {
  ChunkingSpec chunking = 1;
  EmbeddingSpec embedding = 2;
  string vector_field = 3;   // empty = engine convention ("<field>#<model>")
  bool store_chunk_text = 4;
}

repeated float embedding = 3 [(ai.pipestream.proto.index.hints.v1.index) = {
  type: INDEX_FIELD_TYPE_VECTOR
  vector_dims: 768
  vector_similarity: VECTOR_SIMILARITY_COSINE
  hnsw: { m: 16, ef_construction: 128 }
}];

The pieces, in plain words:

• The ChunkingSpec names the strategy (today: sentence-packed, version 1), the target, overlap, min, and max token counts, and a pinned boundary rule-set id. The rule sets are frozen on purpose: rules-v1 is hand-rolled text splitting, and opennlp-v1 pins an OpenNLP sentence model loaded from the classpath, never downloaded. A chunker refuses every other strategy, version, or boundary id by name.
• The EmbeddingSpec names the model id, the dimension, the similarity measure (cosine, dot product, L2, max inner product), and whether vectors are L2-normalized after embedding. Resolving a spec to a provider is strict: the provider registered under the model id must exist and must produce vectors of exactly that dimension, refused loudly otherwise.
• The policy computes a digest, a SHA-256 over a canonical rendering of every component. Two indexes agree on their chunk boundaries and vector spaces exactly when their policy digests agree, and each stored chunk is identified as <doc_id>#<digest, first 12 chars>#<ordinal>. Change any value and you get a new digest, a new chunk generation, and a re-index via replay. This is what the Search page means by chunk-and-embed at index time.

So the model recipe travels with the schema, versioned like the schema, and the certification harness from the previous section is what lets you change the server behind that recipe without lying to the index.

Rerank: order is the only portable currency

A cross-encoder reranker takes the query and one candidate document together and scores how well they match. It is slower than embedding-based retrieval, so it is used as a second stage: recall a deep candidate set cheaply, then rerank precisely. The SPI is a single idea, score(query, texts), with a default rank(…, topK) on top.

Two providers ship: tei over TEI's gRPC rerank API, and ovms over OVMS's OpenAI-style REST endpoint, POST {base}/v3/rerank. The REST choice is deliberate and documented: the OVMS rerank servable's graph expects an HTTP payload packet, so the gRPC ModelInfer path answers ovms::HttpPayload was requested on these servables. Embeddings go gRPC for both providers; rerank goes gRPC for TEI and REST for OVMS.

The consumer that composes the stages is RerankedSemanticSearch: embed the query, kNN-recall a deep candidate set, cross-encoder rerank, return the reordered top-k. Every hit carries both scores side by side, because the kNN similarity and the reranker's relevance score are not commensurable, so neither may stand in for the other. That composition is the rerank harness's tau-b threshold made operational: the two providers inside it had to certify their agreement first. Today the composed rerank pipeline runs in the OpenSearch module; the mounted Lucene service is retrieve-only.

Structured generation: a form a model must fill in legibly

The third job is the fanciest: structured generation, which means asking a language model to produce output that conforms to a protobuf message, not free text. The coordinator is paranoid in a specific order. Before any model is invoked, it validates the request against the request's own declared rules, resolves the target message type in the descriptor registry, and requires the catalog model to declare the structured_output capability. Only then does a model see the prompt.

Each attempt renders a prompt packet (a companion module turns the message descriptor into a form-filling briefing for the model), sends the rendered JSON Schema as the decoder constraint, parses the reply with strict protobuf JSON (unknown fields are rejected), and runs the result through validate.v1. If validation fails, the rejection is rendered back to the model as feedback and one more attempt runs, hard-capped at three. The cap is not a convention; it is a schema rule on the request itself:

dev-tools/protomolt inference/proto, structured.proto: the retry ceiling is declared, not assumed

// Maximum generation attempts, counting the first; 0 applies the
// coordinator default (3). Total attempts NEVER exceed 3.
int32 max_attempts = 4 [
  (ai.pipestream.proto.validate.v1.field) = {
    int32: {gte: 0, lte: 3}
  }
];

The design reasoning is worth stating: a runaway repair loop is a schema violation, never a runtime surprise, so the ceiling lives where all the other rules live. And the retry discipline is asymmetric on purpose: retries happen only from rendered rejection feedback, and provider errors (a down server, a transport failure) abort immediately, never retried. Every attempt is kept as provenance, raw output, outcome, feedback, token usage, and the response carries SHA-256 prompt_fingerprint and schema_fingerprint values, so a downstream system can pin exactly which prompt shape and which schema version produced a given fill.

Structured generation is unary only; streaming generation deliberately omits structured output.

The inference contract behind the LLM lane

The language-model side has its own wire contract, InferenceService, with four RPCs: Generate, GenerateStream, ListModels, and DescribeModel. The contract validates itself like everything else in the platform: temperature must sit between 0.0 and 2.0, the message list must hold between 1 and 4096 entries, and every GenerateResponse names the provider, the catalog model id, and the provider-reported model version, so a downstream truth system can pin which model rendered a given answer, forever.

Behind it, the provider SPI is three methods, and its error rule is the strictest sentence in the module:

public interface InferenceProvider {
    /** The provider id catalog entries reference (e.g. "openvino"). */
    String id();

    GenerateResponse generate(ModelEntry model, GenerateRequest request);

    void generateStream(ModelEntry model, GenerateStreamRequest request, ChunkObserver observer);
}

// Every failure is an InferenceException; providers never fall
// back to another backend, another model, or a default endpoint.

No silent fallback is what keeps the provenance meaningful: if a request was answered, the named provider answered it. The catalog in front is a thread-safe in-memory registry with a mutation generation counter, so a consumer can tell a stale model listing from a current one. Adding a model is a catalog entry, not a code change, and an entry's credential_ref is a pointer such as env:OPENAI_TOKEN, validated against a strict pattern and marked sensitive, resolved host-side at request time. A resolver failure names the failure class only; it never carries the reference or the resolved value. The catalog is in-memory: it is rebuilt from launcher flags at every start, neither persistent nor distributed.

Two providers ship over one shared OpenAI-compatible chat transport: openvino points at OpenVINO Model Server's /v3/chat/completions, and openai points at /v1/chat/completions, which is the dialect Ollama, vLLM, and llama.cpp all speak, covering both the NVIDIA lane and the edge-box lane. Unset sampling knobs are omitted from the wire body rather than sent as defaults, and responses name the provider id, not the shared transport. The OpenVINO provider currently leaves the model version field unfilled, so provenance from that side names the provider and model id only.