Skip to content

Semantic Search

Build a semantic search pipeline using text embeddings and cross-encoder reranking. inference4j provides two complementary models: SentenceTransformerEmbedder for fast retrieval and MiniLMSearchReranker for precision reranking.

Quick example — embeddings

try (var embedder = SentenceTransformerEmbedder.builder()
        .modelId("inference4j/all-MiniLM-L6-v2").build()) {
    float[] embedding = embedder.encode("Hello, world!");
}

Quick example — reranking

try (var reranker = MiniLMSearchReranker.builder().build()) {
    float score = reranker.score("What is Java?", "Java is a programming language.");
}

Full search pipeline

A typical semantic search pipeline uses embeddings for fast candidate retrieval, then a cross-encoder reranker for precision scoring of the top results.

import io.github.inference4j.nlp.SentenceTransformerEmbedder;
import io.github.inference4j.nlp.MiniLMSearchReranker;

public class SemanticSearch {
    public static void main(String[] args) {
        String query = "How do I handle errors in Java?";
        List<String> documents = List.of(
            "Java uses try-catch blocks for exception handling.",
            "Python decorators are a powerful feature.",
            "Error handling in Java includes checked and unchecked exceptions.",
            "The Java Stream API provides functional operations on collections."
        );

        // Stage 1: Embed and retrieve candidates by cosine similarity
        try (var embedder = SentenceTransformerEmbedder.builder()
                .modelId("inference4j/all-MiniLM-L6-v2").build()) {

            float[] queryEmbedding = embedder.encode(query);
            List<float[]> docEmbeddings = embedder.encodeBatch(documents);

            // Rank by cosine similarity (your own similarity function)
            // ...
        }

        // Stage 2: Rerank top candidates with cross-encoder
        try (var reranker = MiniLMSearchReranker.builder().build()) {
            float[] scores = reranker.scoreBatch(query, documents);

            for (int i = 0; i < documents.size(); i++) {
                System.out.printf("%.4f  %s%n", scores[i], documents.get(i));
            }
        }
    }
}

Embedder builder options

Method Type Default Description
.modelId(String) String — (required) HuggingFace model ID
.modelSource(ModelSource) ModelSource HuggingFaceModelSource Model resolution strategy
.sessionOptions(SessionConfigurer) SessionConfigurer default ONNX Runtime session config
.tokenizer(Tokenizer) Tokenizer auto-loaded WordPieceTokenizer Custom tokenizer
.poolingStrategy(PoolingStrategy) PoolingStrategy MEAN Pooling method: CLS, MEAN, or MAX
.normalize() — disabled Enables L2 normalization of output embeddings
.textPrefix(String) String null Text prefix to prepend before encoding. encode(String, String) and encodeBatch(List, String) override it per call
.maxLength(int) int 512 Maximum token sequence length
.truncation(TruncationPolicy) TruncationPolicy TRUNCATE Input longer than the token limit: TRUNCATE keeps the first tokens and logs a warning, FAIL throws InputTooLongException. See Input length
.stride(int) int off Embed input longer than maxLength in full: windows sharing this many tokens (0 = consecutive chunks), averaged by the tokens each covers
.dimensions(int) int off (full size) Matryoshka truncation: keep the first N values, re-normalized when .normalize() is on. Only for models trained for it (nomic-embed-text-v1.5)

Reranker builder options

Method Type Default Description
.modelId(String) String inference4j/ms-marco-MiniLM-L-6-v2 HuggingFace model ID
.modelSource(ModelSource) ModelSource HuggingFaceModelSource Model resolution strategy
.sessionOptions(SessionConfigurer) SessionConfigurer default ONNX Runtime session config
.tokenizer(Tokenizer) Tokenizer auto-loaded WordPieceTokenizer Custom tokenizer
.maxLength(int) int 512 Maximum token sequence length
.truncation(TruncationPolicy) TruncationPolicy TRUNCATE Input longer than the token limit: TRUNCATE keeps the first tokens and logs a warning, FAIL throws InputTooLongException. See Input length
.stride(int) int off Score documents longer than maxLength with MaxP: passages sharing this many tokens, each scored with the full query; the highest score wins

Input length

Task Limit When exceeded Long-input strategies
Embedder maxLength tokens (512 by default) Truncated with a warning (default), or rejected with .truncation(TruncationPolicy.FAIL) .stride(int): windows averaged into one vector
Reranker maxLength tokens (512 by default) for query and document combined; the longer one is shortened first Truncated with a warning (default), or rejected with .truncation(TruncationPolicy.FAIL) .stride(int): MaxP over passages

An embedding of a truncated text only represents its beginning. There are two ways to cover long documents:

  • Split documents into chunks before embedding (for example with a document splitter), and store one vector per chunk. This retrieves best: every passage keeps its own vector.
  • Use .stride(int) when you need exactly one vector per document. The text is split into windows, each window is embedded, and the vectors are averaged, weighted by the tokens each window covers. This matches what LangChain4j's in-process embedding models do.
try (var embedder = SentenceTransformerEmbedder.builder()
        .modelId("inference4j/all-MiniLM-L6-v2")
        .normalize()
        .stride(0)          // consecutive chunks; use e.g. 64 for overlapping windows
        .build()) {
    float[] documentVector = embedder.encode(longDocument);
}

An averaged vector represents each part of the document in proportion to its length. A topic covered by one sentence in a long document carries little weight, which is why chunking is usually better for search. In a test with about 550 tokens about baking followed by about 250 about whales, a query about whales scored 0.04 cosine similarity against the truncated embedding and 0.21 against the averaged one.

Reranking long documents (MaxP)

With .stride(int), a document too long for one pass is split into passages, each passage is scored together with the full query, and the highest passage score is the document's score. This is the established "MaxP" technique.

try (var reranker = MiniLMSearchReranker.builder()
        .maxLength(128)
        .stride(32)
        .build()) {
    float score = reranker.score("How many people live in Berlin?", longDocument);
}

Keep passages short

ms-marco-MiniLM-L-6-v2 was trained on short MS MARCO passages, and a relevant sentence carries less weight the more unrelated text surrounds it. To measure this, we appended the sentence "Berlin has a population of about 3.7 million people…" to the end of 10 real news articles (each over 520 tokens), and scored them for the query "How many people live in Berlin?":

Setting Mean score
No windowing (truncates at 512) 0.000
stride(128) with the default maxLength(512) 0.076
maxLength(256).stride(64) 0.001
maxLength(128).stride(32) 0.962 (lowest 0.872)
The same articles without the sentence, MaxP 0.000

Use maxLength(128).stride(32) for MaxP. The effect also applies without MaxP: when reranking retrieved chunks, shorter chunks (around 100–150 tokens) let a relevant passage stand out. With varied text, the drop is gradual: the same sentence preceded by about 150 to 240 tokens of other text still scored 0.80–0.92 on its own.

Result types

Embeddings

encode() returns a float[] — a dense vector representation of the input text. Use cosine similarity to compare embeddings:

static double cosineSimilarity(float[] a, float[] b) {
    double dot = 0, normA = 0, normB = 0;
    for (int i = 0; i < a.length; i++) {
        dot += a[i] * b[i];
        normA += a[i] * a[i];
        normB += b[i] * b[i];
    }
    return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

Reranking scores

score() returns a float in the range [0, 1] — the sigmoid-normalized relevance score for a query-document pair. Higher means more relevant.

scoreBatch() returns a float[] — one score per document, scored against the same query.

Pooling strategies

The embedder supports three pooling strategies for converting token-level representations into a single sentence embedding:

Strategy Description
MEAN Average of all token embeddings (default, best for most tasks)
CLS Uses only the [CLS] token embedding
MAX Element-wise maximum across all token embeddings

Available embedding models

Model Dim Pooling Normalize Text Prefix MTEB Avg
inference4j/all-MiniLM-L6-v2 384 MEAN Optional — 56.26
inference4j/all-mpnet-base-v2 768 MEAN Optional — 57.78
inference4j/bge-base-en-v1.5 768 CLS Recommended — 63.55
inference4j/gte-base 768 CLS Recommended — 61.36
inference4j/nomic-embed-text-v1.5 768 (Matryoshka: 512, 256, 128, 64) MEAN Recommended search_query: / search_document: 62.28

BGE example

try (var embedder = SentenceTransformerEmbedder.builder()
        .modelId("inference4j/bge-base-en-v1.5")
        .poolingStrategy(PoolingStrategy.CLS)
        .normalize()
        .build()) {
    float[] embedding = embedder.encode("What is machine learning?");
}

Nomic example (long context, prefixes, smaller vectors)

nomic-embed-text-v1.5 reads up to 8,192 tokens, so raise maxLength for long documents. It needs a task prefix: "search_query: " for queries and "search_document: " for documents. Pass the prefix per call, so one embedder (and one model in memory) serves both sides.

try (var embedder = SentenceTransformerEmbedder.builder()
        .modelId("inference4j/nomic-embed-text-v1.5")
        .maxLength(8192)
        .normalize()
        .build()) {
    float[] query = embedder.encode("What is the capital of France?", "search_query: ");
    List<float[]> documents = embedder.encodeBatch(chunks, "search_document: ");
}

A per-call prefix replaces the builder's .textPrefix(String) for that call; null means no prefix. With .stride(int), the prefix is repeated in every window.

The model was trained with Matryoshka representation learning, so .dimensions(int) can shorten its vectors with little loss. Smaller vectors cut index size and search time. Use the same dimensions for queries and documents.

Dimensions MTEB Avg
768 62.28
512 61.96
256 61.04
128 59.34
64 56.10

Scores are from the model card. The card applies a layer norm before truncating. inference4j truncates and re-normalizes without it; the two match to a cosine similarity of 0.9999 or better on our reference set.

E5 example (with text prefix)

E5 models require a text prefix: "query: " for queries, "passage: " for documents.

try (var embedder = SentenceTransformerEmbedder.builder()
        .modelId("inference4j/e5-base-v2")
        .normalize()
        .build()) {
    float[] query = embedder.encode("What is Java?", "query: ");
    float[] passage = embedder.encode("Java is a programming language.", "passage: ");
}

If an embedder only ever sees one kind of input, set the prefix once with .textPrefix("query: ") and call encode(String).

Tips

  • Two-stage pipeline: Use embeddings for fast top-K retrieval (cheap cosine similarity), then rerank the top candidates with the cross-encoder (expensive but more accurate).
  • Batch encoding: Use encodeBatch() when encoding multiple texts — more efficient than calling encode() in a loop.
  • L2 normalization: Enable .normalize() when comparing embeddings with cosine similarity. BGE, GTE, and E5 models all recommend normalization.
  • Text prefix: Some models (E5, Nomic) require a text prefix. Check the model card for the correct prefix. Use encode(text, prefix) when queries and documents need different prefixes.
  • Embedding dimension depends on the model: all-MiniLM-L6-v2 produces 384-dimensional vectors, while BGE/GTE/mpnet/Nomic produce 768-dimensional vectors. Nomic can be shortened with .dimensions(int).