inference4j for LangChain4j¶
inference4j-langchain4j provides inference4j models as implementations of LangChain4j interfaces, for use in
ingestion pipelines, retrievers and AI Services. The models run inside the JVM, without an API key, a model server
or Python.
Adapters¶
| Use case | Adapter | LangChain4j interface | Page |
|---|---|---|---|
| Redact personal data from documents before they are embedded and stored | Inference4jPiiRedactor |
DocumentTransformer, TextSegmentTransformer |
PII redaction |
| Block prompt injection in user messages with a classifier | Inference4jPromptInjectionGuardrail |
InputGuardrail |
Prompt-injection guardrail |
| Read the text in scanned pages, screenshots and photos | Inference4jOcrDocumentParser |
DocumentParser |
OCR document parser |
| Transcribe recordings into documents with timestamps | Inference4jAudioDocumentParser |
DocumentParser |
Audio document parser |
| Embed text, with query and document prefixes applied from the input type | Inference4jEmbeddingModel |
EmbeddingModel |
Embedding model |
| Rerank retrieved content with a cross-encoder | Inference4jScoringModel |
ScoringModel |
Reranking |
RAG pipelines shows how the adapters fit together, with examples for recordings and scanned documents.
Comparison with LangChain4j's built-in options¶
| Capability | LangChain4j 1.23 | inference4j adapter |
|---|---|---|
| PII detection and redaction | Not provided | GLiNER PII model as a DocumentTransformer / TextSegmentTransformer |
| Prompt-injection detection | PatternBasedPromptInjectionGuardrail (regular expressions), DecisionModelInputGuardrail (one LLM call per message) |
DeBERTa-v3 classifier, about 10 ms per message on a laptop CPU |
| OCR | Docling parser (calls a separate docling-serve process), or a multimodal chat model |
PaddleOCR PP-OCRv6 in-process, about 31 MB of models |
| Speech to documents | AudioTranscriptionModel, implemented by hosted providers; responses carry no timestamps |
Moonshine, Whisper or Wav2Vec2 in-process, one Document per timed segment |
| Local embeddings | Packaged in-process ONNX models (all-MiniLM-L6-v2, bge-small, e5-small) and OnnxEmbeddingModel for a custom file |
bge-base, gte-base, nomic-embed-text-v1.5 (8K context, Matryoshka sizes); query/document prefixes from EmbeddingInputType |
| Local reranking | OnnxScoringModel, given paths to a model file and a tokenizer.json |
Cross-encoder downloaded and cached automatically; MaxP for long segments |
| Local chat model | Ollama, Jlama and others | Not provided as a ChatModel |
Choosing an embedding or scoring model¶
Both libraries run embedding and reranking models in-process on ONNX Runtime.
| LangChain4j in-process models | inference4j adapters | |
|---|---|---|
| Model files | Packaged in the jar (embeddings), or paths supplied by the caller (OnnxEmbeddingModel, OnnxScoringModel) |
Downloaded from HuggingFace on first use and cached |
| Models | all-MiniLM-L6-v2, bge-small-en, bge-small-zh, e5-small-v2, or a custom ONNX file | The models listed under Embedding model and Reranking |
| Query/document prefixes | Added to the text by the caller | Applied from EmbeddingInputType |
| Batches | Embedded in parallel on a thread pool | Embedded one text at a time; each run uses all cores |
| Long input | Embeddings: split into partitions and averaged. Scoring: truncated | Truncated by default; windows with .stride(int) (averaged for embeddings, MaxP for scoring) |
Installation¶
Add the adapter module next to the LangChain4j dependencies. It includes inference4j-core.
| inference4j | 0.14.0 or later |
| LangChain4j | 1.23.0 (tested) |
| Java | 17 or later |
- LangChain4j is a
compileOnlydependency of the module. The application supplies its own LangChain4j version. - ONNX Runtime requires the JVM flag
--enable-native-access=ALL-UNNAMED. See Installation. WhisperSpeechModelrequiresinference4j-genai. All other models on these pages are ininference4j-core.
Experimental LangChain4j APIs
LangChain4j marks guardrails and the embedding request API (EmbeddingInputType) as experimental. They may
change in a later LangChain4j release.
ONNX Runtime version
inference4j requires ONNX Runtime 1.28.0. LangChain4j's in-process modules (langchain4j-embeddings-*,
langchain4j-onnx-scoring, and langchain4j-easy-rag, which includes an embedding module) declare ONNX
Runtime 1.23.2. When both are on the classpath, make sure the build resolves 1.28.0: Gradle selects the newer
version, Maven selects the version declared closest to the project. LangChain4j's ONNX modules are not tested
with 1.28.0.
Quick start¶
A retrieval-augmented assistant with a local embedding model and a local guardrail. The chat model can be any
LangChain4j ChatModel:
import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.splitter.DocumentSplitters;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.chat.ChatModel;
import dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever;
import dev.langchain4j.service.AiServices;
import dev.langchain4j.store.embedding.EmbeddingStoreIngestor;
import dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore;
import io.github.inference4j.langchain4j.Inference4jEmbeddingModel;
import io.github.inference4j.langchain4j.Inference4jPromptInjectionGuardrail;
import io.github.inference4j.nlp.DebertaPromptInjectionDetector;
import io.github.inference4j.nlp.SentenceTransformerEmbedder;
interface Assistant {
String chat(String message);
}
ChatModel chatModel = ...; // OpenAI, Anthropic, Ollama: any LangChain4j chat model
List<Document> documents = ...; // loaded with any LangChain4j document loader
var embeddingModel = Inference4jEmbeddingModel.builder()
.embedder(SentenceTransformerEmbedder.builder()
.modelId("inference4j/all-MiniLM-L6-v2").normalize().build())
.build();
var store = new InMemoryEmbeddingStore<TextSegment>();
EmbeddingStoreIngestor.builder()
.documentSplitter(DocumentSplitters.recursive(300, 30))
.embeddingModel(embeddingModel)
.embeddingStore(store)
.build()
.ingest(documents);
Assistant assistant = AiServices.builder(Assistant.class)
.chatModel(chatModel)
.contentRetriever(EmbeddingStoreContentRetriever.builder()
.embeddingStore(store).embeddingModel(embeddingModel).maxResults(4).build())
.inputGuardrails(new Inference4jPromptInjectionGuardrail(
DebertaPromptInjectionDetector.builder().build()))
.build();
assistant.chat("What is the cancellation policy?");
Models are downloaded from HuggingFace on first use and cached under ~/.cache/inference4j. See
Model loading for offline use and custom locations.
Lifecycle¶
Closing an adapter closes its model
Each adapter owns the inference4j model passed to it. close() on the adapter closes that model and frees its
native memory.
- Give each adapter its own model instance. A model wrapped in two adapters is closed when either adapter is closed.
- Calls on a closed model throw
IllegalStateException. - Close adapters on application shutdown. All adapters are
AutoCloseable.
Limitations¶
- Scanned PDFs. The OCR parser reads images. PDF pages must be rasterized first; see OCR document parser.
- Chat messages. The PII redactor implements the ingestion interfaces only. See PII redaction.
- Compressed audio and video. The audio parser reads WAV.
- Chat models. inference4j's text generators are not available as a LangChain4j
ChatModel.