API Overview¶
Package structure¶
inference4j-core¶
| Package | Contents |
|---|---|
io.github.inference4j |
Core contracts: InferenceTask, Classifier, Detector, ZeroShotClassifier, ZeroShotInput, AbstractInferenceTask, Tensor, TensorType, InferenceSession, NativeTensors, InferenceContext |
io.github.inference4j.session |
Session config: SessionConfigurer, SessionOptions |
io.github.inference4j.model |
Model resolution: ModelSource, HuggingFaceModelSource, LocalModelSource |
io.github.inference4j.processing |
Pre/post-processing: Preprocessor, Postprocessor, OutputOperator, MathOps, TruncationPolicy, TruncationGuard, TokenWindows, WindowAggregation |
io.github.inference4j.exception |
Custom exceptions: ModelLoadException, InferenceException, InputTooLongException |
io.github.inference4j.tokenizer |
Tokenizer, EncodedInput, WordPieceTokenizer, BpeTokenizer, DecodingBpeTokenizer, SentencePieceBpeTokenizer, UnigramTokenizer, PrecompiledNormalizer, SpecialTokens, TokenDecoder, TokenStream |
io.github.inference4j.preprocessing.text |
ModelConfig (HuggingFace config.json parser) |
io.github.inference4j.preprocessing.image |
Image transforms pipeline: ImageTransformPipeline, ResizeTransform, CenterCropTransform, ImageLayout, Labels |
io.github.inference4j.preprocessing.audio |
AudioTransformPipeline, AudioTransform, AudioData, AudioLoader, AudioWriter, AudioProcessor |
io.github.inference4j.vision |
ResNetClassifier, EfficientNetClassifier, YoloV8Detector, Yolo26Detector, RtDetrDetector, CraftTextDetector, PaddleTextDetector, PaddleTextRecognizer, TextRecognizer, OcrPipeline, DbPostProcessor, QuadCrop, TextLine, RecognizedText, Point, DepthAnythingEstimator, Sam2Segmenter, Prompt, ImageFeatures, ImageEmbedder, ImageAnnotator, TensorImages, Colormap |
io.github.inference4j.audio |
MoonshineRecognizer, TranscriptionStream, TranscriptListener, TranscriptLine, Wav2Vec2Recognizer, PunctuatedSpeechRecognizer, SegmentedSpeechRecognizer, SileroVadDetector |
io.github.inference4j.nlp |
DistilBertTextClassifier, DebertaZeroShotClassifier, ZeroShotTextClassifier, DebertaPromptInjectionDetector, PromptInjectionDetector, InjectionCheck, SentenceTransformerEmbedder, MiniLMSearchReranker, BertNerRecognizer, GlinerEntityRecognizer, OnnxTextGenerator, FlanT5TextGenerator, BartSummarizer, MarianTranslator, CoeditGrammarCorrector, T5SqlGenerator, PunctCapSegModel, NamedEntityRecognizer, ZeroShotEntityRecognizer, NamedEntity, TextGenerator, Summarizer, Translator, GrammarCorrector, SqlGenerator, Language, PoolingStrategy, QueryDocumentPair |
io.github.inference4j.multimodal |
ClipClassifier, ClipImageEncoder, ClipTextEncoder |
io.github.inference4j.generation |
GenerativeTask, GenerationEngine, GenerationResult, GenerativeSession, EncoderDecoderSession, ChatTemplate, GenerativeModel |
io.github.inference4j.sampling |
LogitsProcessor, LogitsSampler, CategoricalSampler, GreedySampler |
inference4j-genai¶
| Package | Contents |
|---|---|
io.github.inference4j.genai |
AbstractGenerativeTask, ModelSources |
io.github.inference4j.genai.nlp |
TextGenerator (onnxruntime-genai backed) |
io.github.inference4j.genai.audio |
WhisperSpeechModel, WhisperTask |
io.github.inference4j.genai.vision |
VisionLanguageModel, VisionInput |
inference4j-runtime¶
| Package | Contents |
|---|---|
io.github.inference4j.routing |
ModelRouter, Route, RoutingStrategy, WeightedRoutingStrategy, RouterMetrics |
inference4j-langchain4j¶
| Package | Contents |
|---|---|
io.github.inference4j.langchain4j |
LangChain4j adapters: Inference4jEmbeddingModel, Inference4jScoringModel, Inference4jPiiRedactor, Inference4jPromptInjectionGuardrail, Inference4jOcrDocumentParser, Inference4jAudioDocumentParser |
Each adapter implements a LangChain4j interface over an inference4j model:
| Adapter | LangChain4j interface | Wraps |
|---|---|---|
Inference4jEmbeddingModel |
EmbeddingModel (extends DimensionAwareEmbeddingModel) |
SentenceTransformerEmbedder |
Inference4jScoringModel |
ScoringModel |
SearchReranker |
Inference4jPiiRedactor |
DocumentTransformer; asTextSegmentTransformer() gives a TextSegmentTransformer |
NamedEntityRecognizer |
Inference4jPromptInjectionGuardrail |
InputGuardrail |
PromptInjectionDetector |
Inference4jOcrDocumentParser |
DocumentParser |
OcrPipeline |
Inference4jAudioDocumentParser |
DocumentParser, plus parseSegments(InputStream) |
SpeechRecognizer |
All adapters are AutoCloseable and own the model they wrap (close() closes it). LangChain4j is
a compileOnly dependency of the module. See the LangChain4j guide.
inference4j-spring-boot-starter¶
| Package | Contents |
|---|---|
io.github.inference4j.autoconfigure |
Auto-configuration, health indicators, Inference4jProperties |
Interface hierarchy¶
InferenceTask<I, O> // run(I) → O, extends AutoCloseable
├── Classifier<I, C> // classify(I) → List<C>
│ ├── ImageClassifier // classify(BufferedImage/Path) → List<Classification>
│ └── TextClassifier // classify(String) → List<TextClassification>
│ // DistilBertTextClassifier, DebertaPromptInjectionDetector
├── ZeroShotClassifier<I, C> // classify(I, List<String>) → List<C>, run(ZeroShotInput<I>) → List<C>
│ └── ZeroShotTextClassifier // classify(String, List<String>) → List<TextClassification>
│ // DebertaZeroShotClassifier (NLI); ClipClassifier is the image counterpart
├── Detector<I, D> // detect(I) → List<D>
│ ├── ObjectDetector // detect(BufferedImage/Path) → List<Detection>
│ │ // YoloV8Detector, Yolo26Detector (AGPL-3.0); RtDetrDetector (Apache-2.0)
│ ├── TextDetector // detect(BufferedImage/Path) → List<TextRegion>
│ │ // CraftTextDetector (words), PaddleTextDetector (lines)
│ └── VoiceActivityDetector // detect(Path/float[]) → List<VoiceSegment>
├── NamedEntityRecognizer // recognize(String) → List<NamedEntity>
│ // BertNerRecognizer; GlinerEntityRecognizer (default labels)
├── TextRecognizer // recognize(BufferedImage) → RecognizedText (one line image)
├── OcrPipeline // read(BufferedImage/Path) → List<TextLine> (class: detect + crop + recognize)
├── DepthEstimator // estimate(BufferedImage/Path) → DepthMap
├── PromptableSegmenter // encode(BufferedImage/Path) → ImageFeatures
│ // segment(ImageFeatures, Prompt) → SegmentationMask; Sam2Segmenter
├── TextEmbedder // encode(String) → float[]
│ // SentenceTransformerEmbedder.encode(String, prefix) per-call prefix
├── ImageEmbedder // encode(BufferedImage/Path) → float[]
├── SearchReranker // score(String, String) → float
├── SpeechRecognizer // transcribe(Path) → Transcription
│ // MoonshineRecognizer, Wav2Vec2Recognizer,
│ // WhisperSpeechModel (inference4j-genai)
│ // MoonshineRecognizer.stream(vad) → TranscriptionStream (live)
│ // PunctuatedSpeechRecognizer decorates any recognizer
│ // SegmentedSpeechRecognizer cuts long audio at pauses
├── PunctCapSegModel // infer(String) → List<String> (class)
├── TextGenerator // generate(String) → GenerationResult
├── Summarizer // summarize(String) → String
├── Translator // translate(String) → String
├── GrammarCorrector // correct(String) → String
└── SqlGenerator // generateSql(String, String) → String
PromptInjectionDetector // check(String) → InjectionCheck, extends AutoCloseable
ZeroShotEntityRecognizer // recognize(String, List<String>) → List<NamedEntity>, extends AutoCloseable
// GlinerEntityRecognizer
AbstractInferenceTask¶
Most task wrappers extend AbstractInferenceTask<I, O>, which enforces a final run() method with three stages:
- Preprocess:
I → Map<String, Tensor>(viaPreprocessor) - Infer:
Map<String, Tensor> → Map<String, Tensor>(viaInferenceSession) - Postprocess:
InferenceContext<I> → O(viaPostprocessor)
InferenceContext<I> carries data across stages — the original input, preprocessed tensors, and output tensors.
Exceptions: SileroVadDetector (stateful hidden state), MoonshineRecognizer (six graphs),
MiniLMSearchReranker, ClipClassifier, and ClipTextEncoder do not extend AbstractInferenceTask.
Generative AI hierarchy¶
GenerativeTask<I, O> // generate(I) → O, extends AutoCloseable
└── AbstractGenerativeTask<I, O> // owns the autoregressive loop
├── TextGenerator // generate(String) → GenerationResult
├── WhisperSpeechModel // SpeechRecognizer: transcribe(Path) → Transcription
└── VisionLanguageModel // generate(VisionInput) → GenerationResult
GenerativeTask is the generative counterpart to InferenceTask. While InferenceTask
performs a single forward pass, GenerativeTask runs an iterative generate loop
backed by onnxruntime-genai. AbstractGenerativeTask owns the generate loop and
provides hooks for subclasses: prepareGenerator(), createStream(), parseOutput(),
and createParams() (for model-specific generation options like temperature and max length).
See Generative AI for details.
Result types¶
| Type | Fields | Used by |
|---|---|---|
Classification |
label(), index(), confidence() |
ResNetClassifier, EfficientNetClassifier, ClipClassifier |
TextClassification |
label(), index(), confidence() |
DistilBertTextClassifier, DebertaPromptInjectionDetector, DebertaZeroShotClassifier |
InjectionCheck |
probability(), threshold(), isInjection() |
DebertaPromptInjectionDetector |
NamedEntity |
text(), label(), start(), end(), score() |
BertNerRecognizer, GlinerEntityRecognizer |
Detection |
label(), classIndex(), confidence(), box() |
YoloV8Detector, Yolo26Detector, RtDetrDetector |
TextRegion |
box(), polygon(), confidence() |
CraftTextDetector, PaddleTextDetector |
TextLine |
text(), confidence(), region() |
OcrPipeline |
RecognizedText |
text(), confidence() |
PaddleTextRecognizer (any TextRecognizer) |
Point |
x(), y() |
Embedded in TextRegion.polygon() |
BoundingBox |
x1(), y1(), x2(), y2() |
Embedded in Detection, TextRegion |
DepthMap |
values(), width(), height(), at(), min(), max(), toImage() |
DepthAnythingEstimator |
SegmentationMask |
mask(), width(), height(), score(), contains(), area(), boundingBox(), applyTo(), overlay(), toImage() |
Sam2Segmenter |
Transcription |
text(), segments() |
MoonshineRecognizer, Wav2Vec2Recognizer, WhisperSpeechModel |
TranscriptLine |
id(), text(), start(), end(), completed() |
TranscriptionStream (via TranscriptListener) |
VoiceSegment |
start(), end(), duration(), confidence() |
SileroVadDetector |
GenerationResult |
text(), promptTokens(), generatedTokens(), duration() |
OnnxTextGenerator, FlanT5TextGenerator, BartSummarizer, MarianTranslator, CoeditGrammarCorrector, TextGenerator, VisionLanguageModel |
QueryDocumentPair |
query(), document() |
MiniLMSearchReranker, SearchReranker |
VisionInput |
imagePath(), prompt() |
VisionLanguageModel |
Builder pattern¶
All task wrappers follow a consistent builder pattern:
try (var task = SomeWrapper.builder()
.modelId("org/model-name") // HuggingFace model ID
.modelSource(new LocalModelSource(...)) // optional: custom model source
.sessionOptions(opts -> opts.addCoreML()) // optional: ONNX Runtime config
// ... task-specific options ...
.build()) {
task.run(input);
}
The .session(InferenceSession) method exists on all builders but is package-private — the public API uses modelId + modelSource + sessionOptions.
Thread safety¶
Models are expensive to build and cheap to call, so build each one once and share it, for example as a Spring singleton
bean. The split mirrors Hibernate's SessionFactory and Session:
Models are thread-safe; per-stream helpers are not
- Model wrappers and
InferenceSessionare thread-safe. One instance can serve many threads. - Most wrappers run calls in parallel. ONNX Runtime runs concurrent inferences on one session.
- Generators run calls one at a time. That covers
OnnxTextGenerator,FlanT5TextGenerator,BartSummarizer,MarianTranslator,CoeditGrammarCorrector,T5SqlGeneratorand theinference4j-genaitasks. Each keeps a KV cache for the length of a generation, which can take hundreds of MB, and frees it when the call ends. A thread waiting its turn can be interrupted. Callinggenerateagain from a token listener throwsIllegalStateException. MoonshineRecognizeralso runs one call at a time. ItsTranscriptionStreams share those turns withtranscribe, andTranscriptListenercallbacks run during the stream's turn, so keep them short.- Per-stream helpers are not thread-safe. That covers
TranscriptionStream,TokenStream,NativeTensors, and the low-levelOnnxGenerativeSessionandEncoderDecoderSession. Use each from one thread. - Closing is safe.
close()waits for calls in progress to finish. Later calls throwIllegalStateException. Closing a model from its own token or transcript listener throwsIllegalStateException.
Parallel generation and CPU threads
For parallel generation, build one generator per thread. Each instance loads its own copy of the weights, so
memory grows with every instance. Concurrent calls also share the CPU: by default each inference uses all cores
for one call. When you serve several calls at once, call opts.setIntraOpNumThreads(n) in
sessionOptions(...) with a lower n, so the calls do not compete for the same cores. See
Hardware acceleration.
Functional interfaces¶
| Interface | Signature | Description |
|---|---|---|
SessionConfigurer |
void configure(SessionOptions) throws OrtException |
ONNX Runtime session customization |
ModelSource |
Path resolve(String modelId) |
Model file resolution |
Preprocessor<I, O> |
O process(I input) |
Input transformation |
Postprocessor<I, O> |
O process(I input) |
Output transformation |
OutputOperator |
float[] apply(float[] values) |
Activation function (softmax, sigmoid, identity) |
ChatTemplate |
String format(String userMessage) |
Prompt formatting for generative models |