Skip to content

API Overview

Package structure

inference4j-core

Package Contents
io.github.inference4j Core contracts: InferenceTask, Classifier, Detector, ZeroShotClassifier, ZeroShotInput, AbstractInferenceTask, Tensor, TensorType, InferenceSession, NativeTensors, InferenceContext
io.github.inference4j.session Session config: SessionConfigurer, SessionOptions
io.github.inference4j.model Model resolution: ModelSource, HuggingFaceModelSource, LocalModelSource
io.github.inference4j.processing Pre/post-processing: Preprocessor, Postprocessor, OutputOperator, MathOps, TruncationPolicy, TruncationGuard, TokenWindows, WindowAggregation
io.github.inference4j.exception Custom exceptions: ModelLoadException, InferenceException, InputTooLongException
io.github.inference4j.tokenizer Tokenizer, EncodedInput, WordPieceTokenizer, BpeTokenizer, DecodingBpeTokenizer, SentencePieceBpeTokenizer, UnigramTokenizer, PrecompiledNormalizer, SpecialTokens, TokenDecoder, TokenStream
io.github.inference4j.preprocessing.text ModelConfig (HuggingFace config.json parser)
io.github.inference4j.preprocessing.image Image transforms pipeline: ImageTransformPipeline, ResizeTransform, CenterCropTransform, ImageLayout, Labels
io.github.inference4j.preprocessing.audio AudioTransformPipeline, AudioTransform, AudioData, AudioLoader, AudioWriter, AudioProcessor
io.github.inference4j.vision ResNetClassifier, EfficientNetClassifier, YoloV8Detector, Yolo26Detector, RtDetrDetector, CraftTextDetector, PaddleTextDetector, PaddleTextRecognizer, TextRecognizer, OcrPipeline, DbPostProcessor, QuadCrop, TextLine, RecognizedText, Point, DepthAnythingEstimator, Sam2Segmenter, Prompt, ImageFeatures, ImageEmbedder, ImageAnnotator, TensorImages, Colormap
io.github.inference4j.audio MoonshineRecognizer, TranscriptionStream, TranscriptListener, TranscriptLine, Wav2Vec2Recognizer, PunctuatedSpeechRecognizer, SegmentedSpeechRecognizer, SileroVadDetector
io.github.inference4j.nlp DistilBertTextClassifier, DebertaZeroShotClassifier, ZeroShotTextClassifier, DebertaPromptInjectionDetector, PromptInjectionDetector, InjectionCheck, SentenceTransformerEmbedder, MiniLMSearchReranker, BertNerRecognizer, GlinerEntityRecognizer, OnnxTextGenerator, FlanT5TextGenerator, BartSummarizer, MarianTranslator, CoeditGrammarCorrector, T5SqlGenerator, PunctCapSegModel, NamedEntityRecognizer, ZeroShotEntityRecognizer, NamedEntity, TextGenerator, Summarizer, Translator, GrammarCorrector, SqlGenerator, Language, PoolingStrategy, QueryDocumentPair
io.github.inference4j.multimodal ClipClassifier, ClipImageEncoder, ClipTextEncoder
io.github.inference4j.generation GenerativeTask, GenerationEngine, GenerationResult, GenerativeSession, EncoderDecoderSession, ChatTemplate, GenerativeModel
io.github.inference4j.sampling LogitsProcessor, LogitsSampler, CategoricalSampler, GreedySampler

inference4j-genai

Package Contents
io.github.inference4j.genai AbstractGenerativeTask, ModelSources
io.github.inference4j.genai.nlp TextGenerator (onnxruntime-genai backed)
io.github.inference4j.genai.audio WhisperSpeechModel, WhisperTask
io.github.inference4j.genai.vision VisionLanguageModel, VisionInput

inference4j-runtime

Package Contents
io.github.inference4j.routing ModelRouter, Route, RoutingStrategy, WeightedRoutingStrategy, RouterMetrics

inference4j-langchain4j

Package Contents
io.github.inference4j.langchain4j LangChain4j adapters: Inference4jEmbeddingModel, Inference4jScoringModel, Inference4jPiiRedactor, Inference4jPromptInjectionGuardrail, Inference4jOcrDocumentParser, Inference4jAudioDocumentParser

Each adapter implements a LangChain4j interface over an inference4j model:

Adapter LangChain4j interface Wraps
Inference4jEmbeddingModel EmbeddingModel (extends DimensionAwareEmbeddingModel) SentenceTransformerEmbedder
Inference4jScoringModel ScoringModel SearchReranker
Inference4jPiiRedactor DocumentTransformer; asTextSegmentTransformer() gives a TextSegmentTransformer NamedEntityRecognizer
Inference4jPromptInjectionGuardrail InputGuardrail PromptInjectionDetector
Inference4jOcrDocumentParser DocumentParser OcrPipeline
Inference4jAudioDocumentParser DocumentParser, plus parseSegments(InputStream) SpeechRecognizer

All adapters are AutoCloseable and own the model they wrap (close() closes it). LangChain4j is a compileOnly dependency of the module. See the LangChain4j guide.

inference4j-spring-boot-starter

Package Contents
io.github.inference4j.autoconfigure Auto-configuration, health indicators, Inference4jProperties

Interface hierarchy

InferenceTask<I, O>                     // run(I) → O, extends AutoCloseable
├── Classifier<I, C>                    // classify(I) → List<C>
│   ├── ImageClassifier                 // classify(BufferedImage/Path) → List<Classification>
│   └── TextClassifier                  // classify(String) → List<TextClassification>
│                                       //   DistilBertTextClassifier, DebertaPromptInjectionDetector
├── ZeroShotClassifier<I, C>            // classify(I, List<String>) → List<C>, run(ZeroShotInput<I>) → List<C>
│   └── ZeroShotTextClassifier          // classify(String, List<String>) → List<TextClassification>
│                                       //   DebertaZeroShotClassifier (NLI); ClipClassifier is the image counterpart
├── Detector<I, D>                      // detect(I) → List<D>
│   ├── ObjectDetector                  // detect(BufferedImage/Path) → List<Detection>
│   │                                   //   YoloV8Detector, Yolo26Detector (AGPL-3.0); RtDetrDetector (Apache-2.0)
│   ├── TextDetector                    // detect(BufferedImage/Path) → List<TextRegion>
│   │                                   //   CraftTextDetector (words), PaddleTextDetector (lines)
│   └── VoiceActivityDetector           // detect(Path/float[]) → List<VoiceSegment>
├── NamedEntityRecognizer               // recognize(String) → List<NamedEntity>
│                                       //   BertNerRecognizer; GlinerEntityRecognizer (default labels)
├── TextRecognizer                      // recognize(BufferedImage) → RecognizedText  (one line image)
├── OcrPipeline                         // read(BufferedImage/Path) → List<TextLine>  (class: detect + crop + recognize)
├── DepthEstimator                      // estimate(BufferedImage/Path) → DepthMap
├── PromptableSegmenter                 // encode(BufferedImage/Path) → ImageFeatures
│                                       //   segment(ImageFeatures, Prompt) → SegmentationMask; Sam2Segmenter
├── TextEmbedder                        // encode(String) → float[]
│                                       //   SentenceTransformerEmbedder.encode(String, prefix) per-call prefix
├── ImageEmbedder                       // encode(BufferedImage/Path) → float[]
├── SearchReranker                      // score(String, String) → float
├── SpeechRecognizer                    // transcribe(Path) → Transcription
│                                       //   MoonshineRecognizer, Wav2Vec2Recognizer,
│                                       //   WhisperSpeechModel (inference4j-genai)
│                                       //   MoonshineRecognizer.stream(vad) → TranscriptionStream (live)
│                                       //   PunctuatedSpeechRecognizer decorates any recognizer
│                                       //   SegmentedSpeechRecognizer cuts long audio at pauses
├── PunctCapSegModel                    // infer(String) → List<String>  (class)
├── TextGenerator                       // generate(String) → GenerationResult
├── Summarizer                          // summarize(String) → String
├── Translator                          // translate(String) → String
├── GrammarCorrector                    // correct(String) → String
└── SqlGenerator                        // generateSql(String, String) → String

PromptInjectionDetector                 // check(String) → InjectionCheck, extends AutoCloseable
ZeroShotEntityRecognizer                // recognize(String, List<String>) → List<NamedEntity>, extends AutoCloseable
                                        //   GlinerEntityRecognizer

AbstractInferenceTask

Most task wrappers extend AbstractInferenceTask<I, O>, which enforces a final run() method with three stages:

  1. Preprocess: I → Map<String, Tensor> (via Preprocessor)
  2. Infer: Map<String, Tensor> → Map<String, Tensor> (via InferenceSession)
  3. Postprocess: InferenceContext<I> → O (via Postprocessor)

InferenceContext<I> carries data across stages — the original input, preprocessed tensors, and output tensors.

Exceptions: SileroVadDetector (stateful hidden state), MoonshineRecognizer (six graphs), MiniLMSearchReranker, ClipClassifier, and ClipTextEncoder do not extend AbstractInferenceTask.

Generative AI hierarchy

GenerativeTask<I, O>                    // generate(I) → O, extends AutoCloseable
└── AbstractGenerativeTask<I, O>        // owns the autoregressive loop
    ├── TextGenerator                   // generate(String) → GenerationResult
    ├── WhisperSpeechModel              // SpeechRecognizer: transcribe(Path) → Transcription
    └── VisionLanguageModel             // generate(VisionInput) → GenerationResult

GenerativeTask is the generative counterpart to InferenceTask. While InferenceTask performs a single forward pass, GenerativeTask runs an iterative generate loop backed by onnxruntime-genai. AbstractGenerativeTask owns the generate loop and provides hooks for subclasses: prepareGenerator(), createStream(), parseOutput(), and createParams() (for model-specific generation options like temperature and max length). See Generative AI for details.

Result types

Type Fields Used by
Classification label(), index(), confidence() ResNetClassifier, EfficientNetClassifier, ClipClassifier
TextClassification label(), index(), confidence() DistilBertTextClassifier, DebertaPromptInjectionDetector, DebertaZeroShotClassifier
InjectionCheck probability(), threshold(), isInjection() DebertaPromptInjectionDetector
NamedEntity text(), label(), start(), end(), score() BertNerRecognizer, GlinerEntityRecognizer
Detection label(), classIndex(), confidence(), box() YoloV8Detector, Yolo26Detector, RtDetrDetector
TextRegion box(), polygon(), confidence() CraftTextDetector, PaddleTextDetector
TextLine text(), confidence(), region() OcrPipeline
RecognizedText text(), confidence() PaddleTextRecognizer (any TextRecognizer)
Point x(), y() Embedded in TextRegion.polygon()
BoundingBox x1(), y1(), x2(), y2() Embedded in Detection, TextRegion
DepthMap values(), width(), height(), at(), min(), max(), toImage() DepthAnythingEstimator
SegmentationMask mask(), width(), height(), score(), contains(), area(), boundingBox(), applyTo(), overlay(), toImage() Sam2Segmenter
Transcription text(), segments() MoonshineRecognizer, Wav2Vec2Recognizer, WhisperSpeechModel
TranscriptLine id(), text(), start(), end(), completed() TranscriptionStream (via TranscriptListener)
VoiceSegment start(), end(), duration(), confidence() SileroVadDetector
GenerationResult text(), promptTokens(), generatedTokens(), duration() OnnxTextGenerator, FlanT5TextGenerator, BartSummarizer, MarianTranslator, CoeditGrammarCorrector, TextGenerator, VisionLanguageModel
QueryDocumentPair query(), document() MiniLMSearchReranker, SearchReranker
VisionInput imagePath(), prompt() VisionLanguageModel

Builder pattern

All task wrappers follow a consistent builder pattern:

try (var task = SomeWrapper.builder()
        .modelId("org/model-name")              // HuggingFace model ID
        .modelSource(new LocalModelSource(...)) // optional: custom model source
        .sessionOptions(opts -> opts.addCoreML()) // optional: ONNX Runtime config
        // ... task-specific options ...
        .build()) {
    task.run(input);
}

The .session(InferenceSession) method exists on all builders but is package-private — the public API uses modelId + modelSource + sessionOptions.

Thread safety

Models are expensive to build and cheap to call, so build each one once and share it, for example as a Spring singleton bean. The split mirrors Hibernate's SessionFactory and Session:

Models are thread-safe; per-stream helpers are not

  • Model wrappers and InferenceSession are thread-safe. One instance can serve many threads.
  • Most wrappers run calls in parallel. ONNX Runtime runs concurrent inferences on one session.
  • Generators run calls one at a time. That covers OnnxTextGenerator, FlanT5TextGenerator, BartSummarizer, MarianTranslator, CoeditGrammarCorrector, T5SqlGenerator and the inference4j-genai tasks. Each keeps a KV cache for the length of a generation, which can take hundreds of MB, and frees it when the call ends. A thread waiting its turn can be interrupted. Calling generate again from a token listener throws IllegalStateException.
  • MoonshineRecognizer also runs one call at a time. Its TranscriptionStreams share those turns with transcribe, and TranscriptListener callbacks run during the stream's turn, so keep them short.
  • Per-stream helpers are not thread-safe. That covers TranscriptionStream, TokenStream, NativeTensors, and the low-level OnnxGenerativeSession and EncoderDecoderSession. Use each from one thread.
  • Closing is safe. close() waits for calls in progress to finish. Later calls throw IllegalStateException. Closing a model from its own token or transcript listener throws IllegalStateException.

Parallel generation and CPU threads

For parallel generation, build one generator per thread. Each instance loads its own copy of the weights, so memory grows with every instance. Concurrent calls also share the CPU: by default each inference uses all cores for one call. When you serve several calls at once, call opts.setIntraOpNumThreads(n) in sessionOptions(...) with a lower n, so the calls do not compete for the same cores. See Hardware acceleration.

Functional interfaces

Interface Signature Description
SessionConfigurer void configure(SessionOptions) throws OrtException ONNX Runtime session customization
ModelSource Path resolve(String modelId) Model file resolution
Preprocessor<I, O> O process(I input) Input transformation
Postprocessor<I, O> O process(I input) Output transformation
OutputOperator float[] apply(float[] values) Activation function (softmax, sigmoid, identity)
ChatTemplate String format(String userMessage) Prompt formatting for generative models