Skip to content

Hardware Acceleration

inference4j supports GPU and hardware acceleration via ONNX Runtime execution providers. The .sessionOptions() API is available on every model wrapper.

CoreML (macOS)

CoreML is bundled in the standard ONNX Runtime dependency on macOS — no additional setup needed.

try (var classifier = ResNetClassifier.builder()
        .sessionOptions(opts -> opts.addCoreML())
        .build()) {
    classifier.classify(Path.of("cat.jpg"));
}

CoreML results can differ from CPU results

addCoreML() converts the model to CoreML's older NeuralNetwork format. CoreML runs it on the Apple GPU or Neural Engine with reduced numeric precision, so scores shift slightly, and for some models the output changes. Models that pick their top results inside the graph, such as YOLO26's 300 proposals, are the most affected: a small change in scores changes which boxes survive.

The newer MLProgram format keeps CPU-level accuracy for the models that support it:

try (var detector = RtDetrDetector.builder()
        .sessionOptions(opts -> opts.addCoreML(Map.of("ModelFormat", "MLProgram")))
        .build()) {
    detector.detect(Path.of("street.jpg"));
}

Other CoreML provider options, such as MLComputeUnits (CPUAndGPU, CPUAndNeuralEngine, ALL), go in the same map; see the ONNX Runtime CoreML documentation.

Not every model speeds up, and not every model converts. Measured with inference4j on Apple silicon (ONNX Runtime 1.28, warm runs, per image):

Model CPU addCoreML() (NeuralNetwork) MLProgram
CRAFT (CraftTextDetector) 587 ms 214 ms, same regions fails to load
YOLO26n (Yolo26Detector) 26 ms 15 ms, different detections 25 ms, same as CPU
D-FINE small (RtDetrDetector) 57 ms 112 ms, scores within 0.01 102 ms, same as CPU
RT-DETRv2 (RtDetrDetector.rtDetrV2()) 106 ms 122 ms, some labels differ 98 ms, same as CPU
D-FINE nano (RtDetrDetector.dfineNano()) 32 ms fails when run fails to load
PP-OCRv6 small (OcrPipeline) ~400 ms ~860 ms not measured

Small, CPU-optimized models (YOLO26n, D-FINE, PP-OCRv6) gain little or nothing from CoreML: ONNX Runtime splits them into many CoreML and CPU pieces, and copying data between them costs more than the GPU saves. Large convolutional networks such as CRAFT gain the most. Measure your model on your hardware, and compare its output with the CPU's, before switching.

CUDA (Linux/Windows)

For NVIDIA GPU acceleration, swap the ONNX Runtime dependency:

implementation('io.github.inference4j:inference4j-core:${inference4jVersion}') {
    exclude group: 'com.microsoft.onnxruntime', module: 'onnxruntime'
}
implementation 'com.microsoft.onnxruntime:onnxruntime_gpu:${onnxruntimeVersion}'
<dependency>
    <groupId>io.github.inference4j</groupId>
    <artifactId>inference4j-core</artifactId>
    <version>${inference4jVersion}</version>
    <exclusions>
        <exclusion>
            <groupId>com.microsoft.onnxruntime</groupId>
            <artifactId>onnxruntime</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>com.microsoft.onnxruntime</groupId>
    <artifactId>onnxruntime_gpu</artifactId>
    <version>${onnxruntimeVersion}</version>
</dependency>

Then enable CUDA in the builder:

try (var classifier = ResNetClassifier.builder()
        .sessionOptions(opts -> opts.addCUDA(0))  // device ID 0
        .build()) {
    classifier.classify(Path.of("cat.jpg"));
}

Plugin execution providers

ONNX Runtime can also load execution providers from separately packaged plugin libraries, rather than only the ones compiled into the runtime. Microsoft ships several this way — the CUDA and WebGPU plugin EPs, for example — versioned independently of ONNX Runtime itself.

This is additive. addCoreML() and addCUDA() above are unaffected and remain the simplest path. Reach for a plugin EP when you need a backend that isn't built into your ONNX Runtime distribution.

Register the plugin library

Registration happens on the OrtEnvironment, not on the session. inference4j uses ONNX Runtime's default environment, so anything you register there applies to every model you build afterwards:

OrtEnvironment env = OrtEnvironment.getEnvironment();
env.registerExecutionProviderLibrary("my_ep", "/path/to/plugin_ep_library.so");

Pick devices and attach them

getEpDevices() lists every execution provider / device combination available, including ones contributed by registered plugins. Filter down to the provider you want, then pass the result to addExecutionProvider:

OrtEnvironment env = OrtEnvironment.getEnvironment();

List<OrtEpDevice> devices = env.getEpDevices().stream()
        .filter(d -> "my_ep".equals(d.getEpName()))
        .toList();

try (var classifier = ResNetClassifier.builder()
        .sessionOptions(opts -> opts.addExecutionProvider(devices, Map.of()))
        .build()) {
    classifier.classify(Path.of("cat.jpg"));
}

Two rules worth knowing:

  • Every OrtEpDevice in the list must belong to the same execution provider, though they may point at different devices.
  • Providers apply in the order added — the first provider added gets first choice when graph nodes are assigned.

Inspect what's available

OrtEpDevice exposes getEpName(), getEpVendor(), getEpMetadata(), getEpOptions() and getDevice(), which is enough to see what a machine actually offers:

for (OrtEpDevice d : OrtEnvironment.getEnvironment().getEpDevices()) {
    System.out.println(d.getEpName() + " (" + d.getEpVendor() + ") -> " + d.getDevice());
}

To remove a plugin again:

env.unregisterExecutionProviderLibrary("my_ep");

Requires ONNX Runtime 1.24.3 or later

registerExecutionProviderLibrary(), getEpDevices() and addExecutionProvider() are not present in 1.23. inference4j depends on ONNX Runtime 1.28.0 as of 0.13.0, so they are available with no extra setup.

The sessionOptions API

Every model wrapper exposes .sessionOptions(SessionConfigurer) in its builder. SessionConfigurer is a @FunctionalInterface that receives the ONNX Runtime SessionOptions:

@FunctionalInterface
public interface SessionConfigurer {
    void configure(OrtSession.SessionOptions options) throws OrtException;
}

This gives you full access to ONNX Runtime configuration:

.sessionOptions(opts -> {
    opts.addCoreML();
    opts.setIntraOpNumThreads(4);
})

Common options

Method Description
opts.addCoreML() Enable CoreML (macOS)
opts.addCUDA(deviceId) Enable CUDA (Linux/Windows)
opts.addExecutionProvider(devices, options) Enable a plugin execution provider
opts.setIntraOpNumThreads(n) Set number of threads for intra-op parallelism
opts.setInterOpNumThreads(n) Set number of threads for inter-op parallelism
opts.setOptimizationLevel(level) Set graph optimization level

Benchmarks on Apple Silicon (M-series)

Model Capability CPU CoreML Speedup
ResNet-50 Image Classification 37 ms 10 ms 3.7x
CRAFT Text Detection 831 ms 153 ms 5.4x

Measured with 3 warmup runs + 10 timed runs.

Tips

  • CoreML is available on macOS only. The addCoreML() call will fail on other platforms.
  • CUDA requires the onnxruntime_gpu artifact and a compatible NVIDIA driver + CUDA toolkit.
  • If the execution provider fails to initialize, ONNX Runtime silently falls back to CPU. Check logs for warnings.
  • For production workloads, benchmark both CPU and GPU — small models (like MiniLM) may be faster on CPU due to GPU data transfer overhead.
  • .sessionOptions() is composable — you can set multiple options in a single lambda.