Hardware Acceleration¶
inference4j supports GPU and hardware acceleration via ONNX Runtime execution providers. The .sessionOptions() API is available on every model wrapper.
CoreML (macOS)¶
CoreML is bundled in the standard ONNX Runtime dependency on macOS — no additional setup needed.
try (var classifier = ResNetClassifier.builder()
.sessionOptions(opts -> opts.addCoreML())
.build()) {
classifier.classify(Path.of("cat.jpg"));
}
CoreML results can differ from CPU results¶
addCoreML() converts the model to CoreML's older NeuralNetwork format. CoreML runs it on the Apple GPU or Neural
Engine with reduced numeric precision, so scores shift slightly, and for some models the output changes. Models that
pick their top results inside the graph, such as YOLO26's 300 proposals, are the most affected: a small change in scores
changes which boxes survive.
The newer MLProgram format keeps CPU-level accuracy for the models that support it:
try (var detector = RtDetrDetector.builder()
.sessionOptions(opts -> opts.addCoreML(Map.of("ModelFormat", "MLProgram")))
.build()) {
detector.detect(Path.of("street.jpg"));
}
Other CoreML provider options, such as MLComputeUnits (CPUAndGPU, CPUAndNeuralEngine, ALL), go in the same
map; see the ONNX Runtime CoreML documentation.
Not every model speeds up, and not every model converts. Measured with inference4j on Apple silicon (ONNX Runtime 1.28, warm runs, per image):
| Model | CPU | addCoreML() (NeuralNetwork) |
MLProgram |
|---|---|---|---|
CRAFT (CraftTextDetector) |
587 ms | 214 ms, same regions | fails to load |
YOLO26n (Yolo26Detector) |
26 ms | 15 ms, different detections | 25 ms, same as CPU |
D-FINE small (RtDetrDetector) |
57 ms | 112 ms, scores within 0.01 | 102 ms, same as CPU |
RT-DETRv2 (RtDetrDetector.rtDetrV2()) |
106 ms | 122 ms, some labels differ | 98 ms, same as CPU |
D-FINE nano (RtDetrDetector.dfineNano()) |
32 ms | fails when run | fails to load |
PP-OCRv6 small (OcrPipeline) |
~400 ms | ~860 ms | not measured |
Small, CPU-optimized models (YOLO26n, D-FINE, PP-OCRv6) gain little or nothing from CoreML: ONNX Runtime splits them into many CoreML and CPU pieces, and copying data between them costs more than the GPU saves. Large convolutional networks such as CRAFT gain the most. Measure your model on your hardware, and compare its output with the CPU's, before switching.
CUDA (Linux/Windows)¶
For NVIDIA GPU acceleration, swap the ONNX Runtime dependency:
<dependency>
<groupId>io.github.inference4j</groupId>
<artifactId>inference4j-core</artifactId>
<version>${inference4jVersion}</version>
<exclusions>
<exclusion>
<groupId>com.microsoft.onnxruntime</groupId>
<artifactId>onnxruntime</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.microsoft.onnxruntime</groupId>
<artifactId>onnxruntime_gpu</artifactId>
<version>${onnxruntimeVersion}</version>
</dependency>
Then enable CUDA in the builder:
try (var classifier = ResNetClassifier.builder()
.sessionOptions(opts -> opts.addCUDA(0)) // device ID 0
.build()) {
classifier.classify(Path.of("cat.jpg"));
}
Plugin execution providers¶
ONNX Runtime can also load execution providers from separately packaged plugin libraries, rather than only the ones compiled into the runtime. Microsoft ships several this way — the CUDA and WebGPU plugin EPs, for example — versioned independently of ONNX Runtime itself.
This is additive. addCoreML() and addCUDA() above are unaffected and remain the simplest path. Reach for a plugin EP when you need a backend that isn't built into your ONNX Runtime distribution.
Register the plugin library¶
Registration happens on the OrtEnvironment, not on the session. inference4j uses ONNX Runtime's default environment, so anything you register there applies to every model you build afterwards:
OrtEnvironment env = OrtEnvironment.getEnvironment();
env.registerExecutionProviderLibrary("my_ep", "/path/to/plugin_ep_library.so");
Pick devices and attach them¶
getEpDevices() lists every execution provider / device combination available, including ones contributed by registered plugins. Filter down to the provider you want, then pass the result to addExecutionProvider:
OrtEnvironment env = OrtEnvironment.getEnvironment();
List<OrtEpDevice> devices = env.getEpDevices().stream()
.filter(d -> "my_ep".equals(d.getEpName()))
.toList();
try (var classifier = ResNetClassifier.builder()
.sessionOptions(opts -> opts.addExecutionProvider(devices, Map.of()))
.build()) {
classifier.classify(Path.of("cat.jpg"));
}
Two rules worth knowing:
- Every
OrtEpDevicein the list must belong to the same execution provider, though they may point at different devices. - Providers apply in the order added — the first provider added gets first choice when graph nodes are assigned.
Inspect what's available¶
OrtEpDevice exposes getEpName(), getEpVendor(), getEpMetadata(), getEpOptions() and getDevice(), which is enough to see what a machine actually offers:
for (OrtEpDevice d : OrtEnvironment.getEnvironment().getEpDevices()) {
System.out.println(d.getEpName() + " (" + d.getEpVendor() + ") -> " + d.getDevice());
}
To remove a plugin again:
Requires ONNX Runtime 1.24.3 or later
registerExecutionProviderLibrary(), getEpDevices() and addExecutionProvider() are not
present in 1.23. inference4j depends on ONNX Runtime 1.28.0 as of 0.13.0, so they are
available with no extra setup.
The sessionOptions API¶
Every model wrapper exposes .sessionOptions(SessionConfigurer) in its builder. SessionConfigurer is a @FunctionalInterface that receives the ONNX Runtime SessionOptions:
@FunctionalInterface
public interface SessionConfigurer {
void configure(OrtSession.SessionOptions options) throws OrtException;
}
This gives you full access to ONNX Runtime configuration:
Common options¶
| Method | Description |
|---|---|
opts.addCoreML() |
Enable CoreML (macOS) |
opts.addCUDA(deviceId) |
Enable CUDA (Linux/Windows) |
opts.addExecutionProvider(devices, options) |
Enable a plugin execution provider |
opts.setIntraOpNumThreads(n) |
Set number of threads for intra-op parallelism |
opts.setInterOpNumThreads(n) |
Set number of threads for inter-op parallelism |
opts.setOptimizationLevel(level) |
Set graph optimization level |
Benchmarks on Apple Silicon (M-series)¶
| Model | Capability | CPU | CoreML | Speedup |
|---|---|---|---|---|
| ResNet-50 | Image Classification | 37 ms | 10 ms | 3.7x |
| CRAFT | Text Detection | 831 ms | 153 ms | 5.4x |
Measured with 3 warmup runs + 10 timed runs.
Tips¶
- CoreML is available on macOS only. The
addCoreML()call will fail on other platforms. - CUDA requires the
onnxruntime_gpuartifact and a compatible NVIDIA driver + CUDA toolkit. - If the execution provider fails to initialize, ONNX Runtime silently falls back to CPU. Check logs for warnings.
- For production workloads, benchmark both CPU and GPU — small models (like MiniLM) may be faster on CPU due to GPU data transfer overhead.
.sessionOptions()is composable — you can set multiple options in a single lambda.