Audio document parser¶
Inference4jAudioDocumentParser is a LangChain4j DocumentParser that turns a recording into text documents with
inference4j speech-to-text, in-process. Segments carry their start and end time in
the recording.
Quick example¶
try (var parser = new Inference4jAudioDocumentParser(new SegmentedSpeechRecognizer(
MoonshineRecognizer.builder().build(),
SileroVadDetector.builder().build()))) {
// One document with the whole transcript
Document transcript = FileSystemDocumentLoader.loadDocument(Path.of("meeting.wav"), parser);
// One document per timed segment
try (InputStream in = Files.newInputStream(Path.of("meeting.wav"))) {
List<Document> segments = parser.parseSegments(in);
}
}
SegmentedSpeechRecognizer cuts long recordings at pauses and produces the timed segments. Whisper requires it
for audio longer than 30 seconds.
Two ways to parse¶
| Method | Returns | Use it for |
|---|---|---|
parse(InputStream) |
One Document with the whole transcript, no timing metadata |
LangChain4j document loaders; the transcript is split by a document splitter |
parseSegments(InputStream) |
One Document per segment, in recording order, with timing metadata |
Retrieval with the time of each segment |
parseSegments is not part of LangChain4j's DocumentParser contract (one stream, one document), so call it
directly and pass the result to the ingestor:
try (InputStream in = Files.newInputStream(Path.of("meeting.wav"))) {
EmbeddingStoreIngestor.builder()
.embeddingModel(embeddingModel)
.embeddingStore(store)
.build()
.ingest(parser.parseSegments(in));
}
Without a document splitter, the ingestor stores each segment document as one text segment, with its metadata.
Segment metadata¶
| Key | Type | Value |
|---|---|---|
segment_index (Inference4jAudioDocumentParser.SEGMENT_INDEX) |
int |
Position of the segment in the recording, from 0 |
start_ms (Inference4jAudioDocumentParser.START_MS) |
long |
Where the segment starts, in milliseconds |
end_ms (Inference4jAudioDocumentParser.END_MS) |
long |
Where the segment ends, in milliseconds |
An InputStream has no file name. To identify the recording, add a metadata key:
List<Document> segments = parser.parseSegments(in);
segments.forEach(segment -> segment.metadata().put("file_name", "meeting.wav"));
The metadata is returned with the retrieved content:
interface Assistant {
Result<String> chat(String message);
}
Result<String> result = assistant.chat("What did we decide about the launch date?");
for (Content source : result.sources()) {
long startMs = source.textSegment().metadata().getLong(Inference4jAudioDocumentParser.START_MS);
System.out.printf("%s at %d:%02d%n",
source.textSegment().metadata().getString("file_name"), startMs / 60_000, startMs / 1000 % 60);
}
Choosing a recognizer¶
The parser has one constructor, Inference4jAudioDocumentParser(SpeechRecognizer), and accepts any inference4j
recognizer.
| Recognizer | Module | Size | Output | Notes |
|---|---|---|---|---|
MoonshineRecognizer |
inference4j-core |
~140 MB (small, default) | Punctuated, cased | English |
WhisperSpeechModel |
inference4j-genai |
~225 MB (tiny) to ~1.1 GB (small) | Punctuated, cased | 99 languages; must be wrapped in SegmentedSpeechRecognizer for audio over 30 seconds |
Wav2Vec2Recognizer |
inference4j-core |
~370 MB | Upper-case, no punctuation | English. Combine with PunctuatedSpeechRecognizer to restore punctuation and casing |
// Wav2Vec2 with punctuation restored, cut at pauses
var parser = new Inference4jAudioDocumentParser(new SegmentedSpeechRecognizer(
new PunctuatedSpeechRecognizer(
Wav2Vec2Recognizer.builder().build(),
PunctCapSegModel.builder().build()),
SileroVadDetector.builder().build()));
Segment length is set on SegmentedSpeechRecognizer: pieces end at pauses and are at most 30 seconds by default.
Pass a Duration as the third constructor argument for shorter segments. See
Long recordings.
Notes¶
- WAV only. Input must be WAV (8- or 16-bit PCM, mono or stereo, any sample rate). Convert MP3, M4A and video
first, for example
ffmpeg -i talk.mp4 -ac 1 -ar 16000 talk.wav. Other input fails withInferenceException. - Silence.
parsethrowsInferenceException("No speech detected in audio") because LangChain4j documents cannot be blank.parseSegmentsreturns an empty list. - Segment boundaries. A segment spans one piece of audio between pauses, including up to half a pause of silence at each end. There is no speaker identification.
- Without
SegmentedSpeechRecognizer,parseSegmentsreturns a single document spanning the whole recording. - Memory and disk. The stream is copied to a temporary file, which is deleted afterwards, and the decoded audio is held in memory while it is transcribed.
- Lifecycle.
close()closes the recognizer and, throughSegmentedSpeechRecognizer, the models it wraps. See Lifecycle.
inference4j does not implement LangChain4j's AudioTranscriptionModel interface, whose response carries text
without timestamps.