How we built private Whisper transcription entirely in the browser

Why local transcription?

Audio transcription usually begins with an upload. That is convenient for a service operator, but it creates a privacy and infrastructure problem: recordings can contain meetings, interviews, research, customer conversations, or personal notes, and every minute must be transferred and processed somewhere.

LocalScribe takes a different route. The website converts files and microphone recordings to text inside the browser. Its Chrome extension captures audio from a tab the user explicitly authorizes and runs the same kind of local speech-recognition workflow from an offscreen extension document. There is no LocalScribe transcription endpoint and no account is required.

This decision moves cost and complexity from a server fleet to the client. It also introduces hard constraints: browsers have limited memory, model files are large, media codecs vary, workers have different APIs from pages, and Manifest V3 service workers cannot own a long-running audio pipeline. The architecture is largely a response to those constraints.

1. Decode once, normalize early

The transcription pipeline accepts browser-supported audio and video files. We pass the selected file to AudioContext.decodeAudioData(), mix every decoded channel into a single mono channel, and resample it to 16 kHz with OfflineAudioContext. Whisper then receives a predictable Float32Array regardless of the source file’s channel count or sample rate.

file → decodeAudioData → mono mix → 16 kHz resample → Float32Array

The array buffer is transferred to a dedicated module worker instead of copied. This avoids one full duplicate when crossing the main-thread boundary. It does not make memory use constant: the current design still decodes the complete recording before inference, so very long media can put pressure on browser memory.

2. Keep inference away from the interface

Whisper initialization and transcription run in a Web Worker. That keeps model loading and long inference calls away from React’s rendering work and gives the interface a simple message protocol for setup progress, transcription progress, results, and errors.

The worker retains a loaded pipeline while the selected model and execution path remain unchanged. Switching models disposes the old pipeline before creating another. The website chooses WebGPU with FP16 when the browser exposes a compatible adapter with shader-f16; otherwise it uses quantized WASM. This is an implementation choice, not a promise that one path will be fastest on every device.

3. Split long recordings with overlap

We process audio sequentially in 30-second windows. Each new window begins 25 seconds after the previous one, creating a five-second overlap:

window length: 30 seconds
overlap:        5 seconds
stride:         25 seconds

The overlap gives Whisper context around boundaries instead of making every cut absolute. For each window, the worker requests segment timestamps, adds the window’s absolute offset, and filters later-window segments that do not extend beyond the overlap boundary. This is a pragmatic duplicate-reduction rule—not word-level alignment—and difficult boundary speech can still produce omissions or repeated phrases.

Processing remains sequential. The progress percentage reflects completed windows, not a prediction of remaining wall-clock time.

4. Treat timestamps as structured output

The worker returns both a clean text string and timestamped Whisper segments. LocalScribe uses the segments for an on-screen timestamp view and for SRT and WebVTT generation. SRT timestamps use comma-separated milliseconds; WebVTT uses dots.

The plain transcript is editable. Timed exports intentionally retain the original generated segments because arbitrary paragraph edits cannot be mapped safely back to audio time without a new alignment pass. The interface calls this out rather than implying that rewritten text has automatically acquired accurate cue boundaries.

5. Add anonymous speaker labels without identifying people

Speaker diarization is optional on the website and disabled by default. It lazily loads an ONNX conversion of PyAnnote segmentation 3.0 and WeSpeaker VoxCeleb ResNet34-LM.

The segmentation model analyzes overlapping ten-second windows. Candidate speech turns are assembled into short voice samples, converted into normalized embeddings, and clustered by cosine similarity. Transcript segments receive the speaker whose turn overlaps them most, with a nearest-turn fallback.

This is heuristic diarization, not identity recognition. Results are deliberately anonymous—Speaker 1, Speaker 2, and so on—and overlapping voices, short utterances, noise, and similar voices can reduce quality. A diarization error never discards a successful transcript.

6. Manifest V3 needs three cooperating contexts

The Chrome extension separates responsibilities across a service worker, offscreen document, and dedicated transcription worker.

1. Service worker: receives the toolbar gesture, records the exact authorized tab, obtains a tab-capture stream ID, and coordinates state.
2. Offscreen document: redeems that stream ID, preserves tab playback through an AudioContext, records WebM/Opus audio, decodes the completed recording, and owns the persistent inference worker.
3. Dedicated worker: loads the pinned Whisper model and performs local transcription.

Chrome’s automatic side-panel opening initially consumed the action gesture without producing the activeTab grant needed by capture. Opening the panel explicitly inside chrome.action.onClicked fixed the authorization boundary. The extension also verifies that the active tab still matches the tab authorized by that click.

Tab transcription is record-then-transcribe, not live captions. Recording duration therefore affects both the retained compressed Blob and decoded PCM memory.

7. Prepare models before the recording stops

A first transcription felt slow because the extension originally waited until capture ended before creating the worker and initializing Whisper. We changed the lifecycle so users can prepare a model manually, while starting capture begins preparation in parallel.

One persistent offscreen worker owns a shared loading promise. Manual setup, automatic setup, and transcription therefore await the same initialization instead of constructing duplicate ONNX sessions. A compatibility-keyed marker records that a specific model revision, dtype, Transformers.js version, and ONNX Runtime version previously reached a ready state.

The marker means “previously prepared,” not “loaded now.” Chrome can evict model files, and every new worker still has to initialize sessions. The interface distinguishes saved model data from readiness in the current worker.

8. Why the extension uses FP32

Our website’s quantized WASM path did not prove that the extension’s packaged runtime would behave identically. In the extension, a q8 Whisper decoder failed during ONNX session creation with a missing scale in a TransposeDQWeightsForMatMulNBits optimization. Capture, decoding, and worker startup had already succeeded, which isolated the problem to model-session initialization.

Explicitly selecting FP32 made Transformers.js choose the unsuffixed encoder and merged-decoder ONNX graphs, avoiding that optimizer path for the exact pinned model/runtime combination we tested. The tradeoff is a larger initial model download and higher memory use. We chose reliability and documented the compatibility unit rather than claiming that q8 is universally broken.

9. Package executable code; download weights as data

Manifest V3 does not allow remotely hosted executable code. The extension therefore packages its JavaScript and ONNX Runtime WebAssembly locally. Whisper model weights are downloaded as data from pinned Hugging Face revisions and cached by Chrome when possible.

Our build audit rejects remote JavaScript, module, or WASM URLs, rejects an unsafe unguarded chrome.runtime.getURL() call inside the dedicated worker, and verifies that a local WASM runtime was emitted. This distinction—packaged runtime versus downloaded model data—is important both technically and in the Chrome Web Store disclosure.

Privacy boundaries

Local inference does not mean zero network activity. Loading the site and downloading models still sends ordinary request metadata to hosting and model-delivery providers. The extension also keeps operational state, current-session transcript segments, model-readiness markers, and limited diagnostics in Chrome storage. It does not send captured audio or transcripts to a LocalScribe transcription server because no such server exists.

The complete disclosure is available in the LocalScribe privacy policy.

What we would improve next

• Stream or incrementally decode long recordings to reduce peak memory.
• Use stronger boundary alignment than segment-end filtering.
• Evaluate smaller extension dtypes only against real packaged-runtime tests.
• Add a supported alignment pass when users need edited text reflected in subtitle cues.
• Measure device-specific speed, memory, and model-setup costs in a reproducible benchmark.

Try the workflow

Open LocalScribe and transcribe a local audio or video file

Whisper was created by OpenAI. LocalScribe is an independent product and is not affiliated with OpenAI. Browser model conversions referenced above are hosted by ONNX Community; model-specific terms and attribution are available on their linked model cards.