WebCodecs Decodes the Video. Reading the File Is Still Your Job.
WebCodecs gives JavaScript direct access to the encoders and decoders the browser already ships. Hand VideoDecoder an encoded chunk and you get frames back, on hardware, at speeds ffmpeg compiled to WebAssembly does not reach. For a video compressor that never uploads a file, it is the right foundation.
It is also a much narrower API than it first appears. WebCodecs is a codec interface. Everything around the codec, which turns out to be most of the work, is left to you.
What follows is the list of things that broke while building a compressor and an editor on it. Some are our bugs. Some are gaps in the platform that anyone building this will hit.
1. The Codecs Are There. The Container Is Not.
VideoDecoder handles H.264, H.265, VP8, VP9 and AV1. AudioDecoder handles AAC and Opus. What neither of them accepts is a file. They take encoded chunks, and getting chunks out of an MP4 or a WebM is a container parsing problem that WebCodecs does not touch.
The usual answer is mp4box.js, which is excellent and reads ISOBMFF. MP4, MOV and M4V are ISOBMFF. WebM and Matroska are not: they are EBML, a completely different structure.
This matters more than it sounds, because WebM is what the browser itself records. MediaRecorder in Chrome and Firefox writes WebM by default. So a screen recording made in the same browser, in the same tab, could not be opened by our tool. The error users saw was Failed to parse video file: ISOFile, which is mp4box being handed a file it was never going to understand.
We looked at the maintained MIT Matroska library first. It bundles to 241 KB, more than mp4box costs, because it ships Node stream and Buffer polyfills. It also does not run in a worker, which is where all of our decoding happens.
An EBML element is a variable-length ID followed by a variable-length size followed by that many bytes. Reading that is a page of code. Our reader is 12 KB, understands about twenty elements, and steps over everything else using the length the file declares rather than trying to infer it. It returns the same shape mp4box does, so the decode loop, the timestamp handling and the muxing did not change at all. It is dynamically imported only when a file turns out not to be ISOBMFF, so opening an MP4 never fetches it.
The lesson is not “write your own parser”. It is that WebCodecs leaves a whole layer to you, and the size of that layer is not obvious from the API surface. Budget for it.
While we were at it: our pages claimed AVI support. AVI is RIFF, a third format again, and nothing in the pipeline reads it. That claim is gone.
2. A Callback That Silently Discarded Every Audio Frame
This one is a WebCodecs-specific trap and it cost us the most time.
// Wrong. Every decoded frame is dropped.
const decoder = new AudioDecoder({ output: () => {}, error: onError });
decoder.output = handleFrame;The output callback is bound when the decoder is constructed. Assigning decoder.output afterwards does nothing at all: WebCodecs keeps calling the function it was given in the constructor, which in our case was an empty stub written as a placeholder. Every decoded audio frame went into it and vanished. No error, no warning.
// Right.
const decoder = new AudioDecoder({ output: handleFrame, error: onError });What made it survive so long is that MP4 hid it. AAC audio is copied through to the output rather than re-encoded, so the decode path was never exercised. The first input that actually took the re-encode branch was Opus inside a WebM, which only became possible once the container reader in the previous section existed. Three seconds of audio came back as a 1 KB track. It is 48 KB now.
The same line was in the video editor. If you have a WebCodecs pipeline with a copy-through fast path, check that your re-encode path has ever actually run.
3. Rotation Lost to Array.isArray
Every video shot in portrait on a phone came out on its side.
Phones do not rotate pixels while recording. They store the frame landscape and attach a display matrix saying “turn this 90 degrees for playback”. Our helper read that matrix and asked:
if (Array.isArray(matrix)) { /* ... */ }mp4box hands the display matrix over as an Int32Array. Array.isArray returns false for typed arrays, so the branch never ran, and every file ever opened reported no rotation.
The interesting part is what the obvious fix would have done. Swapping the encoder’s width and height, which is what the dimensions helper would have done with a rotation it never received, does not work. The decoder returns frames exactly as they were coded, and knows nothing about the display matrix. A portrait-shaped encoder fed landscape-coded frames letterboxes them into a tall frame with bars.
The rotation has to travel the way the source expressed it: as metadata on the output track. That costs nothing to reproduce and the player does the same thing the phone intended.
One consequence. Captions are burnt into the coded frame, so once the track carried a rotation, text that had been upright started rendering sideways. Captions are now laid out against the frame as it will be displayed, with the canvas turned back by the same angle first.
Verified against a phone clip stored 1920x1080 with a 90 degree matrix: source and output both display 1080x1920 upright, and a caption placed at the bottom lands at the bottom.
4. B-Frames, and Why a Shared Timestamp Offset Cannot Work
Some files were refused outright by the muxer.
A file encoded with B-frames carries a reorder delay in its composition times. The first video sample sits at 66 ms rather than 0, because the decoder needs to see a later frame before it can present this one. The audio track, which has no such reordering, still starts at 0. Our muxer requires every track’s first chunk to be at zero, so it rejected the file.
The fix is to shift each track by its own earliest composition time. A single shared offset across both tracks cannot do it: one number cannot put two tracks that begin at different times both at zero.
This is what the edit list those files carry would have done at playback anyway. Tracks that already start at zero subtract zero, so files that worked before are byte-for-byte unchanged.
The video editor had always accepted these files, which was a useful clue. It retimes every frame from zero as part of trimming, so it was accidentally immune.
5. “Fit Under 25 MB” Is the Actual Requirement
Almost nobody opens a compressor wanting “medium quality”. They want the file under a limit: an email attachment, a Discord upload, a form that refuses anything over 10 MB.
A quality preset cannot answer that, because the size is only known once the encode has finished. So getting under a limit meant picking a preset, waiting, missing, and guessing again.
Making the target size the input is mostly arithmetic. Audio is copied through, so its bytes come out of the budget first. Leave a slice for the container’s own boxes. What remains, spread over the running time, is the video bitrate.
Two things make it more than arithmetic.
A low bitrate against a full-size frame buys blocking, not detail. Below a certain bits-per-pixel the encoder cannot describe the frame at all, so the frame is scaled down until each pixel gets a workable share. A 20 second 720p clip aimed at 3 MB comes back at 960x540, which looks far better than a 720p smear.
Variable bitrate aims at an average and can overshoot. A pass that lands over the line is followed by one corrective pass scaled by how far it missed. Measured on a 14.68 MB source: a 5 MB target lands at 4.86, 2 MB at 1.95, 14 MB at 13.60. Each is inside the limit and each takes about 1.7 seconds.
Two cases are refusals rather than surprises. A target smaller than the audio alone says so and names the audio’s size. A target larger than the source says the file is already smaller, which we added after watching a 14.68 MB file asked to fit 50 MB come back at 34 MB: working backwards from a budget roomier than the source had the encoder dutifully spend it.
6. A Flat Bitrate Does Not Mean the Same Thing at 360p and 1080p
Our quality presets used to carry a hidden scale factor. “Maximum” halved the frame, “Balanced” took three quarters, “High” left it alone. Two consequences: you could not ask for high quality at 720p, and nobody choosing Balanced was told their 1080p was coming back at 810p.
Resolution is a separate control now, and it constrains the short side, so 720p means 720 across for a clip shot in portrait rather than 720 tall. It never enlarges: a 480x270 source asked for 1080p stays 480x270.
Quality had to change with it. A preset is now bits per pixel per frame, and the bitrate follows the output size. Measured on a 14.68 MB 720p clip, the two axes move independently:
| Setting | Result |
|---|---|
| 720p, Maximum | 2.30 MB |
| 720p, High | 8.22 MB |
| 360p, High | 2.29 MB |
The source’s own bitrate caps the result, so “High” on an already-compressed file cannot inflate it. A 480x270 clip carrying 300 kbps comes back at 0.36 MB rather than the 8 MB the preset would otherwise have spent.
7. Our Support Gate Was Wrong in Both Directions
We checked browser versions. Both numbers were wrong, in opposite directions, which is a good argument for not checking versions at all.
Firefox was being turned away when it works. Firefox has had the full WebCodecs API on desktop since 130. Our gate predated that and refused it.
Safari was being promised something that would fail. Safari has had VideoEncoder and VideoDecoder since 16.4, and it is tempting to gate on that. But our pipeline re-encodes audio, and AudioEncoder and AudioDecoder did not arrive until Safari 26. Users on 16.4 were told it would work and then watched it break.
The gate now asks for the interfaces the worker actually calls:
const supported =
typeof VideoEncoder !== 'undefined' &&
typeof AudioEncoder !== 'undefined';That is true everywhere, needs no maintenance, and gets both cases right for free. Version tables go stale; feature detection does not.
8. The Memory Ceiling Is the Tab, Not the Machine
Large files failed with RangeError: Invalid array length, which tells a user nothing.
The instinct is to compare the file against the device’s RAM. That is the wrong number. A browser tab has its own heap limit well below the machine’s memory. The laptop this was tested on has 32 GB; the tab is capped at about 4 GB.
Chrome exposes this as performance.memory.jsHeapSizeLimit. Firefox and Safari do not, so we fall back to a fixed 700 MB assumption there. A file past roughly a third of the available ceiling now gets told, before it starts, that it will be held in memory and may not finish on a device with less to spare. A 229 MB clip passes without comment; a 1.5 GB one does not.
And when it does run out, the message says the video is too large to hold in memory and that trimming it or choosing a smaller resolution will get the same device through. That is both true and actionable, which the RangeError was neither.
What This Adds Up To
WebCodecs is genuinely good at the thing it claims: it hands you the browser’s hardware codecs with a small, predictable API. Nothing here is an argument against using it.
But if you are estimating a browser video project, the codec is not where the work is. The work is in the container parsing WebCodecs does not do, the muxer’s constraints it does not tell you about, the metadata that has to survive a decode and re-encode, and a memory ceiling that belongs to the tab rather than the computer.
Three of the eight problems above were only visible on real files: a phone recording with a rotation matrix, a file with B-frames, a WebM from the browser’s own recorder. Synthetic test clips encoded with default settings will pass while all three are broken. If you build one of these, test it on video that came off a phone.
The tools this came out of are the video compressor, the video editor and the GIF maker. They run entirely in the browser tab, which is the point of using WebCodecs in the first place: the file has nowhere to be uploaded to.