Putting the S back

A de-esser takes sibilance out. This is about the other direction: what to do when the S is already gone. If you work on speech you have run into this more often than you think. A restoration model rebuilds the top end and the esses come back as a smear. A codec throws away the band they lived in. A de-esser upstream was set too hard, on every file in the batch. The words stay intelligible and the voice sounds like it is talking through a scarf.

This is how I approached restoration during vocal DSP research for NoiseWorks. Same rules as the detection write-up: concept level, the tuning values stay with the client. The short version of that piece, if you have not read it, is that sibilance detection works by comparing two signals instead of watching one. Everything here sits on top of that.

Where the S goes missing

Four things kill an S, and they do not kill it the same way. That distinction turned out to matter more than anything else in the project.

The first two are attenuation problems. The last two are structural. Treat them as one problem and you build a tool that fixes one case and quietly wrecks three.

Restoration is a detection problem too

The detector still comes first, but the job inverted. A de-esser looks for moments that are too loud, and loud announces itself. A restorer looks for moments that should be there and are not. Absence does not trigger anything.

Which leads to the trap that cost me the most time in this project. The worse the damage, the less the detector sees. A healthy S is a burst of energy well above the body of the voice, and that contrast is exactly what the ratio gate is built to catch. Take enough of it away and the burst stops being a burst. The events you most need to fix are the ones that have stopped announcing themselves, and a detector tuned on healthy speech will look at a badly damaged file, find nothing, and hand it straight back to you.

Lowering the threshold does not fix it. That turns every breath and every bit of room tone into a candidate. What worked was two changes. Widen the search downward, because the remains of a damaged S sit lower than a healthy one and how much lower depends on the voice. And stop letting the file grade itself.

Offline changes what is possible

All of that assumes you get to look around. A real-time plug-in has a few milliseconds of lookahead and spends that budget catching the onset. Working offline, the entire file exists before the first sample is written, and that is not a convenience. It changes the class of things that can be done at all.

You can profile the speaker before touching them: what a healthy S looks like for this voice, in this room, at this distance from the microphone. You can run several passes, so detection, classification and the decision about what to do are separate steps with every event on the table at once. And you can borrow, which turns out to be the single most useful consequence. The best material for repairing this speaker is almost always this speaker, twenty seconds later in the same recording.

For post production, dialogue and batch cleanup, which is where damaged speech actually lives, offline is the right place to stand.

Three tiers, not one repair

Because the damage comes in different kinds, the repair has to as well. Three of them, chosen per event rather than per file, because one recording usually contains several kinds at once.

Transplant. Find a healthy S elsewhere in the same recording, match it against the damaged one, and transfer what is missing. Same voice, same room, same microphone, so it sits in the file without a seam. This gives the best result whenever it is available, and it is available more often than you would guess, because damage is rarely perfectly even.

Synthesize. When there is no healthy donor anywhere, because everything got the same treatment, the sound has to be built. Shaped noise, matched to the spectral envelope this voice should have had at that moment. This is the tier everyone assumes will sound artificial.

Lift. The damage is mild, the structure survived, and raising what is already there is enough. Cheapest and safest, and the one with a hard ceiling.

Why gain alone hits a wall

The obvious design is to use the third tier for everything. Find the damaged events, turn them up, done. No synthesis, no donor matching, nothing that could sound invented. I tried to make that work for longer than I should have.

It runs into physics. Sibilance is aperiodic noise. Once something upstream has pushed it far enough down, what remains is not a quiet copy of the original, it is the noise floor of the recording. Amplify it and you amplify the noise floor. The S does not come back. The hiss between the words does.

On files with seven to thirteen decibels of damage, gain alone closed about one. Not because the gain stage was badly tuned, but because there was nothing left to scale. That measurement is the reason the synthesis tier exists at all. Once you accept it, the question stops being whether to synthesize and becomes how much, and when to stop.

Knowing when not to repair

A restoration tool that always restores is more dangerous than one that sometimes declines.

The case that taught me this: material that has already been through a chain. A finished podcast, compressed, limited, equalised, maybe already restored once by something else. Run a repair over that and the tool sees a spectral shape it does not recognise, concludes the esses are damaged, and adds energy that was never missing. The output is worse than the input, and confidently so.

So a veto runs before any tier is picked. If the material looks like it sits outside what the tool was built for, nothing happens at all: not a gentler repair, an actual bypass. That check is the least interesting part of the system to build and the part I would defend hardest. A tool that quietly damages one file in twenty does not get used, and it does not deserve to be.

Testing: reconstruction first, then ears

Two tests carried most of the weight. The first is structural. Before anything can be repaired the signal has to be split into the sibilant part and everything else, and that split has to be exact. Add the two halves back together and you have to get the original back, not something close to it. If the separation leaks, every repair downstream is built on a wrong picture of the file, and no amount of tuning further down will save it.

The second is the delta test from the detection write-up, run backwards. There you subtract to hear what the algorithm took out. Here you subtract to hear what it put in. If the repair is honest, that difference is a file of esses and nothing else. If there are vowels or room tone in it, the tool is inventing material where it has no business inventing anything.

And the part I want to be straight about: I tried to score this numerically, error against the original, and it is an incomplete measure. A repair can score well and sound synthetic. It can score badly and be the better take, because the original was not the target either, it was damaged too. The number narrows the field. The ear picks the winner, and it took weeks of going back and forth before I trusted any of it.

Where this ends up

Restoration turned out to be three problems sharing one name. Find what is missing, when missing means invisible. Pick a repair that matches the kind of damage rather than the amount. And know when to refuse. None of the DSP underneath is exotic. The judgement about which of the three situations you are in is where the work actually went.

If you build speech tools and your pipeline has a strong neural half and no classical one, the fricatives are the seam I would look at first. That is where those models give up, and they give up quietly.