The S that was never there

Can you hear it?

Every S in these takes got replaced. Not ducked, not repaired. Replaced. Four speakers, and you don't get told which take is which. Have a listen before you read on.

Why replace at all

A de-esser ducks. It spots an S running too hot and pulls it down until it fits. The level drops and that's it. Whatever was wrong with that S is still in there, just quieter.

Levelling does something else. Every sibilant gets moved into a corridor with a floor and a ceiling. Loud ones come down, quiet ones come up. You end up with sibilants sitting at roughly one height across the whole file, instead of just a lid on the loud ones.

The pulling up is the awkward half. If an S is quiet because it got damaged, turning it up turns up the damage with it. Noise, distortion, a smeared top end, all of it comes along. At some point gain stops helping. That's where you start building part of the S instead of amplifying it.

What replace means

Replacing isn't a swap or a cut. It's a blend with a dial on it, the replacement rate, r, from 0 to 1. At r = 0 you get the original event untouched. At r = 1 it's all new material. Anything between is a mix of the two.

You can't just crossfade linearly between them. The original and the replacement aren't correlated, so they add up in power rather than amplitude, and a linear fade loses energy right in the middle. The level dips exactly where you'd notice it, around half replacement.

So the blend runs at constant power instead. D is the level the event should land on inside the corridor.

Both ends fall out of that on their own: original only at r = 0, replacement only at r = 1, and no dip in between.

Useful side effect. Since r is just a number, you can render the same file at r = 0 and at r = 1 and put the two next to each other. That's the pair sitting at the top of this page.

Level accuracy inside the corridor came out at -0.12 dB average deviation, worst case -0.63 dB.

Borrowed texture

So what is the new material. Mostly not noise. The first move isn't to synthesize anything, it's to go looking for something to borrow.

The stage searches the rest of the recording for healthy sibilants from the same speaker, takes one, stretches it to the length it needs through STFT magnitude interpolation and splices it in with its own phase. What lands in the gap is real texture from that voice, that room, that mic.

The donor gets picked by health, not by frequency. Going for the spectrally closest event feels right and isn't: events sitting near a damaged one have usually taken the same hit. You want one that's intact, and then you shape it.

If there's nothing in the file worth borrowing, the fallback builds shaped noise instead, with a spectral envelope from an empirical voice class picked by fundamental frequency, and frame to frame texture out of a library built up front. Calibration is a separate problem and gets the next chapter.

There's an order to it because the three aren't equally good. Material from the same recording brings the speaker and the room along for free. Synthesis has to guess at both. So it borrows while there's anything left to borrow. The tier logic came out of the restoration work, there's more on it in the piece on restoration.

Why absolute thresholds fail

A corridor needs a floor and a ceiling, and the obvious move is to pick two levels. That falls over on the second speaker you try.

Sibilant level is a property of the voice. Across the four speakers in the test set the medians sat almost 20 dB apart. A corridor tuned for one of them isn't slightly off for the next, it's in the wrong place entirely, either clamping everything or touching nothing.

So the corridor comes out of the material instead, from the median sibilant level of the file. Which works, and brings its own problem along.

If the file has already been de-essed, that median is damaged too. Build the corridor from it and you've written the damage into everything downstream, very faithfully. Calibrating against your own broken input doesn't get you anywhere.

The way out is that de-essing doesn't touch everything. There's a band below the sibilant range that survives whatever was done to the file, and from that you can estimate where this speaker's sibilants should have been sitting. The measured median gets blended toward that estimate. The file still picks its own corridor, it just doesn't get to diagnose itself.

How it was tested

One listener. Me. Four speakers, spread over a few weeks. Assignment randomized per file, key in a hidden folder, opened after the answers were written down.

That's not a panel and it isn't a controlled study, and I'm not going to dress it up as one. It's also the one listener who knows exactly what was done to the file and what artifact to go hunting for. I'd take that over ten people who were never told what to listen for.

When borrowed texture replaced the old synthesis I didn't test against the original again. Old against new, three speakers, six files each. Passing meant answering "no difference", which is an odd thing to aim at.

A listening test on its own only proves you didn't hear anything, not that anything happened. So the objective check ran alongside it: replacement was measurably active across every event span, median delta depth +3 dB against the sibilant component. The files aren't indistinguishable because nothing was done to them.

Then 1562 real files went through, no crashes, 82.9 percent clean. The rest got sorted into buckets by failure shape and worked through by ear, which is slower than a metric and catches what the metric wasn't written to look for.

Where it breaks

Not every detected event gets processed. Health checks sit in front, and anything that fails gets passed through. Passed through literally: for vetoed events the measured maximum deviation between input and output is zero. Not small. Zero.

A voiced sibilant isn't the same problem as an unvoiced one. A /z/ has pitch underneath it, and treating it like an /s/ takes that with it, so voiced events get spotted through fundamental frequency and a reduced replacement rate.

The most honest result came from a file with nothing but voiced material in it. Detector produced zero events. The stage got bypassed, signal came out unchanged. Correct behaviour, and a fairly clear note about where the detector would need recalibrating if that material ever mattered.

One known problem is still open. Voice class detection through fundamental frequency misfires on heavily damaged and on sung material. The damage is contained: it hits the envelope shape in the fallback and a calibration offset, and it doesn't reach detection or the health checks. No audible follow-on problems across the large run or the edge cases. Understood, and not fixed.

The edge cases that do come out clean: whispering, high singing, shouting, silence and noise floor, telephone bandwidth.

Wrap-up

The shift that matters here is a small one. An S isn't a spot in the file you have to turn down. It's material, and material can be replaced, as long as you know when that's safe and what to build it out of.

Levelling is what forces the question, because the moment you start lifting quiet events you start lifting whatever's wrong with them. Borrowing from the same recording is what got it through a blind test.

Anyone generating or restoring speech at scale runs into the same thing, without an engineer sitting next to it deciding case by case.