The backwards rule: how object-aware training teaches an inpainter to remove, not hallucinate.

I got stuck on one line of the CM-GAN paper for an embarrassingly long time. It describes how they place the holes they train an inpainting network on, and the rule reads backwards. When a hole would cover most of an object, they carve the object back out of the hole. My first reaction was: that is the opposite of what I want. I want to remove objects, not protect them. It took a while to see that the carve-out is exactly what teaches removal, and the confusion came from a wrong mental model of what the training target even is.

This post is that whole chain of reasoning, written out slowly, because the payoff is a clean way to think about when a generative inpainter fills with background versus when it invents a new object. If you work on anything downstream of inpainting (object removal, distractor cleanup, blemish retouching), this distinction is the difference between a tool and a hallucinator.

What CM-GAN is, and what this post is about

CM-GAN has two halves, and it is worth separating them cleanly, because this post is only about the second one.

The first half is the network architecture, and it is where the name comes from. The CM is Cascaded Modulation. An encoder built from Fast Fourier Convolution blocks gives every layer a global receptive field, so features on one side of a large hole can be informed by pixels on the far side. A dual-stream decoder then fills the hole in two chained steps: a global modulation lays down the coarse semantic structure of what belongs there, and a spatial modulation refines it per location so the detail lines up with the surrounding real pixels. Global sets the plan, spatial does the location-aware fix-up. That is the machinery that makes CM-GAN good at large holes in complex scenes.

The second half is the training methodology: how you generate the holes you train that network on. This is object-aware training, and it is almost entirely orthogonal to the architecture. Same network, different training holes. You could bolt object-aware training onto a completely different inpainter and get the same behavioural shift, and you could train CM-GAN's architecture on naive random masks and watch it hallucinate. The two halves solve different problems: the architecture decides how well the hole gets filled, the training scheme decides what the model tries to put there in the first place.

Everything below is about that second half. If you want the architecture in depth, the paper is the place. What follows is the training-side story, which is the part I found genuinely counterintuitive.

What inpainting training actually is

Start with the thing that trips everyone up, because everything else follows from it.

Inpainting is trained self-supervised. You take one real photo, punch a hole in it yourself, feed the holed version to the network, and ask it to reconstruct the original. The loss compares the network's output to the untouched original image. That is the only image you own.

Hold that fact, because it flips the carve-out logic into focus. If clean plates existed, that is, paired (with-object, without-object) images, you would skip every clever trick in this post and just train supervised: input the with-object image, target the without-object image. Those pairs do not exist at scale. Nobody has a photo of the exact scene with a specific cup digitally spirited away and the true pixels behind it restored. That information is occluded. It is gone.

So the CM-GAN authors do the only thing left. Instead of engineering the target (impossible), they engineer where the hole goes, so that the task "reconstruct the untouched original" happens to teach the behaviour they want.

The setup: panoptic masks, not random rectangles

Every training image is first run through a panoptic segmentation model (they use PanopticFCN), so the pixel footprint of every object instance is known: this person, that car, this cup. Then a candidate hole is sampled from a mix of shapes. The appendix gives the exact recipe:

So almost half the time the hole is deliberately shaped like a real object. That is the raw material. Now the question is what to do when a sampled hole lands on top of a segmented object, and this is where the interesting decision lives.

The overlapping ratio

For each object instance s_i, they measure how much of that object the hole m swallows:

Read the denominator carefully. It is the object's area, not the hole's. So r_i answers one specific question: what fraction of object i is covered by the hole? A value of r_i = 1 means the hole completely buries the object. A value of r_i = 0.1 means only a sliver of it is masked.

The 0.5 threshold, and why it goes the way it does

Here is the rule that reads backwards on first pass:

When the hole covers most of an object, they subtract the object's pixels back out of the mask, so the object stays visible in the input. When the hole covers only a small part, they leave it alone. Why protect the object precisely when it is most covered? Because of what the target teaches.

Suppose you mask an object and leave it masked. The target behind that hole is the object, because the target is the untouched original. So you would be training the network to reproduce a whole object from a hole shaped like that object. That is not a neutral outcome. It is a lesson, and the lesson is: hole appears, invent an object to fill it. Do that across a whole dataset and you have built a hallucinator. The paper says this plainly in the intro: models trained on randomly placed masks bleed colour across object boundaries and generate object-like blobs inside holes.

So when the hole would mostly cover an object, they pull the object out. Now the remaining hole sits on the background around a preserved object. And critically, the target behind that remaining hole is real background pixels that genuinely exist in the source photo. The network is only ever asked to reconstruct things that actually existed. It learns what plausible background near an object boundary looks like, and it learns to respect the boundary instead of smearing across it. That is the "distractor removal" skill: clear the clutter, get clean background back, leave the real subject untouched.

The other branch: object completion

When r_i ≤ 0.5, only part of the object is masked, so they leave the hole. Now the network sees a partially visible object and is trained to complete it from the visible remainder. That is the "object completion" skill, and it is genuinely what you want: regenerate the missing corner of a partly occluded object from the part you can still see.

So a single threshold cleanly splits training into two opposite behaviours:

Two housekeeping details in the same passage

Two smaller steps ride along in the same recipe and are easy to skip past. First, the object masks are randomly dilated and translated before use, to stop the network overfitting to exact segmentation outlines. Second, and more consequentially, the hole is dilated along the instance's segmentation boundary. That boundary dilation stops background pixels sitting right at the object edge from leaking into the inpainting region. It is the reason CM-GAN's removed-object edges come out clean instead of ringed with a halo of bled colour.

A worked example, and the mistake I nearly made

Take a person holding a cup. Two ways to mask it, two outcomes, and one of them is not what intuition says.

Mask only part of the cup. This one your intuition gets right. r_cup ≤ 0.5, so the hole stays, and the model completes the cup from the visible remainder. If the masked part happened to contain a chip or a smudge, the model regenerates it based on the clean surrounding cup, so the imperfection is replaced with plausible clean cup. That is the blemish-removal mechanism exactly, just with "cup" swapped for "skin".

Mask the whole cup. Here intuition says: the model conjures a fresh cup. That is backwards, and it is the failure mode object-aware training exists to kill. Trace it through. Covering the whole cup means r_cup > 0.5, which triggers the carve-out: the cup is pulled out of the mask. So the network is essentially never shown a training example that says "whole cup hidden, reconstruct a cup". (And if carving the cup out leaves the hole tiny, a minimum-area check throws the sample away and resamples.) At inference, masking the whole cup makes the model fill with background: the hand's grip, the table, whatever plausibly continues behind where the cup was. The cup is removed, not reborn.

One honest caveat: this is a bias, not a hard guarantee. If the surrounding context screams "object here", say a hand frozen in an obvious cup-gripping curl, the model may still produce something cup-shaped, because the visible context demands it. Object-aware training tilts the odds heavily toward background fill. It does not forbid object synthesis outright. And your original intuition is not wrong, it is just aimed at the wrong model: a naively-trained inpainter really would hallucinate a new cup into a fully-masked object, because it saw plenty of "whole object hidden, object is the answer" examples. That baseline failure is the entire thing this mask engineering fixes.

Why this matters downstream

If you are building anything on top of an inpainter, this training bias is the property you are inheriting, and you should know which regime you live in.

For blemish and distractor removal, you want to live permanently in the completion regime: tight masks over just the imperfection, never a mask so large it engulfs a whole facial feature. A blemish is a tiny distractor. Mask it, and an object-aware model fills it with surrounding skin instead of inventing a new feature. Stay in "complete from surrounding clean skin" and you never risk tipping into "hallucinate a new feature".

But notice the flip side of the same coin. This scheme is deliberately tuned so the model does not synthesise whole objects into large holes. So if your masked region is large, or you ever genuinely wanted the model to reconstruct a missing structure from nothing, this bias works against you. It is optimised to fill with plausible background or skin, not to regenerate structure. For pure removal that is a feature. Just know it is there, baked in, at the level of where the training holes were allowed to fall.

That is the whole trick, and it is a quietly elegant one. They could not build the target they wanted, so they built the placement of the damage instead, and let an unchanged reconstruction loss teach two opposite skills off a single threshold.

Thanks for reading ✦

Based on CM-GAN (Zheng et al., Image Inpainting with Cascaded Modulation GAN and Object-Aware Training, 2022), specifically the object-aware mask-sampling scheme and its appendix.