You upload a crisp, well lit selfie, type a simple request, and the AI hands back a person who is almost you. Same haircut, same general vibe, but the nose is wrong, the eyes sit too far apart, and the jaw belongs to a cousin nobody has met.

Spend a week pushing portraits through AI editors and generators and that near miss becomes the most predictable part of the whole process. The face drifts, every time, in roughly the same ways. The useful thing to know is that the drift is not random, and it is not a flaw you are stuck with. AI tools change faces for a short list of technical reasons, and each reason has a fix that holds up in real work. What follows is why it happens, then seven fixes that keep a face recognizably yours, ordered from the single change that solves most cases to the heavier options for when the likeness has to be exact.

The short version. Most face drift comes from two things working together: the AI is rebuilding your photo from scratch rather than editing it, and nothing in the process is told to protect your identity. Turn the rebuild strength down, give the model a real identity reference, and the face stops wandering. Everything below is detail on those two levers and the smaller ones around them.

Why AI changes your face

Six things cause almost all of it. Usually more than one is working at the same time.

1.  It rebuilds, it does not retouch

An AI editor does not paint over your photo the way a person would. It compresses the image into a dense block of numbers, a kind of mathematical summary, then paints a brand new picture to match that summary. Fine identity details, the exact spacing of your eyes or the curve of your lip, are the first things lost in the summary. The result looks like your description, not your face.

2.  Faces slide toward an average

These models learned from millions of faces, and that training pulls every output gently toward a statistical middle. Skin gets smoother, features get more symmetrical, proportions get more conventional. That is why AI so often makes people look younger and slightly generic. It is not flattering you, it is reaching for the most probable face, and the most probable face is an average one.

3.  The rebuild strength is too high

In image to image tools there is a setting called denoising strength, sometimes labeled image weight or transformation strength. It controls how much freedom the model has to repaint. Set high, the model treats your photo as a loose suggestion and invents the rest. A large share of botched faces trace back to this one slider sitting far too high for a portrait.

4.  Nothing holds your identity in place

A plain generation model has no concept of who you are. It knows what a face looks like in general, not what your face looks like specifically. Without a dedicated identity signal added on top, no part of the pipeline has the job of keeping the output matching you. The model is free to drift because nothing told it not to.

5.  Restorers invent the detail they fix

Tools that sharpen or restore faces, and the face stage inside many upscalers, do not recover your real features. They generate plausible new ones. Eyelashes, pores, and teeth that were soft get replaced with crisp inventions that may belong to someone else. Run a face through an aggressive restorer and the likeness often takes the hit at the exact moment the image looks sharpest.

6.  The source photo never stood a chance

If your face is small in the frame, poorly lit, turned to the side, or buried under a heavy filter, there is very little identity information to hold on to. Low resolution is the quiet killer. When a face occupies a few hundred pixels, there are not enough details to preserve, so the model fills the gaps with guesses.

Match the symptom to the cause

Before changing anything, read what the output is actually doing wrong. The fix is usually obvious once the cause is named.

What you are seeingMost likely causeFastest fix
Sort of you, but clearly offRebuild strength too highLower denoising to roughly 0.2 to 0.45
Younger, smoother, genericPull toward the average faceAdd an identity reference, reduce guidance
Great until the final sharpenRestorer inventing detailLower restorer weight or skip it
Only the face is wrong, scene is fineWhole image regeneratedMask the face and inpaint only
Wrong in every attemptNo identity anchor, weak sourceUse a face adapter and a better photo
Worse when the prompt is detailedPrompt describing the faceRemove facial descriptors from the prompt

7 fixes that actually work

1   Turn the rebuild strength down first

Before touching anything else, find the denoising or transformation strength and bring it down. For keeping a face intact, the useful band sits roughly between 0.2 and 0.45. Low values stay close to the original and change little. Push past the middle and identity starts to slip. Aim for the lowest value that still makes the edit you wanted. This single change resolves more face complaints than anything else, which is why it comes first.

Best for: any editing workflow                  Effort: seconds

2   Give the model an identity reference

Identity adapters exist specifically to solve this. Tools such as IP-Adapter FaceID and InstantID take a photo of your face and feed it into generation as a signal the model has to respect, separate from the prompt. Dedicated face swap nodes like ReActor go further and transplant your features onto the finished result. Instead of hoping the model remembers you, you hand it your face and require it to use it.

Best for: new scenes, same you                Effort: moderate, one time setup

3   Train a small personal model

For true, repeatable likeness across many images, nothing beats a lightweight personal model, usually a LoRA. Feed it fifteen to thirty varied photos of the same face, with different angles, lighting, and expressions, and it learns your specific features rather than a generic stand in. The payoff is consistency: once trained, every generation starts from a model that already knows you. It costs the most time up front and returns the most reliable result.

Best for: a recurring character                  Effort: high, one time

4   Fix the source before blaming the model

A clean input does half the work. Use the highest resolution photo available, with even front lighting, the face large in the frame, eyes open, a neutral or gentle expression, and no heavy beauty filter already baked in. If the only photo is small or soft, upscale it carefully first so the model has real detail to protect. Weak input produces a stranger no matter how good the settings are.

Best for: every workflow              Effort: low

5   Edit only the face by masking

When the scene is right but the face is wrong, there is no reason to regenerate the whole image. Use inpainting with a mask so the model only touches the area you select, or invert it to change the background and clothing while the face stays frozen. Masked editing is the difference between adjusting one part of a picture and rolling the dice on the entire thing.

Best for: outfit or background swaps       Effort: low to moderate

6   Go light on restoration and upscaling

If a face restorer is part of the pipeline, treat it with suspicion. Lower its strength, and where a fidelity control exists, such as the one in CodeFormer, set it toward preserving the original rather than maximizing sharpness. Better still, swap a blunt restorer for a face aware detailer run at low rebuild strength, which cleans up the face without reinventing it. Sharp and wrong is worse than soft and right.

Best for: any sharpen or enlarge step      Effort: low

7   Stop describing the face in the prompt

Every facial adjective competes with the real face. Words like beautiful, youthful, sharp jawline, or big eyes push the output toward those words and away from you. Strip the prompt down to pose, scene, style, and clothing, and let the reference carry identity. Lowering the guidance or CFG value helps too, since it loosens the model’s grip on the prompt and lets your face come through.

Best for: reference based generation      Effort: seconds

How the methods compare

Different jobs call for different fixes. This is where each one earns its place.

MethodLikeness strengthEffortBest when
Lower rebuild strengthModerateSecondsYou just want edits to stop drifting
Identity adapterStrongModerateNew scenes, one subject, light setup
Personal model (LoRA)StrongestHighMany images or a recurring character
Better source photoSupportingLowAlways, as a baseline
Masked inpaintingStrong for editsLow to moderateScene is fine, face needs a touch up
Light restorationSupportingLowAny sharpening or upscaling step

The order that works in practice

1.  Start with a strong source photo.

2.  Keep the rebuild strength low.

3.  Add an identity reference or a face swap.

4.  Mask the face for any targeted edit.

5.  Keep facial descriptors out of the prompt.

6.  Restore and upscale gently at the very end.

7.  If this comes up often, train a personal model once.

What to expect, honestly

Holds up well

•  Keeping an edited photo close to the original

•  One subject across a themed set of images

•  Fixing a face without touching the scene

•  Everyday likeness that reads as clearly you

Still rough

•  Results that are identical pixel for pixel

•  Tiny faces in busy group shots

•  Extreme style changes that fight realism

•  Fine detail pulled from very low resolution

Final verdict

After enough hours inside these tools, the pattern is hard to miss. Almost every wandering face comes down to the same two causes: the model is rebuilding the image with too much freedom, and nothing in the process is protecting the identity. Fix those two and the rest is polish. Pull the rebuild strength down, hand the model a real reference through an adapter or a face swap, and feed it a clean photo, and faces that used to arrive as strangers start arriving as you.

The heavier options earn their place only when the stakes rise. A trained personal model is worth the setup for a recurring character or professional work, and masked inpainting is the quiet hero for edits where the scene is already right. What does not work is hoping a detailed prompt will carry a likeness on its own, or trusting an aggressive sharpener at the finish line. Treat identity as something you actively hold in place rather than something the model will remember, and keeping your face the same stops being a fight.