Hmm… I think this is one of those areas where there isn’t really a single definitive “this is the solution” answer:
Yes — there are methods much closer to what you want than ordinary inpainting, but I would separate the problem into a few controls rather than look for one stronger reference adapter.
The closest research terms are things like reference-based object insertion, identity-preserving object composition, and localized subject-driven editing. The difficult combination in your case is:
- replace only a specified object/region,
- preserve the actual product identity, not merely its category/style,
- adapt it to a different viewpoint,
- and still harmonize perspective, lighting, shadows, etc. with the original room.
There are now dedicated methods for this, for example Insert Anything, OSInsert, IMPRINT, ObjectMate, and CareCom.
But I would not expect any of them to guarantee:
one reference photo + a substantially different camera angle + exact unseen product geometry.
That last part is important. If the reference only shows the front/three-quarter view of a bed, the exact geometry of an unseen side or back simply is not present in the image. A model can generate a plausible novel view, but “plausible” is not necessarily “this exact manufactured product”.
Also, the approaches you already tried are not necessarily the wrong direction. They just control different pieces of the problem:
- inpainting: mainly controls where regeneration happens;
- ControlNet: is very useful if you already have the correct depth/edges/geometry to condition on;
- IP-Adapter: supplies image-derived conditioning, but that conditioning is not a hard 3D product-geometry constraint (Diffusers IP-Adapter guide);
- LoRA/DreamBooth: can help if this is a product you will reuse many times, especially when you have multiple reference views, but it does not by itself solve placement + viewpoint conversion + inpainting.
So before replacing the whole stack, I would try a very small set of controls.
What I would try first
For the same source/reference and the same seed:
1. Use a clean crop/cutout of the reference object.
2. If possible, try a reference view close to the required target view.
3. Compare:
A. your current/tight mask
B. a modestly expanded mask
C. a larger bbox-like "placement region"
4. If you have more photographs of the same product,
add a front / side / 3/4 reference rather than trying
to recover everything from one image.
5. Judge the result by parts:
- headboard shape
- legs
- proportions
- frame structure
- upholstery / pattern
- logos/text if relevant
I would do those A/Bs before spending much time tuning CFG, adapter strength, prompts, or trying ten more models.
The mask test in particular can be surprisingly important. If your current mask tightly follows the silhouette of the old bed, that silhouette can become a geometric constraint when the new bed needs a different shape or viewpoint.
The official Insert Anything inference code, for example, does not simply take an arbitrary tight object mask and leave everything else unchanged: it preprocesses/crops the reference, expands the target region, and works on a larger local crop. Its README also explicitly says that the mask needs to fully cover the area to be edited.
OSInsert goes even further: its conservative path uses Insert Anything directly, while its aggressive path first adapts pose/viewpoint with ObjectStitch and then restores reference details with Insert Anything. In aggressive mode it even changes the mask strategy during denoising, from a SAM-derived object mask to a bbox-like mask.
That is quite close to the underlying conflict here:
preserve this exact object
versus
change this object’s geometry enough to fit this camera and scene.
A practical default route
If this is a one-off / zero-shot replacement, I would probably try:
- Insert Anything as the closest directly runnable baseline for
background + target region + reference object.
- Same/near-view reference first.
- Tight vs expanded/bbox target region.
- Multiple reference views if available.
- Only then move to a more aggressive geometry-adaptation approach such as OSInsert.
If this is instead a catalog/production workflow where the same SKU will be inserted repeatedly, the economics change. Multi-view product photos plus product-specific tuning become much more attractive. The CareCom authors explicitly note that their pretrained model can be weak on rich-detail objects and suggest object-specific fine-tuning when several images of the object are available.
And if by “exact” you mean CAD-level geometry, including unseen sides/dimensions, I would switch branches entirely: use a known 3D/CAD asset, or reconstruct a 3D proxy, control the pose in 3D, then use the generative model mainly for harmonization/relighting. DIRECT is a recent example of this design: it separates appearance, geometry and scene context, using a reconstructed 3D proxy for explicit pose control.
A reconstructed proxy is still only an estimate if it came from one photograph, though. If you already have the manufacturer’s CAD/3D model, that is a much stronger source of geometric truth.
Why reference identity gets difficult when the viewpoint changes
There are really two different things hidden inside “preserve the reference”:
- preserve appearance/identity, and
- preserve or correctly transform geometry.
Those are not the same problem.
The current Diffusers documentation describes IP-Adapter as image-based guidance: an image encoder extracts features which are injected through additional cross-attention layers. It can be masked to particular output regions and combined with multiple image references, but that still means “apply these image features here”, not “this is a complete geometric specification of the object”.
That distinction becomes much more visible with large viewpoint changes.
IMPRINT explicitly separates identity-preserving representation learning from background alignment/harmonization. Its motivation is essentially that existing methods often trade identity fidelity against the geometric/color changes needed to fit the target scene.
ObjectMate frames the goal similarly:
- preserve object identity,
- while generating an appropriate pose,
- lighting,
- and composition.
One interesting part of ObjectMate is that it builds training supervision from multiple real views of the same recurring object, rather than assuming one view contains enough information about the complete object.
That is why I would regard additional product views as much more valuable than another round of prompt engineering if the failure only starts when the camera changes significantly.
A useful way to think about the requirement is:
"same product" can mean three different things:
1. recognizable as the same SKU
2. part-level exact
(legs, headboard, proportions, logo, stitching, etc.)
3. geometrically exact
(including unseen surfaces / CAD geometry)
Level 1 is increasingly practical with 2D generative editors.
Level 2 is still a hard identity-preservation problem, especially under pose/view changes.
Level 3 requires information that a single 2D reference may simply not contain.
Why I would A/B the mask before changing models
There are several different things people call a “mask” in these workflows:
- an edit region: pixels that may change;
- a placement region: approximately where the new object should go;
- a shape constraint: what silhouette the result should follow;
- an image-conditioning routing mask: where a reference embedding should influence generation.
Those have very different semantics.
If your mask is the exact silhouette of the old bed, but the new bed has:
- a taller headboard,
- different legs,
- different proportions,
- or a different projected silhouette because the camera angle changes,
then the requirements may conflict.
That does not mean “always use a huge mask”. Too much freedom can damage placement/background consistency.
I would simply run:
same source
same reference
same seed
A = current mask
B = slightly dilated mask
C = bbox-like placement region
Then compare the actual structural details.
This is also roughly consistent with how several dedicated compositing systems are designed.
OSInsert, for example, has:
- a conservative mode using Insert Anything within a bbox;
- an aggressive mode using ObjectStitch → SAM → Insert Anything;
- a two-phase aggressive mask schedule, using a SAM mask first and a bbox mask later.
AnyDoor also treats shape/location control as something that can be made stronger or weaker rather than assuming one mask meaning for every edit.
So I would treat the mask as a first-class experimental variable, not just a preprocessing detail.
A small rigid-object sanity check
I also tried a small controlled sanity check because I was curious about this exact mask/viewpoint interaction.
I used a rigid sofa with multiple turntable views and kept the insertion model/seed fixed.
The conditions were approximately:
0° reference + tight 0° sofa silhouette -> insertion worked
15° reference + tight 0° sofa silhouette -> insertion worked
60° reference + tight 0° sofa silhouette -> insertion nearly failed
60° reference + loose placement region -> insertion worked
The last two are the interesting pair: the reference was identical; only the target-region geometry changed.
That obviously does not diagnose your bed case, and it does not establish a magical angle threshold. The test also used a generated/public fixture rather than your source images.
But it did convince me that I would test the target mask before concluding that the reference-conditioning model itself is the whole problem.
It also showed the other side of the issue: even when insertion succeeded, the rigid object was still regenerated to fit the scene. It was not a pixel-perfect re-render of a known 3D product.
So I would distinguish:
identity-preserving generative insertion
from:
exact rigid-object reprojection.
They overlap, but they are not identical tasks.
Which approach I would try under different constraints
A compact decision tree might be more useful than a model ranking.
1. Do you want to keep the old object’s shape?
Yes — mostly transfer appearance/material/design cues
A shape-preserving reference-editing approach is more appropriate. For example, MimicBrush explicitly exposes a “keep original shape” style of control.
No — the new bed should have its own real geometry
Then I would avoid using the old bed silhouette as an unnecessarily hard constraint.
2. Is the desired target view close to the reference?
Yes
Start with the conservative route:
- clean reference;
- same/near view;
- Insert Anything or another dedicated insertion model;
- minimum geometry modification.
No
Then the problem includes actual viewpoint synthesis.
Try, in roughly this order:
- extra views of the same product;
- a method designed to adjust pose/viewpoint;
- or an explicit 3D branch if geometry is important.
OSInsert is particularly interesting here because its aggressive path explicitly separates “make the foreground pose/view fit the background” from “restore the reference details”.
3. Do you have multiple photographs of the same product?
Yes
Use them.
ObjectMate is built around multiple recurring views of identical objects.
CareCom extends ObjectStitch to accept multiple reference images, and its maintainers explicitly say that more reference images generally help.
For repeated use of one rich-detail product, CareCom also documents a fine-tuning path using a few images of the object.
4. Is exact hidden geometry actually required?
No
Stay in the 2D generative branch. The output may be sufficient for visualization even if some unobserved geometry is inferred.
Yes
I would stop treating the image editor as the source of geometric truth.
If a CAD/3D product asset exists:
product CAD / 3D asset
↓
choose target camera / pose
↓
render geometry/depth/masks
↓
generative harmonization / relighting / cleanup
If no 3D asset exists, an image-to-3D proxy can still help control pose, but it should not be mistaken for ground-truth geometry.
DIRECT is a useful recent example. It reconstructs a 3D proxy, allows explicit 6-DoF pose control, and then injects decomposed geometry/appearance/context conditions into the image-generation stage.
There is also a broader family of “2D → 3D → 2D” editing work based on the same idea: rigidity and pose are much easier to define explicitly in 3D than to hope they emerge from a soft reference embedding.
5. Is this one image, or a repeated product workflow?
One/few images
Use the zero-shot/tuning-free branch first.
Same SKU across many scenes
At that point, multi-view capture and product-specific tuning can be worthwhile.
This is where LoRA/DreamBooth-style customization becomes much more attractive economically — but it still does not invent reliable hidden-side geometry that was never observed.
Current runnable/open-weight options I would keep on the shortlist
Insert Anything
Insert Anything is probably the closest directly runnable baseline I found for your input contract.
It supports object insertion/replacement from a reference and a target area. The project provides:
- standard inference;
- ComfyUI integration;
- training code;
- and a Nunchaku low-VRAM path documented for about 10 GB VRAM.
For a first specialized baseline, this is where I would start.
OSInsert
OSInsert is more interesting if the reference must change pose/viewpoint substantially.
Its two modes are conceptually useful even if you do not ultimately use the implementation:
conservative:
preserve reference more directly
aggressive:
first adapt pose/viewpoint to the scene
then restore reference details
That is almost exactly the design trade-off in your question.
The aggressive stack is considerably heavier than running Insert Anything alone.
Qwen-Image-Edit-2511
Qwen-Image-Edit-2511 is also worth having as a modern general-editor baseline. Its official materials emphasize improved consistency, geometric reasoning, new-view generation and industrial-design use cases.
I would treat it as a different comparison branch, though, not as proof that the specialized mask/reference problem has been solved.
At the time of writing, current Diffusers QwenImage documentation exposes multi-image editing through QwenImageEditPlusPipeline, while hard-mask inpainting is handled by a separate inpainting pipeline. So I would not assume that “multi-reference + hard target mask” is one simple unified Plus call.
Research references rather than first installs
For understanding the problem itself, I would also keep:
as useful maps of the current design space.
Background preservation, evaluation, hardware, and licensing
“Keep the background unchanged” has two possible meanings
These are not quite the same:
- keep the room/furniture/walls unchanged semantically, or
- keep every pixel outside the object mask unchanged.
If a newly inserted bed should cast a different shadow, reflection, contact shadow or ambient occlusion, physically natural integration may require changing pixels outside the literal object silhouette.
If requirement (2) is strict, a safer architecture can be:
generate/edit a controlled local region
↓
explicitly composite only the allowed pixels back
but that also limits the model’s freedom to generate environmental effects.
So I would define that requirement before evaluating “background preservation”.
Evaluate product identity separately from overall image quality
A visually nice image can still contain the wrong product.
I would not rely on a single CLIP/DINO similarity score.
ObjectMate specifically discusses the difference between semantic similarity and true instance identity retrieval.
A recent product-focused benchmark, ProductConsistency, likewise evaluates product identity as its own problem and still shows substantial room for improvement in current editors including Qwen-Image-Edit-2511 and FLUX.1-Kontext-dev.
For your bed example I would make the manual checklist primary:
headboard shape
leg count / leg shape
overall proportions
frame thickness
upholstery / stitching
decorative parts
logo/text if present
Then evaluate perspective, placement, lighting and background preservation separately.
Hardware changes the practical recommendation
The model list changes a lot depending on available VRAM and whether cloud inference is acceptable.
For example, the current Insert Anything README documents:
- a Nunchaku path for ~10 GB VRAM;
- higher-memory regular paths.
I was able to run the Nunchaku route on a Colab T4 for a small test, so a high-end workstation is not necessarily required just to test the idea.
On the other hand, pipelines such as DIRECT are much heavier: its demo pulls DIRECT, FLUX.1-Fill-dev, TRELLIS-image-large, SigLIP2 and RMBG-2.0, and its environment includes compiled CUDA extensions.
So I would choose the conceptual branch first, then choose an implementation that fits the hardware budget — rather than the other way around.
One licensing detail worth checking
Some of the most useful public-code pipelines depend on FLUX.1 [dev] models.
For example, Insert Anything depends on FLUX.1-Fill-dev / Redux-dev.
The current FLUX.1 Fill dev license permits use of the model/derivatives for non-commercial/non-production purposes unless a separate commercial license is obtained.
It separately states that generated outputs may be used for commercial purposes, subject to its terms.
So if this is eventually a production/product-visualization pipeline, I would check the complete dependency/license chain rather than assuming that the GitHub repository’s source-code license automatically governs the underlying model weights.
So, for your exact case, my default path would currently be:
first:
reference crop quality
+ same/near-view reference
+ tight-mask vs expanded-region A/B
then:
Insert Anything as a direct specialized baseline
if viewpoint adaptation is the main failure:
multi-view references
and/or an OSInsert-style geometry-adaptation branch
if exact product geometry is non-negotiable:
move geometry into CAD/3D,
and use the image model for compositing/harmonization
If you post the source image, current mask, reference image, and current result you mentioned, those four images would probably make the next diagnosis much more concrete. In particular, the actual mask shape and the source/reference viewpoint difference should make it possible to tell whether you are mainly hitting identity conditioning, a geometry/mask conflict, novel-view ambiguity, or scene harmonization.
And GPU/VRAM (or whether cloud inference is acceptable) would help narrow down which implementation branch is practical.
: