Hugging Face's logo

Diffusers documentation

JoyAI-Image-Edit-Plus

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

JoyAI-Image-Edit-Plus

JoyAI-Image is a unified multimodal foundation model for image understanding, text-to-image generation, and instruction-guided image editing. It combines an 8B Multimodal Large Language Model (MLLM) with a 16B Multimodal Diffusion Transformer (MMDiT).

JoyAI-Image-Edit-Plus is a multi-image instruction-guided editing model that accepts multiple reference images and a text instruction to generate a new image that combines elements from the references according to the instruction. It supports 1–5 reference images per sample.

ModelDescriptionDownload
JoyAI-Image-Edit-PlusMulti-image instruction-guided editing with element composition from multiple referencesHugging Face
import torch
from PIL import Image
from diffusers import JoyImageEditPlusPipeline

pipeline = JoyImageEditPlusPipeline.from_pretrained(
    "jdopensource/JoyAI-Image-Edit-Plus-Diffusers", dtype=torch.bfloat16
)
pipeline.to("cuda")

images = [
    Image.open("reference_0.png").convert("RGB"),
    Image.open("reference_1.png").convert("RGB"),
]

target_h, target_w = pipeline.image_processor.get_default_height_width(images[-1])

output = pipeline(
    images=images,
    prompt="Combine the person from the second image with the scene from the first image.",
    negative_prompt="low quality, blurry, deformed",
    height=target_h,
    width=target_w,
    num_inference_steps=30,
    guidance_scale=4.0,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
output.save("joyimage_edit_plus_output.png")

JoyImageEditPlusPipeline

class diffusers.JoyImageEditPlusPipeline

< >

( scheduler: FlowMatchEulerDiscreteSchedulervae: AutoencoderKLWantext_encoder: Qwen3VLForConditionalGenerationtokenizer: Qwen2Tokenizertransformer: JoyImageEditPlusTransformer3DModelprocessor: Qwen3VLProcessortext_token_max_length: int = 2048 )

Parameters

  • scheduler (FlowMatchEulerDiscreteScheduler) — A scheduler to be used in combination with transformer to denoise the encoded image latents.
  • vae (AutoencoderKLWan) — Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations.
  • text_encoder (Qwen3VLForConditionalGeneration) — Multimodal text encoder for prompt encoding with inline image understanding.
  • tokenizer (Qwen2Tokenizer) — Tokenizer for text processing.
  • transformer (JoyImageEditPlusTransformer3DModel) — Conditional Transformer (MMDiT) architecture to denoise the encoded image latents.
  • processor (Qwen3VLProcessor) — Processor for multimodal inputs (text + images).
  • text_token_max_length (int, defaults to 2048) — Maximum token length for text encoding.

Diffusion pipeline for multi-image instruction-guided editing using JoyImage Edit Plus.

Supports multiple reference images with different resolutions. Each reference image is independently VAE-encoded and patchified, then concatenated with the target noise patches for joint denoising.

__call__

< >

( images: list[PIL.Image.Image] | list[list[PIL.Image.Image]] | None = Noneprompt: str | list[str] = Noneheight: int | None = Nonewidth: int | None = Nonenum_inference_steps: int = 30timesteps: list = Nonesigmas: list = Noneguidance_scale: float = 4.0negative_prompt: str | list[str] | None = Nonegenerator: typing.Union[torch.Generator, list[torch.Generator], NoneType] = Nonelatents: typing.Optional[torch.Tensor] = Noneprompt_embeds: typing.Optional[torch.Tensor] = Noneprompt_embeds_mask: typing.Optional[torch.Tensor] = Nonenegative_prompt_embeds: typing.Optional[torch.Tensor] = Nonenegative_prompt_embeds_mask: typing.Optional[torch.Tensor] = Noneoutput_type: str | None = 'pil'return_dict: bool = Truecallback_on_step_end: typing.Union[typing.Callable[[int, int, dict], NoneType], diffusers.callbacks.PipelineCallback, diffusers.callbacks.MultiPipelineCallbacks, NoneType] = Nonecallback_on_step_end_tensor_inputs: list = ['latents']max_sequence_length: int = 4096 ) → JoyImageEditPlusPipelineOutput or tuple

Parameters

  • images (list[Image.Image] or list[list[Image.Image]], optional) — Reference images for editing. Each image can have a different resolution. If a flat list is provided, it is treated as one sample with multiple references.
  • prompt (str or list[str], optional) — The prompt or prompts to guide the image generation. If not defined, one has to pass prompt_embeds instead.
  • height (int, optional) — The height in pixels of the generated image. If None, determined from the last reference image.
  • width (int, optional) — The width in pixels of the generated image. If None, determined from the last reference image.
  • num_inference_steps (int, optional, defaults to 30) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.
  • timesteps (list[int], optional) — Custom timesteps to use for the denoising process. If not defined, equal spacing is used.
  • sigmas (list[float], optional) — Custom sigmas to use for the denoising process.
  • guidance_scale (float, optional, defaults to 4.0) — Classifier-free guidance scale. Higher values encourage the model to generate images more aligned with the prompt at the expense of lower image quality.
  • negative_prompt (str or list[str], optional) — The prompt or prompts not to guide the image generation. If not defined, a blank prompt is used for classifier-free guidance.
  • generator (torch.Generator or list[torch.Generator], optional) — One or a list of torch generator(s) to make generation deterministic.
  • latents (torch.Tensor, optional) — Pre-generated noisy latents to be used as inputs for image generation.
  • prompt_embeds (torch.Tensor, optional) — Pre-generated text embeddings. Can be used to easily tweak text inputs.
  • prompt_embeds_mask (torch.Tensor, optional) — Attention mask for pre-generated text embeddings.
  • negative_prompt_embeds (torch.Tensor, optional) — Pre-generated negative text embeddings.
  • negative_prompt_embeds_mask (torch.Tensor, optional) — Attention mask for pre-generated negative text embeddings.
  • output_type (str, optional, defaults to "pil") — The output format of the generated image. Choose between "pil" (PIL.Image.Image), "np" (np.ndarray), "pt" (torch.Tensor), or "latent" for raw latent output.
  • return_dict (bool, optional, defaults to True) — Whether or not to return a JoyImageEditPlusPipelineOutput instead of a plain tuple.
  • callback_on_step_end (Callable, optional) — A function called at the end of each denoising step with arguments: the pipeline, step index, timestep, and a dict of callback tensor inputs.
  • callback_on_step_end_tensor_inputs (list[str], optional, defaults to ["latents"]) — The list of tensor inputs for the callback_on_step_end function.
  • max_sequence_length (int, optional, defaults to 4096) — Maximum sequence length for the text encoder.

Returns

JoyImageEditPlusPipelineOutput or tuple

If return_dict is True, JoyImageEditPlusPipelineOutput is returned, otherwise a tuple is returned where the first element is a list of generated images.

Function invoked when calling the pipeline for generation.

Examples:

>>> import torch
>>> from diffusers import JoyImageEditPlusPipeline
>>> from diffusers.utils import load_image

>>> model_id = "jdopensource/JoyAI-Image-Edit-Plus-Diffusers"
>>> pipe = JoyImageEditPlusPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16)
>>> pipe.to("cuda")

>>> images = [
...     load_image("dog.png"),
...     load_image("person.png"),
... ]
>>> output = pipe(
...     images=images,
...     prompt="Let the person lovingly play with the dog.",
...     height=1024,
...     width=1024,
...     num_inference_steps=30,
...     guidance_scale=4.0,
...     generator=torch.manual_seed(42),
... )
>>> output.images[0].save("output.png")

encode_prompt_multiple_images

< >

( prompt: str | list[str]device: typing.Optional[torch.device] = Noneimages: list[PIL.Image.Image] | None = Nonemax_sequence_length: int | None = None )

Encode prompts with inline <image> tokens via the Qwen3-VL processor.

prepare_latents

< >

( batch_size: intnum_channels_latents: intheight: intwidth: intdtype: dtypedevice: devicegenerator: typing.Union[torch.Generator, list[torch.Generator], NoneType]reference_images: list[list[PIL.Image.Image]] | None = Nonelatents: typing.Optional[torch.Tensor] = None ) → padded_latents

Parameters

  • latents — Optional pre-computed noise for the target slot. Shape (B, C, 1, H', W') where H' and W' are the latent-space dimensions. When None, random noise is sampled.

Returns

padded_latents

[B, max_patches, C, pt, ph, pw] target_mask: [B, max_patches] (True for target patches) shape_list: per-sample list of (t, h, w) tuples for each component

Prepare 6D padded latent tensor with target noise + reference image latents.

JoyImageEditPlusPipelineOutput

class diffusers.JoyImageEditPlusPipelineOutput

< >

( images: typing.Union[typing.List[PIL.Image.Image], numpy.ndarray] )

Output class for JoyImage Edit Plus multi-image editing pipelines.

Update on GitHub

←JoyImage Edit Kandinsky 2.1→