IP-Adapter — Definition, How It Works & Use With ControlNet
IP-Adapter (Image Prompt Adapter) is a lightweight add-on that lets an AI image model take a reference image as a prompt: you give it a photo of a face, a style or an object, and the model generates new pictures that keep that look. It brings "an image is worth a thousand words" to /glossary/diffusion models without retraining them.
What is IP-Adapter?
IP-Adapter, short for Image Prompt Adapter, is a small neural-network module that gives a text-to-image /glossary/diffusion model the ability to use an image as a prompt. Normally you steer a model like /alternatives/stable-diffusion with a text /glossary/prompt only; IP-Adapter lets you also feed a reference picture — a person's face, a painting's style, a product — and the generated images inherit that subject or look. It was introduced in 2023 by Hu Ye and colleagues at Tencent AI Lab in the paper "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models" (arxiv.org/abs/2308.06721). The whole adapter is only about 22M parameters, yet the authors report it can match or beat a fully fine-tuned image-prompt model.
How IP-Adapter works
IP-Adapter runs your reference image through a CLIP image encoder to get a compact feature representation, then projects it into a short sequence of image tokens. Its key idea is a "decoupled cross-attention" mechanism: for every text cross-attention layer in the model's U-Net it adds a second, separate cross-attention layer just for those image tokens, and only these new layers are trained while the base model stays frozen. The two attention outputs are summed, so the text prompt and the image prompt each get their own dedicated pathway instead of fighting over one. Because the base model is untouched, one trained IP-Adapter transfers to any custom checkpoint fine-tuned from the same base.
What IP-Adapter is used for
Because it treats a picture as a prompt, IP-Adapter is the go-to way to carry a look from one image into new ones. Common uses:
- Face consistency — the IP-Adapter FaceID variant reads a face embedding so a person keeps their likeness across many generated shots.
- Style transfer — feed a reference painting or photo and generate fresh scenes in that exact style.
- Subject / product reference — keep a specific object or character recognisable across images.
- Mixing image + text — combine a reference photo with a text prompt (e.g. "this face, cyberpunk city at night") for multimodal control.
- Stacking with ControlNet — use IP-Adapter for who/what it looks like and /glossary/controlnet for pose and composition at the same time.
IP-Adapter vs ControlNet vs LoRA
These three levers are easy to confuse because all add control on top of a /glossary/diffusion model, but they answer different questions. A /glossary/lora retrains the model to learn a new concept — a specific face or style baked into weights — which needs a training run. /glossary/controlnet fixes structure — pose, edges, depth — for a single generation. IP-Adapter carries appearance from a reference image without any training at all: you just supply the picture at generation time. In practice they combine — an IP-Adapter for the face, a ControlNet for the pose, a text /glossary/prompt and /glossary/cfg for everything else — all sharing the same frozen base model.
How Twin AI uses image-prompt technology
Twin AI is built on exactly what IP-Adapter enables: you upload a few photos and it generates new images that keep your face and style, without you touching adapters, encoders or a /glossary/seed. That reference-driven likeness powers /use-cases/ai-photoshoot, /use-cases/avatar and /use-cases/business-portrait, where consistency across a whole shoot is the point. Because Twin runs several image models behind one interface, your reference works across them, and you can compare results — for example in /compare/midjourney-vs-flux. Everything is payable with a Russian card or SBP, with no VPN required. Start free in /create/photo.
FAQ
What is IP-Adapter?
IP-Adapter (Image Prompt Adapter) is a lightweight module that lets a text-to-image diffusion model use a reference image as a prompt. You give it a photo of a face, style or object and it generates new images that keep that look, working alongside the text prompt.
How does IP-Adapter work?
It encodes your reference image with a CLIP image encoder into image tokens, then uses a "decoupled cross-attention" mechanism that adds a separate cross-attention layer for those tokens in each U-Net block. Only the new layers are trained; the base model stays frozen, so one adapter transfers across checkpoints.
What is the difference between IP-Adapter and ControlNet?
IP-Adapter controls appearance — who or what the image looks like — from a reference picture, while ControlNet controls structure like pose, edges or depth. They are complementary and are often used together, for example an IP-Adapter face plus an OpenPose ControlNet.
What is the difference between IP-Adapter and LoRA?
A LoRA retrains the model to learn a concept and needs a training run, baking it into weights. IP-Adapter needs no training — you simply supply a reference image at generation time. LoRA is best for a concept you reuse constantly; IP-Adapter is best for a one-off or on-the-fly reference.
Do I need IP-Adapter in Twin AI?
No. Twin AI already generates images that keep your face and style from the photos you upload, so you get image-prompt results without setting up IP-Adapter, encoders or control models yourself. Just upload photos in /create/photo.