README.md

---
license: apache-2.0
tags:
  - text-to-image
  - image-generation
---

<div align="center">
<h1>ReNeg: Learning Negative Embedding with Reward Guidance</h1>
</div>


<div align="center">

[![arXiv](https://img.shields.io/badge/arXiv%20paper-2412.19637-b31b1b.svg)](https://arxiv.org/abs/2412.19637)&nbsp;
[![code](https://img.shields.io/badge/%F0%9F%A4%96%20Code-AMD--AIG--AIMA/ReNeg-green)](https://github.com/AMD-AIG-AIMA/ReNeg)&nbsp;

</div>



We present **ReNeg**, a **Re**ward-guided approach that directly learns **Neg**ative embeddings through gradient descent. The global negative embeddings learned using **ReNeg** exhibit strong generalization capabilities and can be seamlessly adaptable to text-to-image and even text-to-video models. Strikingly simple yet highly effective, **ReNeg** amplifies the visual appeal of outputs from base Stable Diffusion models.

## Examples

Using the [🤗's Diffusers library](https://github.com/huggingface/diffusers) to run ReNeg in a simple and efficient manner.

```bash
pip install diffusers transformers accelerate
git clone https://github.com/AMD-AIG-AIMA/ReNeg.git
```
We provide three negative embeddings, including SD1.4, SD1.5, and SD2.1-base. Running ReNeg with a specific SD version as follows:

```python
import os
from pathlib import Path
import torch
from diffusers import (
    StableDiffusionPipeline,
    DDIMScheduler,
)
from safetensors.torch import load_file

model_path = "stable-diffusion-v1-5"
neg_embeddings_path = "checkpoints/sd1.5_reneg_emb.safetensors"
pipe = StableDiffusionPipeline.from_pretrained(
    model_path,
    safety_checker=None,
)
pipe.scheduler = DDIMScheduler.from_pretrained(
    model_path, subfolder="scheduler"
)
device = "cuda"
pipe.to(device)

neg_embeddings = load_file(neg_embeddings_path)["embedding"].to(device)  # Assuming the key is "embedding"
output = pipe(
    "A girl in a school uniform playing an electric guitar.",
    negative_prompt_embeds=neg_embeddings,
)

image = output.images[0]
# TextToImageModel is the model you want to evaluate
image.save("output.png")
```

To compare with the inference results using `neg_emb`, you can perform inference using only positive prompt.
+ To perform **inference using only the pos_prompt**, you need to run `inference.py` with `args.prompt_type = only_pos`.
```bash
python inference.py --model_path "your_sd1.5_path" --prompt_type "only_pos" --prompt "A girl in a school uniform playing an electric guitar."
```


## Citation

```
@misc{li2024reneg,
      title={ReNeg: Learning Negative Embedding with Reward Guidance},
      author={Xiaomin Li, Yixuan Liu, Takashi Isobe, Xu Jia, Qinpeng Cui, Dong Zhou, Dong Li, You He, Huchuan Lu, Zhongdao Wang, Emad Barsoum},
      year={2024},
      eprint={2412.19637},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```