---
license: mit
library_name: diffusers
pipeline_tag: text-to-image
language:
- en
tags:
- diffusers
- matfuse
- pbr
- material-generation
- svbrdf
- text-to-image
---


# MatFuse — Controllable Material Generation with Diffusion Models

MatFuse generates tileable PBR material maps (diffuse, normal, roughness,
specular) from text, reference images, sketches, and/or color palettes.

> **Paper:** [MatFuse: Controllable Material Generation with Diffusion Models](https://arxiv.org/abs/2308.11408) — CVPR 2024
> **Project page:** <https://gvecchio.com/matfuse/>

## Quick Start

```python
import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "gvecchio/MatFuse",
    trust_remote_code=True,
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

result = pipe(
    text="red brick wall",
    num_inference_steps=50,
    guidance_scale=4.0,
    generator=torch.Generator("cuda").manual_seed(42),
)

result["diffuse"][0].save("diffuse.png")
result["normal"][0].save("normal.png")
result["roughness"][0].save("roughness.png")
result["specular"][0].save("specular.png")
```

## Conditioning Inputs

All conditions are **optional** and freely composable:

| Input | Type | Description |
|-------|------|-------------|
| `text` | `str` | Text description of the material |
| `image` | `PIL.Image` | Reference image for style/appearance |
| `sketch` | `PIL.Image` (grayscale) | Binary edge map for structure |
| `palette` | `list[tuple]` | Up to 5 RGB colour tuples (0–255) |

```python
from PIL import Image

result = pipe(
    image=Image.open("reference.png"),
    text="rough stone texture",
    palette=[(120, 80, 60), (90, 60, 40), (150, 110, 80), (70, 50, 30), (180, 140, 100)],
    num_inference_steps=50,
    guidance_scale=4.0,
)
```

## Architecture

| Component | Class | Key parameters |
|-----------|-------|----------------|
| **UNet** | `UNet2DConditionModel` | in=16, out=12, blocks=[256,512,1024], cross_attn=512 |
| **VAE** | `MatFuseVQModel` (custom) | 4 encoders + 4 VQ codebooks (4096×3), shared decoder, f=8 |
| **Scheduler** | `DDIMScheduler` | β 0.0015–0.0195, scaled_linear, ε-prediction |
| **Conditioning** | `MultiConditionEncoder` (custom) | CLIP ViT-B/16 · sentence-transformers · palette MLP · sketch CNN |

## 📜 Citation

```bibtex
@inproceedings{vecchio2024matfuse,
  author    = {Vecchio, Giuseppe and Sortino, Renato and Palazzo, Simone and Spampinato, Concetto},
  title     = {MatFuse: Controllable Material Generation with Diffusion Models},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024},
  pages     = {4429-4438}
}
```

## License

This project is licensed under the MIT License.