Given a factorization of an image into a sum of components, we present a zero-shot method to independently control these components through diffusion model sampling. For example, decomposing an image into low and high spatial frequencies, and then prompting these components on different text prompts allows us to produce hybrid images. A decomposition into grayscale and color components results in images that change appearance when colored is added or subtracted. We also show how to make perceptual illusions involving motion blur and hybrid images with three prompts. Finally, by holding one component constant while generating the other components we can create hybrid images from real images.
Given an image decomposition, we control components of the decomposition through text conditioning during image generation. To do this, we modify the sampling procedure of a pretrained diffusion model. Specifically, at each denoising step, \( t \), we construct a new noise estimate, \(\tilde\epsilon\), to use for denoising, whose components come from components of \(\epsilon_i\), which are noise estimates conditioned on different prompts. Here, we show a decomposition into three frequency subbands, used for creating triple hybrid images, but we consider a number of other decompositions, which we explain below.
In order to use our method we need to find image decompositions of the form \( \mathbf{x} = \sum f_i(\mathbf{x}) \). Below, we briefly describe the decompositions that we consider, and what kinds of images they produce.
To make hybrid images, we can decompose an image into high and low frequency components. We use a low pass filter, implemented as a Gaussian blur, \( G_\sigma \), with standard deviation \( \sigma \) as one component, and we take the residual high pass as the other component. \[ \begin{aligned} \mathbf{x} = \underbrace{\mathbf{x} - G_\sigma(\mathbf{x})}_{f_\text{high}(\mathbf{x})} + \underbrace{G_\sigma(\mathbf{x})}_{f_\text{low}(\mathbf{x})} \end{aligned} \]
To make triple hybrid images we decompose an image as a three layer Laplacian pyramid where we use two Gaussian blurs of standard deviation \(\sigma_1\) and \(\sigma_2\). \[ \begin{aligned} \mathbf{x} &= \underbrace{G_{\sigma_1}(\mathbf{x}) - G_{\sigma_2}(G_{\sigma_1}(\mathbf{x}))}_{f_\text{med}(\mathbf{x})} \;+ \\ &\underbrace{\mathbf{x} - G_{\sigma_1}(\mathbf{x})}_{f_\text{high}(\mathbf{x})} + \underbrace{G_{\sigma_2}(G_{\sigma_1}(\mathbf{x}))}_{f_\text{low}(\mathbf{x})} \end{aligned} \]
To make color hybrids, we take the grayscale image as one component, and the residual as the color component. \[ \begin{aligned} f_\text{gray}(\mathbf{x}) &= \frac{1}{3} \sum_c \mathbf{x}_c \\ f_\text{color}(\mathbf{x}) &= \mathbf{x} - f_\text{gray}(\mathbf{x}) \end{aligned} \]
For a motion blur kernel, \( \mathbf{K} \), we can decompose an image into a blurred component and a residual component. \[ \begin{aligned} \mathbf{x} = \underbrace{\mathbf{K}*\mathbf{x}}_{f_\text{motion}(\mathbf{x})} + \;\; \underbrace{\mathbf{x} - \mathbf{K}*\mathbf{x}}_{f_\text{res}(\mathbf{x})}, \end{aligned} \] where \( * \) denotes convolution. This allows us to make motion hybrids.
Given \( N \) binary spatial masks, \( \mathbf{m_i} \), whose disjoint union covers the entire image, we can partition the image into regions with the decomposition \[ \begin{aligned} \mathbf{x} = \sum_i \underbrace{\mathbf{m}_i \odot \mathbf{x}}_{f_i(\mathbf{x})}, \end{aligned} \] where \( \odot \) denotes element-wise multiplication. The effect of this is to enable spatial control of prompts, and is a special case of MultiDiffusion [2].
We may also decompose an image as \[ \begin{aligned} \mathbf{x} = \sum_i^N a_i\mathbf{x} \end{aligned} \] for \( \sum_i^N a_i = 1 \). Setting \( a_i = \frac{1}{N} \) recovers exactly the compositional method of Liu et al. [3], and allows us to produce images that align with multiple prompts.
Diffusion models update noisy images, \(\mathbf{x}_t\), to less noisy images, \(\mathbf{x}_{t-1}\), with an \(\texttt{update}(\cdot,\cdot)\) function1. Commonly used update functions include DDPM and DDIM, and are linear combinations of the noisy image, \(\mathbf{x}_t\), and the noise estimate \(\epsilon_\theta\).2 That is, these updates can be written as \[ \begin{aligned} \mathbf{x}_{t-1} &= \texttt{update}(\mathbf{x}_t, \epsilon_\theta) \\ &=\omega_t \mathbf{x}_t + \gamma_t \epsilon_\theta \end{aligned} \] where \(\omega_t\) and \(\gamma_t\) are determined by the variance schedule and the scheduler. Then given a decomposition \( \mathbf{x} = \sum f_i(\mathbf{x}) \), this means the update rule can be decomposed into a sum of updates on components: \[ \begin{aligned} \mathbf{x}_{t-1} &= \texttt{update}(\mathbf{x}_t, \epsilon) \\ &= \texttt{update}\left( \sum f_i(\mathbf{x}_t), \sum f_i(\epsilon) \right) \\ &= \sum_i \texttt{update}(f_i(\mathbf{x}_t), f_i(\epsilon)) \end{aligned} \] where the last equality is by linearity of \( \texttt{update}(\cdot,\cdot) \). Our method can be understood as conditioning each of these components on a different text prompt. Written explicitly, for text prompts \( y_i \) our method is \[ \begin{aligned} \mathbf{x}_{t-1} = \sum_i \texttt{update}(f_i(\mathbf{x}_t), f_i(\epsilon(\mathbf{x}_t, y_i, t))). \end{aligned} \] Moreover, if the \( f_i \)'s are linear then we have \[ \begin{aligned} f_i(\mathbf{x}_{t-1}) &= f_i(\texttt{update}(\mathbf{x}_t, \epsilon)) \\ &= f_i(\omega_t\mathbf{x}_t + \gamma_t\epsilon_\theta) \\ &= \omega_t f_i(\mathbf{x}_t) + \gamma_t f_i(\epsilon_\theta) \\ &= \texttt{update}(f_i(\mathbf{x}_t), f_i(\epsilon_\theta)), \end{aligned} \] meaning that updating using the \(i\)th component of \(\mathbf{x}_t\) with the \(i\)th component of \(\epsilon_\theta\) will only affect the \(i\)th component of \(\mathbf{x}_{t-1}\).
1The update function also depends on \(t\), which we omit for brevity.
2Noise of the form \( \mathbf{z}\sim\mathcal{N}(0,\mathbf{I}) \) is also often added, which can be safely ignored. Please see the paper for details.
If we know what one of the components must be, perhaps from some reference image \( \mathbf{x}_\text{ref} \), then we can hold that component constant while generating the other components. In practice we do this by reprojecting the noisy image \( \mathbf{x}_t \) at every time step: \[ \mathbf{x}_t \gets f_1\left(\sqrt{\alpha_t}\mathbf{x}_\text{ref} + \sqrt{1 - \alpha_t}\epsilon\right) + \sum_{i=2}^N f_i(\mathbf{x}_t) \] This is effectively a way to solve (noiseless) inverse problems, with forward model \( \mathbf{y} = f_1(\mathbf{x}) \), and can be seen as a rudimentary version of prior work [4][5][6][7][8]. We apply this approach to generating hybrid images from real images, which we show above, and in a gallery of results.
One major limitation of our method is that the success rate is relatively low. While our method can produce decent images consistently, very high quality images are rarer. We attribute this fragility to the fact that our method produces images that are highly out-of-distribution for the diffusion model. In addition, there is no mechanism by which prompts associated with one component are discouraged from appearing in other components. Another failure case of our method is that the prompt for one component may dominate the generated image. Empirically, the success rate of our method can be improved by carefully choosing prompt pairs or by manually tuning decomposition parameters, but we leave improving the robustness of our method in general to future work. We show random results for selected prompt pairs below. Many more random results can be found in our paper.
This project is related to a number of other works, including:
Recent work by a pseudonymous artist, Ugleh, uses a Stable Diffusion model finetuned for generating QR codes to produce images whose global structure subtly matches a given template image. These images can effectively act as hybrid images. A demo of this approach can be found here.
Prior work, Generative Powers of Ten by Xiaojuan Wang et al., first showed that an image could be denoised with different conditioning in different frequency subbands. They apply this to the task of generating extreme zooms of images, where a stack of images represents different zoom levels. Each zoom-level is then conditioned on a different prompt, with zoomed-out images serving as constraints on the low frequency components of zoomed-in images.
Work by Matthew Tancik, and our prior work, Visual Anagrams, show how to make illusions by denoising multiple views of an image simultaneously. These methods work by transforming the inputs to the diffusion model, while factorized diffusion modifies the outputs of the diffusion model, producing a different class of illusions.
Another approach to making illusions with diffusion models is to use score distillation sampling, which is done in Diffusion Illusions, by Ryan Burgert et al.
[1] Oliva et al., “Hybrid Images”, TOG, 2006. ↩
[2] Bar-Tal et al., “MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation”, ICML, 2023. ↩
[3] Liu et al., “Compositional Visual Generation with Composable Diffusion Models”, ECCV, 2022. ↩
[4] Song et al., “Score-Based Generative Modeling through Stochastic Differential Equations”, ICLR, 2021. ↩
[5] Chung et al., “Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction”, CVPR, 2022. ↩
[6] Kawar et al., “Denoising Diffusion Restoration Models”, NeurIPS, 2022. ↩
[7] Lugmayr et al., “RePaint: Inpainting using Denoising Diffusion Probabilistic Models”, CVPR, 2022. ↩
[8] Wang et al., “Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model”, ICLR, 2023. ↩
@article{geng2024factorized,
title = {Factorized Diffusion: Perceptual Illusions by Noise Decomposition},
author = {Geng, Daniel and Park, Inbum and Owens, Andrew},
journal = {arXiv:2404.11615},
year = {2024},
month = {April},
abbr = {Preprint},
url = {https://arxiv.org/abs/2404.11615},
}