Mouyang Cheng

Mouyang Cheng

AI for materials | Computational physics | Mingda Li Group, MIT

Target Score Identity and Denoising Score Identity in Diffusion Models

Diffusion models keep asking for the same object: the score of a noised distribution,

$$ s_t(y)=\nabla_y \log p_t(y). $$

Here $p_t$ is the marginal law after corrupting clean data $X_0\sim p_0$ into a noisy variable $Y_t$. For the usual Gaussian forward process,

$$ Y_t = \alpha_t X_0 + \sigma_t \varepsilon, \qquad \varepsilon\sim\mathcal N(0,I), \qquad p_{t\mid 0}(y\mid x)=\mathcal N(y;\alpha_t x,\sigma_t^2 I). $$

The marginal noised density is

$$ p_t(y)=\int p_0(x)\,p_{t\mid 0}(y\mid x)\,\mathrm dx. $$

The problem is that $p_t$ is usually intractable, so $s_t(y)$ is not directly available. The denoising score identity (DSI) and target score identity (TSI) give two different unbiased conditional signals for the same score.

1. The Two Identities

For the Gaussian corruption above, define the clean target score

$$ s_0(x)=\nabla_x\log p_0(x). $$

Then the two identities are

$$ \boxed{ s_t(y) = \mathbb E\!\left[ \nabla_y\log p_{t\mid 0}(y\mid X_0) \mid Y_t=y \right] } \qquad\text{(DSI)} $$

and

$$ \boxed{ s_t(y) = \frac{1}{\alpha_t} \mathbb E\!\left[ s_0(X_0) \mid Y_t=y \right] } \qquad\text{(TSI).} $$

Since

$$ \nabla_y\log p_{t\mid 0}(y\mid x) = \frac{\alpha_t x-y}{\sigma_t^2}, $$

DSI is often written as

$$ s_t(y) = \mathbb E\!\left[ \frac{\alpha_t X_0-y}{\sigma_t^2} \mid Y_t=y \right]. $$

The important point is not that these formulas look similar. They use different information:

  • DSI uses only the known noising kernel $p_{t\mid 0}(y\mid x)$ and paired samples $(X_0,Y_t)$.
  • TSI uses the score of the clean target density, $s_0(x)=\nabla_x\log p_0(x)$.

So DSI is natural for ordinary data-driven diffusion training. TSI is natural when $p_0$ is an energy, posterior, Boltzmann distribution, or any differentiable target whose log-gradient is available up to a normalizing constant.

2. Proof of DSI

Start from the marginal density

$$ p_t(y)=\int p_0(x)\,p_{t\mid 0}(y\mid x)\,\mathrm dx. $$

Differentiate under the integral:

$$ \nabla_y p_t(y) = \int p_0(x)\,\nabla_y p_{t\mid 0}(y\mid x)\,\mathrm dx. $$

Rewrite $\nabla_y p_{t\mid 0}=p_{t\mid 0}\nabla_y\log p_{t\mid 0}$:

$$ \nabla_y p_t(y) = \int p_0(x)\,p_{t\mid 0}(y\mid x) \nabla_y\log p_{t\mid 0}(y\mid x)\,\mathrm dx. $$

Divide by $p_t(y)$ and recognize Bayes’ posterior

$$ p_{0\mid t}(x\mid y) = \frac{p_0(x)p_{t\mid 0}(y\mid x)}{p_t(y)}. $$

This gives

$$ \nabla_y\log p_t(y) = \int \nabla_y\log p_{t\mid 0}(y\mid x)\, p_{0\mid t}(x\mid y)\,\mathrm dx. $$

That is DSI. For Gaussian noising, the conditional score is exactly $(\alpha_t x-y)/\sigma_t^2$.

3. Proof of TSI

TSI uses integration by parts. Assume $p_0$ is differentiable and boundary terms vanish. For isotropic Gaussian noising,

$$ \nabla_x p_{t\mid 0}(y\mid x) = -\alpha_t \nabla_y p_{t\mid 0}(y\mid x). $$

Now compute the posterior expectation of the clean score:

$$ \mathbb E[s_0(X_0)\mid Y_t=y] = \frac{1}{p_t(y)} \int \nabla_x\log p_0(x)\,p_0(x)\,p_{t\mid 0}(y\mid x)\,\mathrm dx. $$

Since $\nabla_x\log p_0(x)\,p_0(x)=\nabla_x p_0(x)$,

$$ \mathbb E[s_0(X_0)\mid Y_t=y] = \frac{1}{p_t(y)} \int \nabla_x p_0(x)\,p_{t\mid 0}(y\mid x)\,\mathrm dx. $$

Integrate by parts:

$$ \int \nabla_x p_0(x)\,p_{t\mid 0}(y\mid x)\,\mathrm dx = -\int p_0(x)\,\nabla_x p_{t\mid 0}(y\mid x)\,\mathrm dx. $$

Use $\nabla_x p_{t\mid 0}=-\alpha_t\nabla_y p_{t\mid 0}$:

$$ \int \nabla_x p_0(x)\,p_{t\mid 0}(y\mid x)\,\mathrm dx = \alpha_t \int p_0(x)\,\nabla_y p_{t\mid 0}(y\mid x)\,\mathrm dx = \alpha_t \nabla_y p_t(y). $$

Therefore

$$ \mathbb E[s_0(X_0)\mid Y_t=y] = \alpha_t\nabla_y\log p_t(y) = \alpha_t s_t(y), $$

or

$$ s_t(y) = \frac{1}{\alpha_t} \mathbb E[s_0(X_0)\mid Y_t=y]. $$

That is TSI.

4. Connection to Tweedie’s Formula

Tweedie’s formula says that, under Gaussian corruption, the posterior mean of the clean variable is obtained from the noisy observation plus a score correction. In the simplest variance-exploding case,

$$ Y=X+\sigma\varepsilon, \qquad \varepsilon\sim\mathcal N(0,I), $$

DSI gives

$$ s_\sigma(y) = \mathbb E\!\left[ \frac{X-y}{\sigma^2} \mid Y=y \right] = \frac{\mathbb E[X\mid Y=y]-y}{\sigma^2}. $$

Rearranging,

$$ \boxed{ \mathbb E[X\mid Y=y] = y+\sigma^2 s_\sigma(y). } $$

For the more common scaled form $Y_t=\alpha_tX_0+\sigma_t\varepsilon$,

$$ \boxed{ \mathbb E[X_0\mid Y_t=y] = \frac{y+\sigma_t^2 s_t(y)}{\alpha_t}. } $$

This is why DSI, Tweedie’s formula, and denoising are almost the same story told from different angles:

  • the score tells you how to move a noisy point toward regions of higher marginal probability;
  • the posterior mean tells you the Bayes-optimal denoised clean estimate;
  • the residual $\mathbb E[\varepsilon\mid Y_t=y]$ satisfies
$$ \mathbb E[\varepsilon\mid Y_t=y] = -\sigma_t s_t(y). $$

In neural diffusion models, these are just different parameterizations. Predicting score, noise, clean data, and velocity are algebraically connected once $\alpha_t$ and $\sigma_t$ are fixed.

5. Score Matching View

Suppose we train a network $s_\theta(y,t)$ with squared loss. DSI uses the target

$$ b(X_0,Y_t,t) = \nabla_y\log p_{t\mid 0}(Y_t\mid X_0) = \frac{\alpha_tX_0-Y_t}{\sigma_t^2}. $$

TSI uses the target

$$ c(X_0,t) = \frac{1}{\alpha_t}s_0(X_0). $$

Both have the same conditional mean:

$$ \mathbb E[b(X_0,Y_t,t)\mid Y_t=y] = \mathbb E[c(X_0,t)\mid Y_t=y] = s_t(y). $$

Therefore, for either target $U\in{b,c}$, the minimizer of

$$ \mathbb E\left[ \lambda(t)\lVert s_\theta(Y_t,t)-U\rVert^2 \right] $$

is the same ideal function:

$$ s_\theta^\star(y,t)=s_t(y). $$

The difference is variance, not the population optimum. DSI and TSI supervise the same score through different noisy labels.

6. When to Use DSI

Use DSI when you have samples but not the clean data score.

This is the standard image/audio/text-latent diffusion setting. You can sample $X_0$ from the dataset, sample $\varepsilon$, construct $Y_t=\alpha_tX_0+\sigma_t\varepsilon$, and compute the denoising target from the known Gaussian kernel. You do not need $\nabla_x\log p_0(x)$, which is good because for an empirical dataset it is usually not a meaningful smooth object.

DSI is especially convenient at moderate and high noise levels. But at very low noise,

$$ b(X_0,Y_t,t) = -\frac{\varepsilon}{\sigma_t}, $$

up to the usual scaled-noising convention. As $\sigma_t\to 0$, this target can have large variance even though its conditional mean remains finite. This is one reason low-noise score estimation can be delicate.

7. When to Use TSI

Use TSI when the clean target score is available or can be estimated reliably:

  • energy-based sampling, where $p_0(x)\propto \exp(-E(x))$ and $s_0(x)=-\nabla E(x)$;
  • Bayesian inverse problems, where the posterior score is the gradient of log prior plus log likelihood;
  • molecular, statistical physics, or Monte Carlo settings with differentiable unnormalized densities;
  • score distillation or hybrid methods where a proxy clean score is available.

The normalizing constant of $p_0$ is irrelevant because

$$ \nabla_x\log p_0(x) = \nabla_x\log \bar p_0(x) $$

when $p_0(x)=\bar p_0(x)/Z$.

TSI is attractive at low noise: as $\alpha_t\approx 1$, its signal is close to the clean target score. But at very high noise in variance-preserving or Ornstein-Uhlenbeck noising, $\alpha_t\to 0$, so

$$ c(X_0,t)=\frac{s_0(X_0)}{\alpha_t} $$

can become high-variance or numerically unstable.

8. Practical Rule of Thumb

DSI and TSI are two unbiased labels for the same conditional mean. Pick the one whose random label has lower variance in the regime you care about.

Regime / information Prefer Reason
Only dataset samples are available DSI No clean target score is needed.
Clean density score $\nabla\log p_0$ is available TSI at low noise Avoids the $1/\sigma_t$ low-noise explosion of denoising targets.
High-noise VP/OU regime, $\alpha_t\ll 1$ DSI TSI has the $1/\alpha_t$ scaling.
Physics, Bayesian, or energy-based sampling TSI or a blend Target score is often cheap relative to learning a low-noise denoiser.
Full diffusion schedule with both signals available Blend / control variate Both identities have the same mean but different variance profiles.

A useful mental model is:

$$ \text{DSI: score from the corruption residual,} \qquad \text{TSI: score from the clean target force.} $$

In a physical system, DSI says “infer the force from how the noisy observation was generated.” TSI says “average the true clean force over all clean states compatible with the noisy observation.”

9. References

  • Vincent, “A Connection Between Score Matching and Denoising Autoencoders” (2011): Neural Computation.
  • Song and Ermon, “Generative Modeling by Estimating Gradients of the Data Distribution” (2019): arXiv:1907.05600.
  • Song et al., “Score-Based Generative Modeling through Stochastic Differential Equations” (2021): ICLR / OpenReview.
  • De Bortoli, Hutchinson, Wirnsberger, and Doucet, “Target Score Matching” (2024): arXiv:2402.08667.
  • Duston and Bui-Thanh, “Variance-Reduced Diffusion Sampling via Target Score Identity” (2026): arXiv:2601.01594.