Light

PaperNeurIPS 2026Decision-making under uncertainty

Thompson Sampling using Prior-fitted Diffusion Transformers

Sihwa Park, Jingsen Zhu, Vinamr Jain, Sheng-Yen Chou, Alexander Terenin

The problem

Imagine you work in a startup and are responsible for running experiments to design a new superconducting material. But the investors are in a hurry! ⏳ So you only get a handful of attempts. Is there a better strategy than randomly guessing suitable designs?

A black box: you choose a design, run the experiment and get results. The oracle to the material world 🌎 is hidden

Explore or exploit?

The range of conductivity values of different design possibilities varies continuously. You may have some domain knowledge, for eg. similar materials have similar properties. (Is there a way you can use this prior knowledge or belief about material properties?) Suppose you've already designed a few materials and you need to decide what to design next. Maybe you should design a material similar to the best one so far, but maybe if you design one completely different from the rest you might be pleasantly surprised (High risk = High reward?). This is known as the exploration-exploitation tradeoff in decision problems. Since you don't know anything about the landscape of material properties (except that maybe it's a continuous function), this particular kind of decision problem is known as black-box optimization. If only someone had been in a similar situation 🥹 so you could take inspiration from what they did!

To Explore or To Exploit, that is the question: test next to your best result if you're risk averse; A real G explores and goes all or nothing!

Practice makes perfect

It is now you remember that god is within you, so using your domain knowledge:

  1. You create a simulator on your computer that generates synthetic "worlds" or function landscapes of material properties, similar to the ones observed in the real world
  2. In these synthetic worlds you similarly run a bunch of experiments
  3. Based on those experiments, you ask "Given the designed materials and their observed properties, what design would have the highest conductivity?", only this time, since you created this world, you know the answer!

Wait, why is this a good idea? Assuming you're not living under a rock, you've heard the average tech-bro talk about training AI models: Using a bunch of input-output examples you can get a model which, given an input, gives a prediction of the output! In fact, you can use this simulator to generate synthetic datasets, where you input a bunch of experiment results as context and you want to predict the likelihood of a candidate being the best design (aka the posterior belief). The AI models (aka artificial neural networks) fitted (trained) using synthetic data generated using your prior (domain) knowledge are known as Prior-fitted Networks!

The simulator makes up worlds where the best design is known. Each one is a practice question, with its answer, for the model.

Test what's best

The flavour of decision rules (aka acquisition rules) that uses a likelihood or probability distribution to model the uncertainty to perform an "optimization" task, is called Bayesian Optimization (Remember Bayesian probability? you'd remember if you didn't skip high school 😏). A popular acquisition rule is to pick the next candidate design with probability proportional to the posterior belief of it being the best candidate. This rule is known as Thompson Sampling 🎲.

Finally how do we get these posterior belief values over a continuous range of design possibilities? Diffusion Transformers are AI models that work by diffusing or transporting potential design points randomly drawn from noise to more concentrated regions proportional to the posterior belief to get a continuous distribution.

The model starts from random dots and moves them to where the best design is likely to be. Across: the material design; up: how likely that design is to be the best. Picking a dot at random and testing it is Thompson sampling, so likelier designs get picked more often.

All together now

Bringing all these parts together, this work developed Thompson Sampling using Prior-fitted Diffusion Transformers for black-box optimization! (P.S. Random search lowkey sucks!)

Left: the model practices on simulated worlds, where the best design is known. Right, this time for real: The model assembles noise to areas it thinks are likely to be the best design, a candidate is then picked and tested, and future guesses take this new result in context before making new predictions
Left: the model practices on simulated worlds, where the best design is known. Right: this time for real, The model assembles noise to areas it thinks are likely to be the best design, a candidate is then picked and tested, and future guesses take this new result in context before making new predictions

The problem

In Black-box optimization, we ask for the maximum of an unknown function f:X→Rf : X \to \mathbb{R}, typically when you have limited evaluations (eg. a few dozen to a few hundred), so every choice of where to evaluate next has to trade off exploring against exploiting. Bayesian optimization handles this by placing a prior over ff and applying a decision rule to the posterior. Thompson sampling is the most common randomized rule: draw one function from the posterior and evaluate at its maximizer,

xt+1=arg max⁡x∈X (f∣x1:t,y1:t)(x).x_{t+1} = \operatorname*{arg\,max}_{x \in X} \, (f \mid x_{1:t}, y_{1:t})(x).

How well this works depends on the prior, yet in continuous domains Thompson sampling has in practice been limited to Gaussian process priors, because those are essentially the only ones whose posterior over functions is tractable. Existing diffusion-based optimizers avoid Gaussian processes but rely on guidance, bespoke losses and regularizers, or several interacting models to control exploration.

Key idea

Thompson sampling only needs samples of the maximizer xf∗x^*_f given the data, not the whole posterior over functions. Prior-fitted networks learn posterior distributions in-context from synthetic data drawn from the prior, and a diffusion model can represent a distribution over a continuous xx without the output-space discretization earlier prior-fitted networks used. Combining the two gives a prior-fitted diffusion transformer: a model that takes the observations so far as context and generates a draw from the Thompson sampling distribution.

From noise to guesses The model starts from random dots and moves them to where the best design is likely to be. Picking a dot at random is Thompson sampling: likelier designs get picked more often. start: random noisedots concentrate where the best design is likelypick at random: likelier designs come up more often the model's guessespicked to test nextmaterial design →how likely →
What the model outputs, in one dimension: samples that start as noise and end distributed like the maximiser given the data. Taking one of them at random is a Thompson sampling step. (Illustration, not from the paper.)

Method

The pipeline (Figure 2 of the paper, recoloured): sample a random function f from the prior, take in-context points (x_i, y_i) from it, and train a diffusion transformer to predict its maximiser x*.

Pre-training data is generated on the fly. Draw f∼p(f)f \sim p(f) and inputs x1:n∼p(x)x_{1:n} \sim p(x) with a random context size nn, set yi=f(xi)y_i = f(x_i), and use the function’s maximizer xf∗=arg⁡max⁡x∈Xf(x)x^*_f = \arg\max_{x \in X} f(x) as the target, which is either known analytically or found by gradient-based optimization. The model is trained with the standard flow matching loss,

θ=arg min⁡θ∈Θ  E LFM(xf∗,  fθ(x1:n,y1:n)),\theta = \operatorname*{arg\,min}_{\theta \in \Theta} \; \mathbb{E}\, \mathcal{L}_{\mathrm{FM}}\big(x^*_f,\; f_\theta(x_{1:n}, y_{1:n})\big),

which, over the space of all functions, targets the conditional distribution of xf∗x^*_f given the context, that is, the Thompson sampling distribution. The architecture is a diffusion transformer: self-attention over the observations with no positional encodings or causal masking, so the output does not depend on data order, and cross-attention from the flow-matching state to those embeddings.

Priors

Any prior works as long as its functions can be sampled and evaluated pointwise, and their maximum is known or they can be differentiated. Besides Gaussian processes, the paper proposes an envelope-plus-wiggles family,

f(x)=E(x;x∗)+g(x) W(x),f(x) = E(x; x^*) + g(x)\, W(x),

where x∗x^* is uniform over the domain, the envelope EE is a unimodal curve centered at x∗x^*, the wiggles WW add spurious local optima, and gg suppresses them near the optimum. Setting the wiggles to zero and building EE by warping a quadratic through a random normalizing flow gives a prior over unimodal, possibly non-convex functions, which is hard to express with a Gaussian process. The paper also finds that placing some of the pre-training inputs nearer the maximizer, rather than uniformly, can improve performance.

Envelope plus wiggles Draws from an envelope-plus-wiggles prior: a unimodal envelope fixes where the maximum is, wiggles add local optima away from it. envelope plus wigglesenvelope Ea draw f = E + gWits maximiser x*, knownx →f(x) →
Draws from an envelope-plus-wiggles prior in one dimension: the envelope fixes the maximiser, the wiggles add local optima away from it. (Illustration, not from the paper.)

Results

All experiments use batches of 8 parallel evaluations and 16 seeds, and plot medians with 25th and 75th quantiles.

Does the network learn the posterior?

Model-matched setting (Figure 4 of the paper): objectives are drawn from a Gaussian process prior, and the diffusion transformer (PFN, accent) is pre-trained on that same prior, against exact Gaussian process Thompson sampling (GP-TS, orange) and random search (RS, dotted). Medians with 25th and 75th quantiles; d = 4, 8, 16.

With the prior held fixed, any gap to exact Gaussian process Thompson sampling is approximation error from pre-training. The two are close in d=4d=4 and d=8d=8; in d=16d=16 a minor gap appears, which is mild compared with the gap to random search.

Non-Gaussian priors on synthetic benchmarks

Ackley, Rosenbrock and Levy in d = 8, 16, 25 (Figure 5 of the paper). Accent shades and solid grey: PFN under the unimodal, EWP-GP, EWP-U and problem-specific priors. Orange shades: Gaussian process baselines (LogEI, GP-TS, GP-UCB). Dashed: diffusion baselines (DiBO, Diff-BBO, DDOM). Dotted: random search. Medians with 25th and 75th quantiles.

On almost all time horizons at least one prior-fitted diffusion transformer outperforms the diffusion-based baselines; the exception is Ackley in d=8d=8, where DiBO is stronger and Gaussian process baselines are stronger still. Which prior is best depends on the problem: the unimodal prior tends to win in d=8d=8, the two priors with wiggles in d=16d=16 and d=25d=25. The problem-specific prior tends to do worse than the problem-independent ones, which the paper attributes to it exploring more than is ideal on these short horizons.

Empirical benchmarks

Best observed value on Lunar Lander (d = 12), Swimmer (d = 16) and Pest Control (d = 25) (Figure 6 of the paper); higher is better. Colours as in the previous figure, without the problem-specific prior.

Prior-fitted diffusion transformers outperform all baselines on Lunar Lander and approximately tie several of them on Swimmer. On Pest Control, Gaussian process baselines and Diff-BBO are strongest.

Scaling

IsoFLOP analysis (Figure 3 of the paper). Left: final regret against model size at five pre-training budgets, grey (smallest, 1.4 × 10^14 FLOPs) to accent (largest, 2.7 × 10^16). Middle, right: the optimal model size and number of training iterations at each budget, with a linear fit (ink) and a capped fit (orange).

Because training data is synthetic and unlimited, a fixed compute budget can be spent on a larger model or on more pre-training iterations. An isoFLOP analysis under the envelope-plus-wiggles prior picks the best model size at each budget and fits regret against compute, as in scaling-law studies of language models. For the three largest budgets the optimal size is roughly constant, which may mean there is a size beyond which the model is expressive enough for one prior, or may be noise.

Limitations

The network only approximates the posterior, and the approximation costs some regret as dimension grows (the d=16d=16 gap above). No single prior is best across problems, the method loses to Gaussian process baselines on Ackley in d=8d=8 and on Pest Control, and each new prior needs its own pre-training run. The scaling analysis cannot yet distinguish a capped optimal model size from a linear trend. Experiments stop at 200 observations, the model’s maximum context size.

Citation

@inproceedings{park2026thompson,
  title     = {Thompson Sampling using Prior-fitted Diffusion Transformers},
  author    = {Park, Sihwa and Zhu, Jingsen and Jain, Vinamr and Chou, Sheng-Yen and Terenin, Alexander},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  eprint    = {2611.14656},
  archivePrefix = {arXiv}
}