Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes takes a systematic look at questions most teams answer by trial and error.

The framing — "physics of" — signals the useful ambition: treat pretraining behaviour as phenomena with explanations, not as a bag of tricks.

The folklore problem

Ask three teams when to introduce the vision encoder and you get three confident answers with three different justifications. The same is true for modality mixing ratios, for whether to freeze the language model early, and for how much interleaved data is enough.

Most of these beliefs originated in a single ablation on a single scale, propagated through papers and blog posts, and hardened into practice without anyone re-checking whether they survive at a different scale or with a different data mix.

That is not a criticism of individual teams — the experiments are expensive and nobody can run them all. But it means the field's practical knowledge is unusually fragile.

The three questions worth having answers to

Knowledge flow. When the model learns something from images, does it become available to the text pathway, or does it stay siloed? This determines whether multimodal training helps unimodal performance or merely fails to hurt it.

Modality synergy. Do modalities help each other, or compete for capacity? The answer is likely to be scale-dependent, which would explain much of the disagreement in the literature — small models compete for capacity, larger ones have room to share.

Early unification. Does merging modalities early in training produce better representations than joining later? This is the single most consequential recipe decision, because it is the hardest to change once committed.

Why this style of paper is undersupplied

Papers that explain existing practice are harder to publish than papers that beat a benchmark. The result does not fit a leaderboard, the contribution is understanding rather than performance, and reviewers routinely ask "but is it state of the art."

Meanwhile the practical value is high. A team about to spend a large compute budget on a multimodal run would rather have a principled answer on early unification than another half-point on a captioning benchmark.

How to read the results

The thing to check in any paper of this kind is scale coverage. Mechanistic claims from small-scale experiments frequently invert at larger scale, and the multimodal literature has a poor record here specifically.

A finding that holds across two orders of magnitude is worth changing your recipe over. A finding at a single size is a hypothesis.

The paper is on Hugging Face for anyone planning a multimodal run — the recipe section in particular is the sort of thing worth reading before committing compute rather than after.