Multimodal Flow

Unified Flow Modeling of Language and Vision in Embedding Spaces

A fully continuous framework for multimodal understanding, generation, and the flexible sequences in between.

Hongyuan Tao¹   Xinggang Wang^1*   Lianghui Zhu¹   Yongkang Li¹   Yunchao Wei²   Bin Feng¹
Shaoyu Chen³   Qian Zhang³   Chang Huang³   Kai Yu³

¹Huazhong University of Science and Technology    ²Beijing Jiaotong University    ³Horizon Robotics
_{^*Corresponding author: Xinggang Wang}

News

• Oct. 1, 2026: 🌐 Our project page is live — an interactive walkthrough of the chunk formulation, the shared flow objective, and the sequences it admits.
• Oct. 1, 2026: 🎉 Our paper is now available on arXiv! Check it out to learn more about Multimodal Flow.
• Oct. 1, 2026: We have released the first version of the Multimodal Flow code. Welcome to try it and build on it!

Table of Contents

• Introduction
• Getting Started
• Documentation
• License

Introduction

Multimodal Flow explores a general way to model heterogeneous information in a fully continuous space. Rather than forcing every modality into a shared discrete token vocabulary or giving each task its own model, MF represents different modalities as continuous hyperchunks and places them in one ordered, causal stream. Language, images, and other structured signals can remain distinct while sharing the same context, transformations, and generative dynamics.

🌊 One continuous space   ·   🧩 Flexible sequences   ·   ⚙️ Configurable recipes

Different modalities meet in one continuous, chunk-causal flow and return to their native spaces through their decoders.

During training, an ordered multimodal context is encoded into continuous hyperchunks and target chunks are predicted in parallel. During inference, completed chunks are generated sequentially and appended to the context. The same interface therefore supports understanding, generation, and sequences that move between them.

Parallel target prediction during training and sequential continuation during inference.

The open-source implementation turns this idea into a flexible continuous multimodal training and inference framework. A model is built from configurable modality codecs, semantic sequences, task definitions, physical layouts, objectives, and generation paths, so the same backbone can be shaped into a concrete research recipe without rewriting the core system. Text, images, and mixed sequences share one continuous interface, while their native representations, temporal structure, supervision, and decoders remain explicit. This makes the framework a natural foundation for language modeling, visual understanding, image generation, editing, video sequences, and future modalities that can be expressed as ordered continuous chunks. Training and inference follow the same sequence contract: targets are predicted in parallel during training and unfolded step by step at inference time.

Getting Started

🚀 Start locally, configure once, and run. MF is a local-first, configuration-driven framework. The shortest path is:

git clone https://github.com/hustvl/Multimodal-Flow.git
cd Multimodal-Flow
conda env create -f environment.yaml
conda activate multimodal-flow

The released 1.6B pretraining and SFT models are available on Hugging Face. Use the Inference guide to run them or a checkpoint produced by your own training run.

🧭 Follow this path for a first run:

Goal	Guide
1. Prepare datasets and model assets	Data preparation
2. Run pretraining or SFT	Training
3. Use the resulting checkpoint	Inference
Reference	Architecture and extensions

The shortest complete path is:

torchrun --standalone --nproc-per-node=4 -m mf train --config configs/quickstart-1.6b.yaml
torchrun --standalone --nproc-per-node=4 -m mf sft --config configs/sft.yaml --init-from <PRETRAIN_CHECKPOINT>
mf infer --checkpoint <CHECKPOINT> text --prompt "A short continuation"

Documentation

Guide	Contents
Data	Public datasets, model assets, local layout, and data adapters
Training	Pretraining, SFT, configs, launch, resume, and checkpoints
Inference	Text, captioning, image generation, and checkpoint loading
Architecture	Dataflow, sequence contracts, registries, and extension points

Citation

If you use Multimodal Flow in your research, please cite the project and follow the licensing terms of all external assets.

@article{tao2026multimodalflow,
  title={Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces},
  author={Tao, Hongyuan and Wang, Xinggang and Zhu, Lianghui and Li, Yongkang and Wei, Yunchao and Feng, Bin and Chen, Shaoyu and Zhang, Qian and Huang, Chang and Yu, Kai},
  journal={arXiv preprint arXiv:2609.40362},
  year={2026},
  url={https://arxiv.org/abs/2609.40362}
}

Acknowledgements

We thank the authors and contributors of ELF, minit2i, and RAE, along with other high-quality open-source projects that helped make this work possible.

License

The Multimodal Flow source code is released under the Apache License 2.0; see LICENSE. Third-party components retain the licenses and notices specified in THIRD_PARTY_NOTICES.md. Model weights, datasets, and benchmark scorers are external assets and are not redistributed here.