Important
BentoML is now part of Modular. Together, we're building a unified stack for high-performance AI inference.
🍱 BentoML: The Unified Serving Framework for AI/ML Systems
BentoML is a Python library for building online serving systems optimized for AI apps and model inference. It supports serving any model format/runtime and custom Python code, offering the key primitives for serving optimizations, task queues, batching, multi-model chains, distributed orchestration, and multi-GPU serving.
🎨 Examples: Learn by doing!
A collection of examples for BentoML, from deploying OpenAI-compatible LLM service, to building voice phone calling agents and RAG applications. Use these examples to learn how to use BentoML and build your own solutions.
🦾 OpenLLM: Self-hosting Large Language Models Made Easy
Run any open-source LLMs (Llama, Mistral, Qwen, Phi and more) or custom fine-tuned models as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference performance, and a simplified workflow for production-grade cloud deployment.
⚡ Modular MAX: Hardware-optimized inference across any accelerator
For end-to-end performance and portability across the AI execution stack, see Modular MAX.
🔥 Mojo: A faster future for AI inference
Mojo is on the path to a stable 1.0 release in 2026, delivering semantic versioning and a high-performance language for CPU and GPU programming. As Mojo matures, BentoML will leverage Mojo's compiled performance for CPU and GPU kernels to push model inference performance even further. See the Mojo Roadmap for more information.
flowchart TD
A["🚀 Your App / API"]
B["🍱 BentoML — Serving · Orchestration · Batching · Multi-model Chains"]
C["⚡ Modular MAX — Hardware-optimized Runtimes on Any Accelerator"]
D["🔥 Mojo — High-performance Kernel & Systems Programming"]
E["🖥️ Any Hardware · NVIDIA · AMD · CPU"]
A --> B --> C --> D --> E
💬 Join the Modular Discord and Modular Forum
👀 Follow us on X @Modular and LinkedIn
📖 Read our blog