Red Hat, a global leader in open source software has launched llm-d, a new open source project designed to solve a major challenge in generative AI, running large AI models efficiently at scale. By combining Kubernetes and vLLM technologies, llm-d enables fast, flexible, and cost-effective AI performance across different clouds and hardware.
CoreWeave, Google Cloud, IBM Research, and NVIDIA are founding contributors to llm-d. Partners like AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI are also on board. Top UC Berkeley and the University of Chicago researchers backed this project, who developed vLLM and LMCache.
A New Era of Flexible, Scalable AI
Red Hat’s goal is clear. Let companies run any AI model, on any hardware, in any cloud without getting locked into expensive or complex systems. Just like Red Hat helped make Linux a standard for businesses, it now wants to make vLLM and llm-d the new standard for running AI at scale.
By building a strong, open community, Red Hat aims to make AI easier, faster, and more accessible for everyone.
Also Read: kubectl-ai: AI for Kubernetes CLI Management 2025
What llm-d Brings to the Table
llm-d introduces a range of new technologies to speed up and simplify AI workloads:
- vLLM Integration: A widely adopted open-source inference server that works with the newest AI models and many hardware types, including Google Cloud TPUs.
- Split Processing (Prefill and Decode): Breaks the model’s tasks into two steps that can run on different machines to improve performance.
- Smarter Memory Use (KV Cache Offloading): Saves on expensive GPU memory by using cheaper CPU or network memory, powered by LMCache.
- Efficient Resource Management with Kubernetes: Balances computing and storage needs in real time to keep things fast and smooth.
- AI-Aware Routing: Sends requests to servers that already have related data cached, which speeds up responses.
- Faster Data Sharing Between Servers: Uses high-speed tools like NVIDIA’s NIXL to move data quickly between systems.
Red Hat’s llm-d is a powerful new platform for running large AI models quickly and efficiently, helping businesses use AI at scale without high costs or slowdowns.
Conclusion
Red Hat’s launch of llm-d marks a major step forward in making generative AI practical and scalable for real-world use. By combining the power of Kubernetes, vLLM, and advanced AI infrastructure strategies, llm-d enables businesses to run large language models more efficiently, across any cloud, hardware, or environment. With strong industry backing and a focus on open collaboration, Red Hat is not only solving the technical barriers of AI inference but also laying the foundation for a flexible, affordable, and standardized AI future.