The vLLM project, the open-source engine many teams use to serve language models, shipped two notable pieces of infrastructure over the past two weeks: Humming, a library of fast kernels for quantized models built with Ant Group, and vllm-torchtpu, a backend that runs vLLM on Google Cloud TPUs. It also kept up its habit of day-0 support for new models, including Mistral Large 4 on October 29. This edition covers what each release does, the numbers the maintainers published, and the caveats.
Humming: one description of quantization, many fast kernels
Quantization shrinks a model’s weights, and sometimes its activations, to fewer bits so it fits in less memory and runs faster. The catch is that every format needs a matching GPU kernel, and writing them one by one does not scale. Humming, announced October 21, separates the description of a quantized computation from the choice of how to execute it. A unified representation turns different export formats into one description of weights, scales and zero points, and kernels are compiled just in time from shared CUDA components.
Per the project’s post, it supports weight-and-activation combinations from W4A16 to W4A4, formats such as INT4, MXFP4, MXINT4 and NVFP4, and NVIDIA GPUs from Turing onward, plus T-Head’s Zhenwu 810E. The authors credit engineers from Ant Group and the vLLM community.
The reported results are community microbenchmarks, with baselines that vary by chart:
- Dense W4A16 matches or beats vLLM’s Marlin kernels in most tested settings, with larger gains on some B300 cases.
- For mixture-of-experts layers, MXFP4 reaches about twice Marlin on H20 and H200 at medium and large batches.
- In one end-to-end test on DeepSeek-V4.1-Flash with eight H20 GPUs, switching from Marlin W4A16 to Humming W4A8 raised output throughput by 14% to 43% and cut mean time to first token by about 25% to 32%.
You can try it from vLLM 0.29.0, where CUDA installs include Humming as a dependency, using the --linear-backend humming and --moe-backend humming flags, or install the standalone humming-kernels package.
How it compares
Marlin is the established open kernel for 4-bit weights in vLLM; Humming targets a wider range of formats and GPU generations rather than one format. The honest reading is “promising, verify on your hardware.” The post itself notes limits: autotuning is not yet available so heuristics may miss the best configuration on some workloads, the end-to-end test covers one model on one setup, and it names no license in the text we reviewed, so check the repository before adopting. For background, see our quantization explainer.
vllm-torchtpu: vLLM on Google TPUs
Announced October 20, vllm-torchtpu is an out-of-tree platform plugin that vLLM detects automatically once installed. It builds on Google’s TorchTPU stack and comes from a partnership between Inferact and Google. The goal is to keep the same CLI, OpenAI-compatible server, scheduler and KV cache manager that users have on GPUs, so most supported models should serve on TPU without model-code changes.
Key details from the post:
- TPU v7x (Ironwood) is the recommended target and is tested in continuous integration on every pull request; v6e, v5e and v5p are experimental.
- Custom Pallas kernels cover attention variants, fused mixture-of-experts and quantized matrix multiplies in FP8, INT8, MXFP4 and NVFP4.
- All six of vLLM’s parallelism dimensions work across multi-host TPU slices, and the backend supports KV cache offloading to host memory and disaggregated prefill and decode.
- Installation is
pip install vllm-torchtpuon a Cloud TPU VM with Python 3.12; the first start compiles graphs, which can take minutes.
The post publishes no throughput or latency figures, pointing instead to nightly support matrices in the repository. Our report on the TorchTPU backend from Google and Meta gives the platform context.
Day-0 support keeps pace with releases
The vLLM blog also lists day-0 support posts for Mistral Large 4 on NVIDIA Blackwell (October 29) and MiniMax-M3.1-Flash (October 23), and a deployment write-up reporting DeepSeek-V4.1-Flash on a GB300 NVL72 rack. If you run your own inference, those posts are useful recipes. Our primer on how vLLM serves models explains the terms they use.
Who should care
Teams paying for GPU inference should test Humming on a staging model before assuming the gains transfer. Teams with TPU credits now have a path that keeps their existing vLLM tooling. Everyone else benefits indirectly as kernels and backends improve in the open.
What to watch next
- Independent benchmarks of Humming on hardware other than H20 and H200.
- An autotuner for Humming, listed as future work.
- Published performance data for
vllm-torchtpu. - Whether the TPU plugin delivers its stated goal of onboarding new open models within 24 hours of release.
- The license and governance details for Humming.
Key takeaways
- Humming is a quantized-kernel library from Ant Group and the vLLM community, available in vLLM 0.29.0.
- Maintainer benchmarks show up to roughly 2x over Marlin on some mixture-of-experts layers, with caveats.
vllm-torchtpuruns vLLM on Google Cloud TPUs, with v7x as the supported target.- No TPU performance numbers have been published yet.
- Validate vendor and community benchmarks on your own workload.
Frequently asked questions
What is Humming?
Humming is an open-source kernel library for quantized inference, built by Ant Group and the vLLM community, that generates fast GPU kernels for many quantization formats.
Can vLLM run on TPUs?
Yes, through the vllm-torchtpu plugin announced October 20, which targets Google Cloud TPU v7x and treats older generations as experimental.
Are the benchmark numbers independent?
No. They are community microbenchmarks published by the project, with baselines that vary, so test your own model and hardware.
