Overview
OLMoE-1B-7B showed that a small mixture-of-experts can beat dense models with similar active parameters. Like other OLMo models, everything needed to reproduce it is public.
Highlights
- 6.9B total, 1.3B active parameters
- 64 experts, 8 active per token
- Open data, code and checkpoints
- Apache 2.0 license
Typical uses
Research on mixture-of-experts and efficient small models.
Things to consider
- Short 4K context.
Specifications are taken from the model card on Hugging Face and the provider’s announcement. Benchmark results depend on evaluation settings; see how AI benchmarks work.