Recently, I was reading about Moonshot AI’s new Kimi K3 model when one detail caught my attention: it uses a mixture-of-experts architecture.
Kimi K3 reportedly contains 2.8 trillion parameters, but activates only around 50 billion of them for each token it processes. It does this by selecting 16 experts from a total of 896.
Mixture-of-experts models are not new. Moonshot’s earlier Kimi K2 model also used the approach, with 1 trillion total parameters and 32 billion active parameters. But the scale of Kimi K3 made me more interested in what these numbers actually mean.
How can an AI model contain trillions of parameters without using all of them at once?
Here is the simple explanation I arrived at.
What is a mixture-of-experts model?
A mixture-of-experts model, usually shortened to MoE, divides part of its neural network into smaller components called experts.
When the model processes a token, which might be a word or part of a word, it does not send that token through every expert. Instead, it selects a small number of experts that appear most useful for processing it.
The selected experts do the work, while the others remain inactive for that particular token.
This allows the model to have a huge total capacity without using the full model for every calculation.
How does the model choose an expert?
A component called a router makes the decision.
For each token, the router examines the information the model has built up about that token and its surrounding context. It then gives the available experts a score and selects the highest-scoring ones.
The process looks roughly like this:
- The model receives a token.
- The router examines its context.
- It scores the available experts.
- The most relevant experts process the token.
- Their results are combined.
In Kimi K3’s case, this means selecting 16 experts while skipping the other 880 for that token.
The selection is not permanent. The next token may be sent to a completely different combination of experts.
Diagram showing how a mixture-of-experts AI model routes each token to selected experts (AI generated illustration)Are the experts specialists?
The word “expert” makes it sound as though each component has a clearly assigned profession.
You might imagine one expert for programming, another for mathematics and another for French. In reality, the divisions are usually less obvious.
Experts develop their own strengths during training. One may become useful for certain sentence structures, technical language or code-like patterns. Another may respond well to different kinds of information.
These specialisations can overlap, and researchers may not always be able to describe precisely what an individual expert has learned.
Why use this design?
The main benefit is efficiency.
In a traditional dense model, nearly all the model’s parameters are involved whenever it processes a token. Making that model larger therefore increases the computing cost significantly.
A sparse mixture-of-experts model separates two things:
- the model’s total capacity
- the amount of the model used for each token
Kimi K3 may contain 2.8 trillion parameters, but only about 50 billion are active for a particular token. That is roughly 1.8% of the complete model.
The model can therefore store patterns across trillions of parameters without performing calculations with every parameter at once.
Research into MoE language models has found that this kind of conditional computation can make them more compute-efficient than similarly capable dense models, although the advantage varies between tasks and model sizes.
Is it just a 50-billion-parameter model?
No.
The active 50 billion parameters are not necessarily the same for every token. Different tokens can be routed through different groups of experts.
This means the model has access to a much larger pool of parameters, even though it activates only a small portion at a time.
Some parameters outside the experts may also be shared and active throughout the model. The quoted active-parameter number is therefore not always as simple as multiplying the number of selected experts by the size of each expert.
A simple analogy
Imagine a large company with 896 specialist teams.
Whenever a task arrives, a coordinator chooses the 16 teams most suited to it. Those teams contribute their work, and the results are combined.
The other teams do not need to become involved.
For the next task, the coordinator may select a different set of teams.
The company can possess a huge range of knowledge and skills without asking every employee to work on every problem.
That is essentially what a mixture-of-experts model does.
What is the catch?
Mixture-of-experts models still have important costs.
The complete model must be stored somewhere, even when most experts are inactive. The system also has to route tokens quickly, transfer information between hardware and distribute work so that a few popular experts do not become overloaded.
An MoE model with 50 billion active parameters is therefore not necessarily as cheap or simple to operate as a normal dense model containing 50 billion parameters.
It is better understood as a way of giving an AI model much greater capacity without increasing computation at the same rate.