Parameters

What a parameter count actually means

A neural network is a very large set of numbers - its parameters, also called weights - organised into layers. Training sets these values so the model's outputs fit patterns in its data; once training finishes they are frozen, and running the model means multiplying inputs through them. When a model is described as "120B", that is 120 billion such numbers. Parameter count is the headline proxy for capacity, but it is not the whole story: how a model is trained matters as much as its size. The Chinchilla study showed that for a fixed compute budget, model size and training-data volume should scale together - many models are more capable not because they are bigger, but because they were trained on more data.

Not every parameter is necessarily active on every token. Dense models (such as Gemma 4 31B) use all of their parameters for each token. Mixture-of-experts (MoE) models route each token through only a fraction of theirs - DeepSeek's architecture carries hundreds of billions of total parameters but activates only tens of billions per token - which is why a very large MoE model can run far faster than its total size suggests. When comparing models, the number that predicts inference behaviour is the active parameter count, not the total.

Why parameter count governs speed and cost

Generating a token requires reading the active parameters out of memory. The arithmetic itself is cheap; moving the numbers is not. Reading tens of billions of parameters for every single token, one token at a time, is a memory-bandwidth problem - and it is the dominant cost of the decode phase. This is the core reason larger models generate more slowly: not because the maths is harder, but because there are more weights to stream from memory on each step. It is also why inter-token latency and output tokens per second track model size so closely.

Parameter count also sets the hardware footprint. Weights must sit in fast memory to be served efficiently - at 16-bit precision, a billion parameters is roughly two gigabytes - so a frontier-scale model can need far more memory than a single accelerator provides, forcing it to be split across many. Purpose-built inference hardware attacks this directly: SambaNova's RDU uses a three-tier memory system that keeps weights close to compute and can hold multiple large models resident at once, so switching between them takes milliseconds. Quantisation - storing parameters at lower precision, such as 8-bit - is the other common lever, shrinking the memory and bandwidth a given parameter count demands, usually with little quality loss.

Bigger isn't automatically better

For most production workloads the largest available model is rarely the right default. A mid-sized current model, served fast, often beats a larger one that responds slowly - and costs less per token to run. Because parameter count drives memory, bandwidth, and therefore both speed and price, choosing a model is a capacity-versus-cost decision, not a "more is better" one. The practical question is which model is capable enough for the task, then which provider runs it fastest and keeps your data in the right jurisdiction. Open-weight models (see open weights) make that choice yours to make, rather than the model vendor's.

This is also where hosting matters. A single SambaRack packs 16 RDUs, runs frontier-scale models up to the 671-billion-parameter class, and keeps several of them resident in memory at once - so it switches between models in milliseconds rather than reloading tens of gigabytes of weights per request, letting each task run on the right-sized model without a cold-start penalty. On our shared EU platform this sits behind one OpenAI-compatible API; when you need guaranteed capacity or full data isolation, the same SambaNova architecture backs our dedicated capacity and on-premises deployments.

Related terms