Running Gemma 4 Yourself: What Works on a Laptop, and When Your Team Needs More

You type ollama run gemma4, it downloads in a minute and answers on your laptop. Google released Gemma 4 as a family of five models, though, and the benchmark scores that made headlines belong to the largest one, at the precision Google trained it in. What most people run on their own machine is a different model with the same name. For one person experimenting, that is fine. It becomes a problem when a team chooses hardware, judges quality or builds a product on the wrong assumption.

So before you buy a graphics card or a small AI server for a local LLM, or put Gemma 4 in front of your colleagues or customers, find out which Gemma 4 your memory can hold, what compression does to it and how fast it will run. Two numbers answer most of this, and they work for any model.

Five Gemma 4 models, one name

Gemma 4 is Google DeepMind's family of open-weight models, released in April 2026 under the Apache 2.0 license. That means you may download it, run it on your own hardware and use it commercially, with no fee to Google - the background is in our article on open-weight models. It reads text and images, was trained on more than 140 languages, and Google describes it as built for reasoning, coding and agent tasks, with native function calling. Google names chatbots, summarization and pulling information out of documents and images as typical uses. It is also one of the most downloaded model families for local use: more than 27 million downloads in the Ollama library alone.

Gemma 4 comes in five sizes. They share a name and a training recipe, but they are built for very different hardware.

Memory for the model weights, as published by Google, including about 20% overhead for loading. The memory for your conversation comes on top - more on that below.

The default download is the small one. ollama run gemma4 pulls the E4B model, compressed to 4-bit: 6.6 GB. The larger tags are compressed to about 4-bit as well: gemma4:31b is a 19-20 GB download, against 63 GB for gemma4:31b-it-bf16. You get 8-bit or 16-bit only if you ask for it.

The 26B A4B is a mixture of experts: 25.2 billion parameters in total, of which only 3.8 billion work on each token it writes. It runs almost as fast as a small model, yet all 25.2 billion parameters have to sit in memory. Fast like a small model, hungry like a big one.

And when you read that Gemma 4 31B scores well on reasoning or coding benchmarks, that score belongs to the 31B at BF16, the 69.9 GB version.

What compression gives and takes

A model is billions of numbers, its weights. At 16-bit precision each number takes two bytes. Quantization stores the same numbers with fewer bits: 8 bits halves the memory, 4 bits quarters it. That is how a 70 GB model becomes a 17.5 GB file that fits on a graphics card.

Fewer bits mean less detail in each number. How much of that detail your work needs is a question the research answers more clearly than the marketing does.

8-bit is close to lossless. A study by Red Hat and Neural Magic with over 500,000 evaluations found FP8 "effectively lossless across all model scales", and integer 8-bit within 1-3% of the original (Kurtic et al., ACL 2025).

4-bit holds up on English benchmarks and loses more elsewhere. The same study found well-tuned 4-bit models recovering 96-99% of the original scores on standard English tests. Other studies looked where those tests do not. Cohere had human raters judge quantized models: automatic benchmarks showed an average drop of 1.7% in Japanese, while the raters, on realistic prompts, saw a drop of 16% (Marchisio et al., EMNLP 2024). On inputs of 64K tokens and more, 8-bit stayed within about 0.8% of the original, while 4-bit methods lost up to 59% on some tasks, with non-English inputs hit harder (Mekala et al., EMNLP 2025). And quantized models "often struggle with instruction-following and hallucination detection", small ones more than large ones (Lee et al., IJCAI 2025).

If you work in German, Italian or Danish, with long documents, or need a model that follows instructions to the letter, look there first.

There are also two kinds of 4-bit model. Most community files, including the default gemma4 download (Q4_K_M), are rounded after training: the finished model's weights are compressed afterwards. With quantization-aware training (QAT), the model learns during training to cope with 4-bit weights. Google publishes QAT versions of every Gemma 4 model and says they keep "similar quality to bfloat16", though it has published no numbers. In Ollama they are separate tags, for example gemma4:31b-it-qat.

The clearest example that 4-bit is not automatically worse is gpt-oss: OpenAI trained it in a 4-bit format (MXFP4) and evaluated it that way. For gpt-oss, 4-bit is the native precision. For Gemma 4, the native precision is BF16, and 4-bit is a compromise you choose.

Two numbers predict your speed

When a model writes text, it produces one token at a time, and for every token it has to read all of its active weights from memory. On a single machine with one user, the speed of that reading sets the pace. That gives a rule you can apply to any model on any hardware:

Maximum tokens per second ≈ memory bandwidth ÷ size of the weights read per token

The first number is in your hardware's specification. The second follows from the model and its precision. Here is what the rule gives for Gemma 4 31B on common local hardware:

Calculated upper limits for one user, from the manufacturers' bandwidth specifications. The weights read per token are a little below the memory figures earlier, which include loading overhead and the image encoder. Real speeds are lower, because the rule ignores the memory for the conversation, computation and software overhead.

The rule holds up in practice. On a DGX Spark, community measurements show about 3.7 tokens per second for the 31B at 16-bit and about 10.6 at 4-bit, roughly two thirds to 85% of the limit (NVIDIA developer forum).

The DGX Spark and a 128 GB Mac can hold the full 31B at native precision, which no consumer graphics card can. At 273 GB/s, though, the Spark writes it more slowly than most people read. Holding a model and running it fast are separate problems.

The mixture-of-experts model changes the math. The 26B A4B reads only 3.8 billion parameters per token, about 2 GB at 4-bit, so it runs several times faster than the 31B on the same machine (community measurements on the DGX Spark: about 24 tokens per second at 16-bit). For many local setups it is the sweet spot, as long as its quality is enough for your task.

The same rule helps you size your next machine: look up the bandwidth, divide by the size of the model you want to run, and you know the best case before you spend anything.

Where local stops

A local Gemma 4 is excellent for a lot of work: experiments, private documents that should not leave your desk, offline use, one person working with a model all day, at no cost per token. The limits show up as the work grows, usually in this order: context first, then several users, then keeping it running.

Context

Every token of your conversation - the prompt, the documents you paste in, the answer so far - occupies memory too, in the so-called KV cache. Gemma 4 keeps this small with a clever design (most layers only remember the last 1,024 tokens), but it still adds up. For the 31B, with one conversation and a 16-bit cache:

On a 24 GB graphics card, the 4-bit 31B takes 18-20 GB depending on the file, which leaves 4-6 GB. A 32K conversation fits. The 256K context window on the model card does not come close.

More than one user

Each user needs their own conversation memory, and all of them share the same memory bandwidth. A machine that gives one person 50 tokens per second does not give five people 50 tokens per second each. For a team, an application or an agent that sends many requests in parallel, this becomes the limit long before the hardware is "full".

Keeping it running

A model on a desk is a project. A model that a team relies on is a service: someone updates it when a new version ships, watches it when it stops answering, and keeps the machine available. That work does not show up in the hardware price.

When you hit the wall

When a local setup is no longer enough, there are three ways forward. Which one fits depends on your volume, your team and your data.

A different model on the hardware you have

If speed is the problem, the 26B A4B often solves it. If memory is the problem, the 12B at 8-bit may give you better quality than the 31B at 4-bit for your task - test both. This costs nothing but an afternoon.

Bigger hardware, bought or rented

The full 31B at native precision needs about 70 GB for its weights alone. On graphics cards that means a data-center card in the 80 GB class, with little room left for context - in practice often two cards, or 8-bit. You can buy such hardware or rent it from a cloud provider by the hour. Either way you run the serving software yourself (vLLM or similar), choose the precision, and pay for the hardware whether it is busy or not. If you rent, ask where the data is processed, under which country's law, and who has access. With a steady, high volume and your own team to run it, this can be the right choice.

A hosted API

Someone else runs the model; you send requests and pay per token. Here the questions from the sections above still apply, just in a different form: which Gemma 4 do you get, at which precision, and how fast? Independent measurements of hosted Gemma 4 31B APIs range from about 20 to over 500 tokens per second. One of the fastest serves a 4-bit version, and most providers do not state their precision at all (Artificial Analysis). Ask.

Infercom is built for this third way: teams, products and agents that need the full Gemma 4 31B without running the hardware themselves.

• Processed in the EU. Requests to Gemma 4 run on our own hardware in Munich, and prompts and outputs are not stored persistently. Details are on our Trust page.
• The full model, not a compressed one. We serve Gemma 4 31B at its native precision (BF16), the version Google benchmarks.
• Speed that does not depend on your machine. Our hardware is SambaNova RDUs, purpose-built AI chips rather than GPUs, designed around the memory bottleneck described above. Gemma 4 31B writes 486 tokens per second on it, with a time to first token of 1,302 ms for a 10,000-token prompt (server-side median, one request at a time, 1,000 output tokens, measured October 2026; method and all models on our performance page).
• Room for real work. Up to 128K tokens of context per request, and no hardware to size for your busiest hour, because that capacity is ours to run.

The API is OpenAI-compatible. Code written against a local Ollama or vLLM server keeps working with a new base URL, key and model name:

from openai import OpenAI

client = OpenAI(base_url="https://api.infercom.ai/v1", api_key="YOUR_INFERCOM_API_KEY")
reply = client.chat.completions.create(
    model="gemma-4-31B-it",
    messages=[{"role": "user", "content": "Summarize this contract in five points: ..."}],
)
print(reply.choices[0].message.content)

You can also keep both: a small Gemma 4 locally for experiments, and the full 31B through the API for what your colleagues and customers rely on.

We serve the 31B only, not the smaller Gemma 4 models. A Gemma 4 you have fine-tuned on your own data does not run on the Inference API; we host it on Dedicated Capacity, reserved capacity for you on the same infrastructure.

Which setup fits

A common path runs through several of these: start locally with E4B or 12B to learn what the model can do, then move to the full 31B through an API when quality, context or the number of users demands it. Same model family, same prompts.

The bottom line

For one person, running Gemma 4 yourself is often the right choice. Know what you are running, though:

• What runs on your machine is usually a smaller or compressed Gemma 4. Google's benchmark scores belong to the 31B at native precision, a 70 GB model.
• Compression costs most in languages other than English, in long documents and in precise instructions, which is the daily work of most European teams.
• Memory size and memory speed tell you how it will run. Check both before you buy hardware.
• Local stops where the team starts: long context, several users at once, someone on call when it stops answering.

You can run Gemma 4 yourself. What you have to decide is which Gemma 4 your work needs, and you do not need a 70 GB machine to find out. Take the prompts you already use locally, run them against the full Gemma 4 31B in our playground, and compare the answers. Then you know whether your team needs the full model.

Questions

Can I run Gemma 4 locally?

Yes. With 8 GB of memory you can run the E2B or E4B model at 4-bit, with 12-16 GB the 12B, and with 24-32 GB the 26B A4B or the 31B at 4-bit. The 31B at native precision (BF16) needs about 70 GB, which only machines with large unified memory, such as a DGX Spark or a Mac with 128 GB, can hold.

How much memory does Gemma 4 31B need?

About 69.9 GB at BF16, 34.9 GB at 8-bit and 17.5 GB at 4-bit, according to Google's documentation. That covers the weights only. The context comes on top: about 1.5 GB at 8K tokens, 3.5 GB at 32K and 11.6 GB at 128K for one conversation, with a 16-bit cache.

What is the difference between Q4_K_M and QAT?

Q4_K_M is a 4-bit version made after training, by rounding the finished model's weights. It is what the default gemma4 tag in Ollama downloads. QAT (quantization-aware training) means the model was trained to cope with 4-bit weights. Google publishes QAT versions of every Gemma 4 model and says they keep quality close to the original, but has published no numbers.

How many tokens per second does Gemma 4 reach on a DGX Spark?

Community measurements on the NVIDIA developer forum report about 3.7 tokens per second for the 31B at BF16, about 10.6 at 4-bit, and about 24 for the 26B A4B at BF16, for one user. The memory bandwidth of 273 GB/s sets the limit.

Is a local LLM GDPR compliant?

Running a model locally keeps the data on your own hardware, which removes the question of where a provider processes it. GDPR compliance still depends on the whole processing: the purpose, the legal basis, access control and logging. A local model is a strong starting point, not a compliance guarantee.