How many are awake?

Findings

• The two sparse models were the two fastest single-model runs. gpt-oss:20b finished in 108 s, 7.6–11.4× faster than the three dense models run alongside it (825–1,234 s). The sparse gemma4:26b-a4b took 787 s against 1,812 s for its dense sibling.
• Inside one model family, sparsity bought speed, not thoroughness. The sparse Gemma finished in 43% of the dense Gemma’s time and ran a second search round, but that round added no claims. On the eight identical first-round pages it extracted 27 claims to the dense model’s 33, and 16 distinct figures to 27. Every one of its 16 figures was also found by the dense model.
• The gap is in the arithmetic per token. On a cache-busted ~7k-token prompt, the sparse models prefilled at 1,155–1,369 tokens/s and decoded at 61–65 tokens/s. The dense models managed 62–345 and 8.6–19.7.
• The reader role dominates the workload and caps the content. The evaluator made 81–89% of all model calls in every run, and the writer only sees the claims the evaluator extracted.
• A mixed team did not beat its members on content. gpt-oss planning and reading with qwen3.8 writing took 244 s, against 108 s for gpt-oss alone and 1,023 s for qwen3.8 alone. It produced a longer report (850 vs 721 words), but its coverage lead over gpt-oss alone was within the run-to-run variation of the same reader.
• Errors enter at the reader, and the citation gate cannot see them. In one run gpt-oss recorded 59% for a figure its source gives as 60%. The number reached the final report with a valid citation, and every automated check passed.
• Three quirks in Ollama’s own metrics can reverse a conclusion. They are JSON-mode thinking left out of token counts, queue time billed to calls, and prompt-cache-inflated prefill rates.

The system under test

The agent answers a question by iterating plan → search → fetch → evaluate → synthesize. Three roles call a model, and each role can use a different one:

• Planner. Breaks the question into sub-questions with search queries, and re-plans when a sub-question is under-covered.
• Evaluator (the reader). Reads one fetched page against one sub-question. It returns relevance, coverage, whether the page answers the sub-question, and a list of claims. Claims below relevance 0.35 are dropped.
• Synthesizer (the writer). Receives only the kept claims, grouped by source key, and writes a report.

Pages come from a self-hosted SearXNG instance, with robots.txt respected. A sub-question closes once two pages independently answer it. The run stops as covered when every sub-question closes, or budget_exhausted at the round limit. A hard citation gate rejects any report sentence that does not cite a source the run actually retrieved.

Setup and controls

Three comparisons, all on the same question with the same effort settings:

• E1: one model in all three roles. gpt-oss:20b, muse-glimmer:30b-mlx, qwen3.8 and gemma4:31b-mlx.
• E2: the same family, dense against sparse. gemma4:31b against gemma4:26b-a4b.
• E3: a mixed team (gpt-oss plans and reads, qwen3.8 writes) against each member alone.

Models

The two sparse models sit on disk (and in memory) close to their dense neighbours. Sparsity cuts the compute per token, not the memory footprint.

Effort settings and controls

• Shared seed plan. One round-1 plan per comparison, reused by every variant.
• Shared round-1 corpus. Round-1 pages are searched and fetched once and evaluated by every variant. Later rounds search live, so variants diverge after round 1.
• Sequential execution. Variants run one after another on one GPU.
• Request gating. In-flight model calls are capped at what the Ollama server actually runs in parallel, which is one. E1 predates this.

What wall clock means here: a variant’s time runs from its first call to its report. It includes every round-1 evaluation and all of round 2’s searching and fetching. It excludes the shared plan and round-1 fetch, which happen once before any variant starts.

Results

4.1 One model in every role (E1, E2)

Six runs with one model in all three roles. E1 and E2 share settings but not their round-1 pages, so compare within an experiment first.

Figure 2. Wall clock per run with one model in all three roles. The same gemma4:31b weights appear twice as different builds in different experiments, so that is not a controlled build comparison.

A stricter bar in E1—figures found by at least three of the four models—gives the same order: qwen3.8 found all 34, gpt-oss 32, gemma4:31b-mlx 30, muse-glimmer 29. gpt-oss’s 41 claims are more granular rather than more complete.

Every one of the six reports passed the citation gate, and every number in every report appears in that run’s own claims. That is fidelity to the evidence, not correctness.

4.2 Same family, dense vs sparse (E2)

gemma4:31b and gemma4:26b-a4b share a vendor, a generation, a quantization and, within E2, the same round-1 pages. The sparse model took 787 s against 1,812 s. It closed neither sub-question in round 1, so it ran a second round and evaluated 13 pages to the dense model’s 8. That second round added no claims. On the eight shared round-1 page–sub-question pairs, the dense model found claims on 6 pages (33 claims) and the sparse model on 5 (27). The dense model’s 27 distinct figures include all 16 of the sparse model’s, plus 11 more.

One reliability event is worth noting. The sparse Gemma’s first synthesis call spent its entire 16,384-token budget thinking (56,072 characters) and returned no report. The automatic retry succeeded.

4.3 Prefill and decode

A separate micro-benchmark isolates per-token speed. Each model got one ~6–7k-token prompt with a random prefix to defeat prompt caching, one warm-up call, and then the mean of two calls.

4.4 Where the tokens go

Planner and evaluator calls ask for JSON with thinking enabled. For those calls Ollama leaves thinking out of eval_count, which is why the raw completion column is small next to the thinking column.

The sparse Gemma generated six times the thinking of its dense sibling (128,662 vs 21,309 characters) and still finished in under half the time. Its raw completion count is inflated by the failed 16,384-token synthesis call, which ran outside JSON mode.

4.5 A mixed team (E3)

Because the writer sees only claims, the reasoning was to put a fast model where calls repeat—the reader—and a slower model where the work happens once—the writer. E3 tests that against both members run alone, on the same plan and the same six round-1 documents.

The team ran in 244 s, 4.2× faster than qwen3.8 alone and 2.3× slower than gpt-oss alone. All three runs finished with covered, cited 6 sources and passed the citation gate.

The coverage difference between the team and gpt-oss alone is noise. Both used gpt-oss as the reader. Across the two gpt-oss runs on identical inputs, 26 figures matched, 3 appeared only in the team’s run and 1 only in the solo run. That run-to-run variation is as large as the 30-vs-28 gap, and the writer cannot add figures anyway. qwen3.8 was the more thorough reader.

What the slower writer did produce is a longer report: 850 words against 721 from the same claims base. Prose quality was not measured.