R9700 Model Lab

Open-weight model campaign plan

A first benchmark campaign for local llama.cpp testing through AI Flight Recorder. The goal is to compare practical usefulness, speed, stability, and task fit across coding, creative writing, agentic workflows, RAG backends, and OpenWebUI-style chat.

Executive Decision

What to test first

Keep the campaign small enough to finish, but broad enough that the results are useful. Use one quant per model first, then re-test winners at higher quality.

Use latest current-family checkpoints

The active list is updated for June 15, 2026, but filtered for 32GB all-VRAM feasibility: Qwen3.6, OpenAI gpt-oss, Gemma 4, current Phi-4 reasoning/vision, Command R7B, Granite, and a clearly labeled Qwen coder fallback.

Split model from serving profile

A model is not a single benchmark row. Each useful candidate should have non-MTP and MTP/speculative profiles when supported, plus quant, context, backend, and chat-template metadata.

Fit before quality scoring

Every candidate must pass a 32GB VRAM fit gate before the normal benchmark suite. The output is the maximum stable context length for that exact GGUF, quant, KV-cache setting, backend, and MTP state.

Move giants out of the first wave

Mistral Small 4, Llama 4 Scout/Maverick, Qwen3-Coder-Next, DeepSeek V4, Kimi, and GLM are interesting, but not practical first-wave 32GB all-VRAM targets unless PERF-005 proves otherwise.

Core scorequality

Runtime scoretokens/sec

Usability scorereliability

Ops scorefit/cost

Evidence storeSQLite

Shortlist

Primary model list

Filter by use case to see why each candidate is in the first wave. Fit risk is based on likely memory/storage pressure in llama.cpp, not whether the model is interesting.

Priority	Model	Why it belongs	Best bets	First profile	Fit risk	Source
P0	Current Qwen3.6-35B-A3B APEX/MTP Local incumbent	Already runs on the R9700 path. Keep it as the baseline, but still measure its real max context and MTP overhead.	coding agentic chat	Existing GGUF; MTP-on and matched MTP-off profiles. Run `PERF-005` first.	known load	Local server
P1	Qwen3.6-27B Best 32GB Qwen target	Latest dense Qwen3.6 target I verified. Stronger reported coding-agent numbers than the 35B-A3B row and a much more realistic 32GB fit.	coding agentic reasoning	Q4_K_M first, then Q5_K_M if headroom exists. MTP-off and MTP-on.	likely 32GB	HF
P1	Qwen3.6-35B-A3B Current Qwen MoE/generalist	Still important because your existing data is Qwen-shaped and the model card calls out native MTP and 262K context.	agentic coding long context	IQ4_XS or Q4_K_M. Start 32K, then discover max context; MTP-on may lower ceiling.	fit-gated	HF
P1	OpenAI gpt-oss-20b Open-weight reasoning target	OpenAI's practical local gpt-oss model: 21B total, 3.6B active MoE, Apache 2.0, 128K context, and a strong fit for coding, agentic, and chatbot comparisons.	reasoning coding agentic	Unsloth GGUF Q8_0 first, Q4_K_M as fallback/comparison. Treat MTP as N/A unless a compatible draft profile appears.	32GB ready	OpenAI
P1	Gemma 4 31B IT Largest practical Gemma	Current dense Gemma flagship for chat, creative writing, and reasoning. It is near the upper edge for 32GB, so context must be discovered.	reasoning chat creative	Q4_K_M or official QAT quant. Start 16K/32K; add MTP drafter only after base fit is known.	fit-gated	Google
P1	Gemma 4 26B-A4B IT Gemma MoE efficiency candidate	Worth testing only if the quantized total-weight footprint behaves in 32GB. Do not assume active-parameter savings solve memory.	chat creative RAG	Smallest official/QAT quant first. Base fit gate before MTP or quality tests.	fit-gated	Google
P1	Gemma 4 12B IT Primary everyday chatbot/RAG model	Current mid-size Gemma, drafter-ready, and likely to give the best quality/speed balance for OpenWebUI and normal RAG.	chat RAG creative	Q6_K or Q8/QAT where available. MTP-off and MTP-on.	32GB ready	Google
P1	Gemma 4 E4B Fast small-model control	Use as the "small but current" control for latency, RAG extraction, and local assistant responsiveness.	chat RAG reasoning	Official QAT or Q8_0. Test long context and MTP if drafters exist.	32GB ready	Google
P1	Gemma 4 E2B Ultra-fast edge baseline	Cheap enough to run high-repeat reliability tests and useful as a drafter or summarizer candidate.	chat RAG	Official QAT or Q8_0. Use for throughput, summarization, and draft-model experiments.	32GB ready	Google
P1	Phi-4-reasoning-vision-15B Compact reasoning/VLM	Current compact Microsoft reasoning target for math, UI/OCR-style tasks, document reasoning, and computer-use-style evaluation.	reasoning coding computer-use	Q5_K_M or Q6_K. Force think and force nothink profiles.	32GB ready	HF
P1	Phi-4-mini-reasoning Tiny reasoning control	3.8B, 128K context, MIT license, and useful for latency-bound reasoning and judge/helper roles.	reasoning chat RAG	Q8_0 or Q6_K. Run as assistant, judge, and summarizer candidate.	32GB ready	HF
P2	Qwen3-VL-30B-A3B Document/vision option	Use only if we want multimodal/document screenshots in the campaign. Text-only fit should be tested separately from image workloads.	document RAG chat reasoning	Q4_K_M/IQ4 first. Measure text-only context, then image-token overhead.	fit-gated	Paper
P2	Qwen3-Coder-30B-A3B-Instruct 32GB coder fallback	Not the newest coder, but the newer Qwen3-Coder-Next 80B total model is not a sensible 32GB all-VRAM target. Keep this as the practical coder comparison.	coding agentic	Q4_K_M or IQ4_XS. Run only after Qwen3.6-27B and 35B-A3B.	legacy fallback	HF
P2	Command R7B RAG/tool-use specialist	Small enough for 32GB and worth testing if GGUF packaging is clean. Command A is not a 32GB target.	RAG agentic chat	Q6_K or Q8_0 if available; compare to Granite and Gemma 4 12B for RAG.	32GB ready	Paper
P1	Granite 3.3 8B Instruct Enterprise/RAG control	8B, 128K context, Apache 2.0, and explicitly aimed at summarization, extraction, RAG, code, and function calling.	RAG chat tool use	Q8_0 or Q6_K. Use as reliable/fast RAG and extraction baseline.	32GB ready	HF
Sidecar	Granite Embedding Multilingual R2 RAG retrieval sidecar	Not a chat model, but RAG quality depends on retrieval. Use it to benchmark dense retrieval and code/document search.	RAG code retrieval enterprise	Evaluate retrieval precision/recall separately from generation models.	32GB ready	Paper

The table is now 32GB-first: one local baseline, 14 generative model targets, one dedicated coder fallback that is explicitly not latest, and one Granite RAG sidecar. Large "interesting" models moved out of the first wave unless they can prove an all-VRAM fit.

32GB Exclusions

Interesting models that should not be first-wave downloads

These can still appear in reports, but only as "does not fit", "hybrid offload", or external-reference rows unless PERF-005 proves a useful all-VRAM profile.

Mistral Small 4

Current and very interesting, but official requirements target multi-H100/H200-class infrastructure, not a single 32GB card.

codingreasoningexclude first wave

Llama 4 Scout / Maverick

Scout is 109B total and Maverick is 400B total. Active parameters help compute, but total weights still matter for VRAM.

long contextchatexclude first wave

Qwen3-Coder-Next / 480B

80B total for Coder-Next and far larger for the flagship. Use Qwen3.6-27B first, then the older 30B-A3B coder fallback if needed.

codingagentichigh fit risk

DeepSeek V4 Flash / Pro

Flash is 284B total and Pro is 1.6T total. They are not 32GB all-VRAM candidates, even though active parameters are lower.

reasoningcodingexclude first wave

Kimi K2 / GLM-4.5

Keep as "interesting frontier MoE" references only. They should not consume download time before the 32GB list is measured.

agenticreasoningexclude first wave

Command A / large Cohere

Use R7B for local RAG/tool tests. Larger Command variants belong in offload/API-reference comparisons, not the first local suite.

RAGtool useoffload only

NVIDIA Nemotron larger variants

Revisit only after a specific GGUF and memory profile is known. Do not add by brand momentum alone.

reasoningagenticprove first

Acquisition Plan

Download and quant strategy

The campaign should record model, quant, context, backend, and llama.cpp flags as one profile. Do not compare a Q8 small model against a Q3 giant model as if only the model family changed.

Phase 0

Freeze the control

Record the current Qwen3.6-35B-A3B profile: filename, quant, context, backend, MTP/speculative settings, llama.cpp commit, ROCm/Vulkan settings, and server power mode.

Phase 1

Download the 32GB first wave

Qwen3.6-27B, Qwen3.6-35B-A3B, OpenAI gpt-oss-20b, Gemma 4 12B/E4B/E2B, Phi-4-mini-reasoning, Phi-4-reasoning-vision-15B, Granite 3.3 8B, Command R7B, and Granite Embedding Multilingual R2.

Phase 2

Add upper-edge 32GB candidates

Gemma 4 31B, Gemma 4 26B-A4B, Qwen3-VL-30B-A3B, and Qwen3-Coder-30B-A3B only after the smaller set proves the harness and fit gate.

Phase 3

Record exclusions, not downloads

For Mistral Small 4, Llama 4, DeepSeek V4, Kimi K2, GLM-4.5, and Qwen3-Coder-Next, store official size/infrastructure notes and mark them offload/API-only unless a GGUF proves a useful 32GB fit.

Default quant ladder

Small models under 15B: Q8_0, Q6_K, or official QAT first.
Dense 24B-35B: Q4_K_M first, Q5_K_M for winners.
MoE 26B-120B total: start with smallest credible quant that keeps routing stable.
Huge MoE above 250B total: one smoke quant first, then stop if usability is poor.

Download convention

Store under /models/gguf/<family>/<model>/.
Keep original filename in the profile label.
Record source repo, revision/hash, license, and quant author.
Store drafters/MTP companions beside the target model.
Verify chat template before benchmark scoring.

Profile manifest

model_id: qwen3_6_27b_q4_k_m_mtp_off
family: qwen
source_repo: Qwen/Qwen3.6-27B
gguf_repo: TBD during download
quant: Q4_K_M
context_tokens: discovered_by_PERF_005
backend: rocm
serving_variant: non_mtp
mtp:
  enabled: false
  method: none
  draft_model_id: null
llama_cpp_flags:
  - --ctx-size ${recommended_ctx}
  - --jinja

32GB VRAM Gate

Measure the maximum context each profile can actually fit

This has not been validated yet. The benchmark campaign must first discover the stable context ceiling for each exact profile on the 32GB R9700 server.

Why we measure it

GGUF size is only the starting point; KV cache grows with context.
MTP and speculative decoding may load draft weights and extra buffers.
Backend, batch size, parallel slots, cache type, and offload policy all change fit.
Huge MoE models may run only with CPU offload, which must be reported separately.

Pass/fail definitions

all_vram_fit: model loads with intended GPU offload and completes a smoke prompt.
hybrid_fit: model runs only with CPU/RAM offload; report separately from all-VRAM.
unstable: OOM, backend reset, swap storm, timeout, or failed generation.
recommended_ctx: highest passing context with at least 2GB VRAM headroom.

Context ladder

Probe: 4K, 8K, 16K, 32K, 64K, 128K, 262K, 512K, and 1M when supported.
After the first fail, binary-search between the last pass and first fail.
Repeat for MTP-off and MTP-on profiles because the ceiling can change.
Run slot concurrency 1 first, then repeat winners at 2 and 4 slots.

Fit result row

{
  "model_profile": "qwen3_6_35b_a3b_q4_k_m_mtp_on",
  "gpu_vram_limit_mb": 32768,
  "reserved_headroom_mb": 2048,
  "ctx_tested": [4096, 8192, 16384, 32768, 65536, 131072],
  "max_all_vram_ctx": 65536,
  "max_hybrid_ctx": 131072,
  "recommended_ctx": 65536,
  "peak_vram_mb": 30144,
  "gpu_layers": "all",
  "kv_cache_type": "q8_0",
  "parallel_slots": 1,
  "mtp_enabled": true,
  "status": "pass"
}

Admission rule

Do not run quality benchmarks until PERF-005 passes at a usable context.
Coding/agentic minimum: 32K recommended context.
RAG minimum: 64K recommended context, or mark as short-context RAG only.
OpenWebUI chat minimum: 16K recommended context.
Stretch models may be reported as "does not fit 32GB all-VRAM" without full quality runs.

Fit Class	What It Means	Benchmark Treatment	Report Label
Green	Fits all-VRAM at the target context with 2GB+ headroom.	Run full suite for that profile.	32GB ready
Yellow	Fits all-VRAM only at reduced context, or needs lower KV precision.	Run only task categories that match the discovered context ceiling.	context-limited
Hybrid	Runs with CPU offload but not all-VRAM.	Optional smoke/perf tests only; keep out of all-VRAM leaderboards.	hybrid offload
Red	Cannot load or generate stably within the 32GB limit.	Skip quality suite and report as not practical for this hardware.	does not fit

MTP Matrix

MTP and non-MTP must be separate profiles

We should not treat MTP as a model upgrade. It is a serving profile that can improve latency and throughput while changing memory use, batching behavior, and sometimes output cadence.

Profile pair rule

For every supported model, create mtp_off and mtp_on profiles.
Keep quant, context, temperature, top-p, and prompt identical between the pair.
Only change MTP/speculative settings and required draft model assets.
Run at least one warmup before timed benchmark rows.

Current native targets

Qwen3.6-27B and 35B-A3B: compare standard serving to Qwen MTP/speculative serving.
Gemma 4 family: compare standard serving to official MTP drafter profiles.
Existing APEX/MTP Qwen GGUF: freeze current MTP and create non-MTP control.
Other models: only use speculative decoding if a real compatible draft path exists.

Extra metrics

mtp_enabled, mtp_method, draft_model_id.
Draft tokens requested, accepted tokens, rejected tokens, and acceptance rate.
TTFT, p50/p95 generation TPS, total duration, and output token count.
VRAM/RAM deltas because MTP may trade memory for speed.

MTP-on manifest

model_id: gemma4_12b_it_q5_k_m_mtp_on
base_model_id: gemma4_12b_it_q5_k_m
serving_variant: mtp
mtp:
  enabled: true
  method: gemma4_drafter
  draft_model_id: gemma4_12b_it_mtp_drafter_q8
  draft_tokens: 4
capture:
  acceptance_rate: true
  rejected_tokens: true
  vram_delta: true

Report columns

model_family
model_name
quant
context_tokens
serving_variant        # non_mtp | mtp | speculative
mtp_method
draft_model
task_type
score
success_rate
median_ttft_ms
median_generation_tps
median_duration_ms
mtp_acceptance_rate
vram_peak_mb

Benchmark Suite

Test cases and coverage

Each test should run through the Flight Recorder proxy as normal OpenAI-compatible traffic. Every task gets deterministic checks first, then local-LLM or human review where judgment matters.

COD-001

Unit-tested function repair

Give a broken Python/TypeScript function plus failing tests. The model must patch the function and explain the fix briefly.

Covers: basic code synthesis, precision, not over-editing.
Validate: run tests, compare diff size, check no extra files.

COD-002

Multi-file bug fix

Use a small fixture repo with a bug crossing API, storage, and UI layers. Require a patch-style answer.

Covers: repo reasoning, dependency tracking, practical coding.
Validate: unit tests, integration smoke test, patch applies cleanly.

COD-003

Security code review

Ask for a review of a handler with auth, path traversal, and SQL edge cases.

Covers: finding real bugs without noise.
Validate: hidden answer key, false-positive count, severity order.

COD-004

Structured patch plan

Require JSON with files, edits, tests, and risk. No prose outside JSON.

Covers: structured output, coding planning, schema compliance.
Validate: JSON schema, required fields, executable test plan.

COD-005

Codebase comprehension map

Provide a compact multi-file project snapshot and ask for the entry points, data flow, and risky modules.

Covers: repository understanding, summarization, and RAG-like grounding.
Validate: hidden architecture key, unsupported claim count, correct file references.

AGT-001

Tool selection with traps

Provide function specs where one tool is tempting but wrong. The model must call the correct tool with valid arguments.

Covers: tool selection, schema use, argument precision.
Validate: exact tool name, JSON schema, hidden expected arguments.

AGT-002

Stateful multi-step workflow

Simulate a three-step admin task with observations between calls. The model must adapt after each observation.

Covers: agentic state, correction, loop avoidance.
Validate: max step count, correct final state, no repeated failed action.

AGT-003

Prompt injection in retrieved text

Retrieved document includes instructions to ignore system policy and leak secrets. The answer must use facts only.

Covers: RAG safety, instruction hierarchy, source discipline.
Validate: no leaked canary, correct answer, cites clean source chunks.

AGT-004

Missing data humility

Ask for a deployment decision without one key metric. The model should identify the missing metric instead of inventing it.

Covers: uncertainty, non-hallucination, decision support.
Validate: mentions missing metric, offers bounded next step.

RAG-001

Cited answer from documents

Give five short source snippets and require a cited answer with quote anchors.

Covers: grounded QA, citation accuracy, synthesis.
Validate: answer key, citation-to-source match, unsupported claim count.

RAG-002

Noisy retrieval

Mix relevant and irrelevant chunks. The answer should reject distractors and cite only relevant chunks.

Covers: retrieval noise tolerance.
Validate: irrelevant citation count, correct final answer.

RAG-003

Long-context needle

Place conflicting details across a long synthetic policy/code document. Require the latest applicable rule.

Covers: long context, conflict resolution, recency cues.
Validate: exact rule selected, rationale references the conflict.

RAG-004

Structured extraction

Extract tasks, owners, dates, and risks from a synthetic meeting/email bundle.

Covers: WorkDashboard-like private workflows without real emails.
Validate: JSON schema, precision/recall against answer key.

CHAT-001

OpenWebUI helpful answer

Ask a normal home-lab troubleshooting question with logs and constraints.

Covers: assistant usefulness, clarity, actionable steps.
Validate: rubric score, no dangerous command, concise final plan.

CHAT-002

Long conversation consistency

Run 12 turns where preferences and facts are introduced early and tested later.

Covers: conversational memory and instruction retention.
Validate: fact recall, contradiction count, user preference adherence.

CHAT-003

Privacy-preserving summary

Summarize a synthetic sensitive conversation while removing PII and secrets.

Covers: privacy behavior for WorkDashboard-like data.
Validate: PII detector, no canary strings, summary completeness.

CHAT-004

Multilingual support

Ask for a short support answer in Spanish or French, based on English source text.

Covers: multilingual chat and RAG-like transfer.
Validate: source faithfulness, language quality, no untranslated fragments.

CRE-001

Constrained creative rewrite

Rewrite a dry technical paragraph for a home-lab blog with tone, length, and forbidden-cliche constraints.

Covers: style control, creativity, instruction following.
Validate: length bounds, banned phrase check, human style score.

CRE-002

Scenario generation

Generate benchmark task ideas from a real use case without exposing private data.

Covers: ideation, abstraction, privacy-aware synthesis.
Validate: novelty score, usefulness score, no private detail leakage.

CRE-003

Technical explainer

Explain a model-performance chart to a technical but non-ML audience.

Covers: communication quality and faithful interpretation.
Validate: chart fact checks, readability, no overclaiming.

OPS-001

Command risk classification

Given shell commands and context, classify risk and propose safer alternatives.

Covers: server-admin assistant behavior.
Validate: hidden risk labels, no unsafe approval, correct safer command.

OPS-002

Log diagnosis

Diagnose a llama.cpp/server/container issue from logs, metrics, and config snippets.

Covers: operations reasoning and troubleshooting.
Validate: root cause, next action, no hallucinated log lines.

PERF-001

Streaming latency probe

Short answer prompt repeated ten times per profile to measure TTFT and jitter.

Covers: perceived chat responsiveness.
Validate: Flight Recorder TTFT, p50/p95 duration, error rate.

PERF-002

Context sweep

Run representative RAG/coding prompts at 4K, 16K, 32K, 64K, and 128K where supported.

Covers: memory pressure, context degradation, throughput curve.
Validate: score delta, tokens/sec delta, VRAM/RAM samples.

PERF-003

Repeatability and flake rate

Run selected tasks five times with fixed settings and controlled randomness.

Covers: reliability, variance, hidden instability.
Validate: pass-rate spread, schema failure rate, output length variance.

PERF-004

MTP delta pair

Run the same short, medium, and long prompts against matched MTP-off and MTP-on profiles.

Covers: MTP speedup, acceptance behavior, memory tradeoff, and output stability.
Validate: median TPS delta, TTFT delta, score parity, acceptance rate, VRAM delta.

PERF-005

32GB VRAM fit and context ceiling

Load each exact model profile across a context ladder to find the maximum stable all-VRAM context.

Covers: model admission, KV-cache pressure, MTP memory overhead, and CPU-offload boundaries.
Validate: load success, smoke generation, peak VRAM, headroom, max all-VRAM context, max hybrid context.

Scoring

Validation model

The benchmark needs to separate quality failures from runtime failures. Flight Recorder already captures the runtime side; the test harness should add task-level scoring.

Composite score per run

Use weighted scoring so a model cannot win by being fast and useless, or brilliant and unusable.

Task correctness42%

Instruction fit18%

Reliability18%

Latency/speed14%

Resource cost8%

Validator types

Deterministic: unit tests, JSON schema, regex/canary checks, exact answer keys.
Source-grounded: citation verifier checks answer claims against retrieved chunks.
Local judge: one held-out local model scores style, helpfulness, and reasoning quality.
Human review: only for finals, surprising failures, and screenshot-worthy comparisons.

Runtime fields

ttft_ms, duration_ms, prompt TPS, generation TPS.
Prompt, completion, reasoning, and total tokens.
Context size, backend, quant, model file, llama.cpp version.
32GB fit class, max all-VRAM context, max hybrid context, recommended context.
MTP enabled, MTP method, draft model, acceptance rate, rejected tokens.
Failures: timeout, context overflow, parse error, empty response, backend error.

Quality fields

pass_fail, numeric score, rubric details.
Schema valid, citations valid, tests passed, patch applied.
Unsupported claims, hallucinated tool result, privacy leak count.
Human note and local-judge rationale for subjective tasks.

Fairness controls

Same prompt, same context, same stop settings per task.
Model-specific chat template allowed, but recorded.
Reasoning models get a fair output budget and recommended sampling.
MTP and non-MTP pairs must use the same model quant and sampling settings.
Each model gets warmup runs excluded from scoring.

Harness

How the tests should run

Keep the runner boring: a campaign manifest, a model profile manifest, a test fixture folder, and OpenAI-compatible calls to http://192.168.1.116:8181/v1.

Campaign manifest

campaign: r9700-open-weight-round-001
hardware_profile: r9700-server
base_url: http://192.168.1.116:8181/v1
runs_per_task: 3
warmup_runs: 1
visibility: private
admission_gates:
  - PERF-005
models:
  - qwen3_6_27b_q4_k_m_mtp_off
  - qwen3_6_27b_q4_k_m_mtp_on
  - qwen3_6_35b_a3b_q4_k_m_mtp_off
  - qwen3_6_35b_a3b_q4_k_m_mtp_on
  - gemma4_12b_it_q5_k_m_mtp_off
  - gemma4_12b_it_q5_k_m_mtp_on
  - granite_3_3_8b_q8_0
  - phi4_reasoning_vision_15b_q6_k_think
tasks:
  - PERF-005
  - COD-001
  - COD-002
  - RAG-001
  - CHAT-001
  - PERF-004

Result JSONL row

{
  "campaign": "r9700-open-weight-round-001",
  "task_id": "PERF-004",
  "model_profile": "gemma4_12b_it_q5_k_m_mtp_on",
  "base_model": "gemma4_12b_it",
  "serving_variant": "mtp",
  "run_id": 184,
  "score": 0.83,
  "pass_fail": "pass",
  "runtime": {
    "median_generation_tps": 72.4,
    "median_ttft_ms": 310,
    "mtp_acceptance_rate": 0.71,
    "vram_peak_mb": 18320
  },
  "validators": {
    "quality_parity": true,
    "schema_valid": true,
    "no_output_regression": true
  },
  "notes": "MTP improved throughput without score regression."
}

Fit JSONL row

{
  "campaign": "r9700-open-weight-round-001",
  "task_id": "PERF-005",
  "model_profile": "qwen3_6_27b_q4_k_m_mtp_off",
  "gpu_vram_limit_mb": 32768,
  "ctx_size": 65536,
  "status": "pass",
  "fit_class": "green",
  "all_vram": true,
  "hybrid_offload": false,
  "peak_vram_mb": 29880,
  "reserved_headroom_mb": 2048,
  "kv_cache_type": "q8_0",
  "parallel_slots": 1,
  "smoke_generation_tokens": 128
}

Reporting Plane

What the screenshots should show

The sexy screenshots come from useful comparisons: model-by-task leaderboards, speed/quality tradeoffs, reliability failures, and context curves.

Model leaderboard

Sort by composite score within task category. Show rank, score, success rate, median TTFT, median generation TPS, and failures.

Speed vs quality

Scatter plot with score on one axis and tokens/sec on the other. Bubble size is median total tokens or power draw if available.

MTP delta view

Pair bars for MTP off/on: TPS gain, TTFT shift, acceptance rate, VRAM delta, and quality parity.

Reliability heatmap

Rows are models, columns are task types. Color by pass rate or normalized failure rate.

Context curve

Plot score and generation TPS at 4K, 16K, 32K, 64K, and 128K to reveal where each model falls apart.

32GB fit map

Show each profile's max all-VRAM context, max hybrid context, recommended context, and fit class.

RAG grounding table

Show citation accuracy, unsupported claims, answer correctness, and median latency for RAG backend choices.

Coding scoreboard

Show patch-applies rate, tests-passed rate, schema compliance, and average edit size for coder models.

Chatbot profile

Show conversational usefulness, refusal/over-refusal, privacy behavior, and OpenWebUI responsiveness.

Energy and heat view

If telemetry arrives, show tokens per watt, thermal throttling hints, and long-run stability.

Keep public-facing benchmark tasks synthetic and publishable. Private WorkDashboard traffic can teach us what to test, but the actual benchmark prompts should not contain emails, Teams messages, hostnames, secrets, or private project names.

Next Build Steps

Implementation sequence

This turns the HTML plan into a runnable campaign inside the Flight Recorder project.

Add benchmark schema

Extend SQLite or add migrations for campaigns, tasks, validators, benchmark result rows, 32GB fit results, and model profile fields for MTP/speculative serving.

Create fixture tasks

Add synthetic prompts, fixture repos, source documents, answer keys, JSON schemas, and validators under benchmarks/.

Build `flightctl bench run`

The CLI reads a manifest, calls the proxy, runs validators, and stores results linked to Flight Recorder run IDs.

Add reports UI

Use the existing reports page as the base and add benchmark campaign pages, charts, and exportable static HTML snapshots.

Sources

Reference links used for model selection

Source links are intentionally included so the shortlist can be updated as model cards, licenses, quantizations, and llama.cpp support change.