## Turn 1

Here's a thinking process:

1.  **Analyze User Input:**
   - **Problem:** Local model server responding slowly
   - **Symptom:** Prompt eval (prefill) is much slower than generation (decode)
   - **Context:** Using `llama.cpp` on a single GPU
   - **Request:** Concise troubleshooting checklist

2.  **Identify Key Concepts:**
   - `llama.cpp` architecture: Prefill (prompt evaluation) vs. Decode (token generation)
   - GPU acceleration: CUDA, ROCm, Metal, Vulkan, etc.
   - Common bottlenecks for prefill: Memory bandwidth, context window size, KV cache, batch size, offloading, quantization, GPU utilization, driver/software issues
   - `llama.cpp` specific settings: `-ngl`/`--gpu-layers`, `-b`/`--batch-size`, `-t`/`--threads`, `-np`/`--parallel`, `--flash-attn`, `--mlock`, `--mmap`, `--numa`, etc.

3.  **Brainstorm Troubleshooting Points (Prefill vs Decode):**
   - *Prefill is slower than decode* is actually somewhat normal depending on the model/context, but if it's *much* slower, it usually indicates:
     - KV cache not fully on GPU (or partially offloaded)
     - Context window too large for VRAM, causing paging/swapping
     - Batch size too small for prefill parallelism
     - Flash attention not enabled
     - GPU not fully utilized during prefill (memory bandwidth bound vs compute bound)
     - Driver/ROCm/CUDA version mismatch or outdated `llama.cpp`
     - Quantization format causing extra CPU/GPU overhead
     - System RAM/PCIe bottleneck if offloading is misconfigured
     - Multi-query attention vs grouped query attention (model architecture dependent, but can't change)
     - `--batch-size` vs `--n-prompt` (parallel prompt processing)

4.  **Structure Checklist (Concise, Actionable):**
   I'll organize it logically: Configuration/Settings -> GPU/VRAM -> Software/Drivers -> Monitoring/Verification. Keep it tight.

   *Checklist Items:*
   - [ ] **VRAM & Offloading:** Ensure full model + KV cache fit in VRAM. Set `--gpu-layers -1` (or max). Check `llama.cpp` logs for "offload" warnings.
   - [ ] **Context Window:** Reduce `--ctx-size`. Large contexts cause memory bandwidth saturation during prefill.
   - [ ] **Batch/Parallelism:** Increase `--batch-size` (default 512) and `--n-prompt` (default 1) to allow parallel prefill processing.
   - [ ] **Flash Attention:** Enable `--flash-attn` (or `--mlock`/`--mmap` as needed). Critical for prefill speed.
   - [ ] **KV Cache Type:** Use `--cache-type-k f16` or `--cache-type-k bf16` (default fp32 can bottleneck prefill).
   - [ ] **Driver/Runtime:** Verify CUDA/ROCm version matches `llama.cpp` build. Update GPU drivers.
   - [ ] **Quantization:** Ensure model is properly quantized (Q4_K_M/Q5_K_M). Avoid Q2/Q3 if possible. Check for CPU fallback ops.
   - [ ] **Monitoring:** Use `nvidia-smi`/`rocm-smi` to verify GPU utilization during prefill. If <90%, check offloading/batch settings.
   - [ ] **Server/Network:** Rule out API overhead (e.g., vLLM vs llama.cpp server, JSON parsing, streaming config).

5.  **Refine & Verify (Self-Correction/Validation):**
   - *Is prefill naturally slower?* Actually, prefill is compute-bound and parallelizable, while decode is memory-bound and sequential. If prefill is *much* slower, it's often due to KV cache spilling to RAM, small batch size, or disabled flash attention.
   - *Check `llama.cpp` specific flags:* 
     - `--gpu-layers` (or `-ngl`)
     - `--ctx-size` (or `-c`)
     - `--batch-size` (or `-b`)
     - `--n-prompt` (parallel prompt processing)
     - `--flash-attn`
     - `--cache-type-k` / `--cache-type-v`
     - `--mmap` / `--mlock`
   - *Server context:* The user mentions "local model server", likely `llama.cpp`'s built-in server (`main` or `llama-server`). I should mention server-specific flags like `--parallel` or `--batch-size` for the server.
   - *Conciseness:* Keep it to bullet points, action-oriented, no fluff.
   - *Accuracy check:* 
     - KV cache type: `--cache-type-k f16` is correct and recommended for speed.
     - Flash attention: `--flash-attn` is correct.
     - Batch size: `--batch-size` controls prefill parallelism. `--n-prompt` controls concurrent prompts.
     - VRAM: Must fit model + KV cache. If not, prefill slows dramatically due to RAM swap.
     - Driver: CUDA 12.x / ROCm 6.x compatibility matters.
     - Monitoring: `nvidia-smi` utilization check is standard.
   - All points are technically sound for `llama.cpp`.

6.  **Draft Output (Mental):**
   **llama.cpp Prefill Bottleneck Checklist (Single GPU)**
   - [ ] **VRAM Capacity:** Verify model weights + KV cache fit entirely in VRAM. Set `--gpu-layers -1`. If VRAM is exceeded, prefill spills to system RAM, causing severe slowdowns.
   - [ ] **KV Cache Type:** Change from default fp32 to fp16/bf16: `--cache-type-k f16 --cache-type-v f16`. Reduces memory bandwidth pressure during prefill.
   - [ ] **Flash Attention:** Enable `--flash-attn`. Critical for prefill throughput on modern GPUs.
   - [ ] **Batch/Parallel Settings:** Increase `--batch-size` (default 512) and `--n-prompt` (default 1) to allow parallel prefill processing. Server: `--parallel 1` (or higher if VRAM allows).
   - [ ] **Context Window:** Reduce `--ctx-size`. Large contexts saturate memory bandwidth during prefill. Test with `--ctx-size 2048` to isolate.
   - [ ] **Driver/Runtime:** Confirm CUDA/ROCm version matches `llama.cpp` build. Update GPU drivers. Rebuild `llama.cpp` with correct `CMAKE_CUDA_ARCHITECTURES` if custom build.
   - [ ] **Quantization:** Use Q4_K_M or Q5_K_M. Avoid Q2/Q3 (higher CPU/GPU overhead). Ensure model file isn’t corrupted.
   - [ ] **Utilization Check:** Run `nvidia-smi`/`rocm-smi` during prefill. If GPU util < 80%, check offloading, batch size, or driver issues.
   - [ ] **Server Overhead:** Disable unnecessary server features (`--log-disable`, `--no-mmap` if RAM-bound). Ensure streaming isn’t causing token-by-token latency masking.

   *Self-Correction/Refinement during drafting:*
   - `--n-prompt` is for concurrent prompts, not necessarily prefill speed for a single prompt. I'll adjust to focus on `--batch-size` and `--parallel` for the server.
   - `--cache-type-k` is correct.
   - Flash attention is `--flash-attn`.
   - I should mention `--mlock` vs `--mmap` briefly if relevant, but it's more about stability than speed. I'll skip to keep it concise.
   - The checklist should be strictly actionable. I'll format it cleanly.
   - Add a note about model architecture: Some models (e.g., MoE, large context) inherently have slower prefill, but the checklist covers mitigations.
   - Check `llama.cpp` server flags: `--batch-size`, `--ctx-size`, `--gpu-layers`, `--flash-attn`, `--cache-type-k`, `--parallel`. All correct.
   - I'll structure it as a tight, prioritized list.

   Final structure:
   - VRAM & Offloading
   - Memory Bandwidth & Cache
   - Parallelism & Batch
   - Software & Drivers
   - Verification
   Each with 1-2 bullet points. Concise.

7.  **Generate Output:** (matches the refined draft)✅
   I will ensure it strictly follows the prompt: concise, troubleshooting checklist, llama.cpp, single GPU, addresses prefill vs decode. No extra fluff. Output format: just the checklist. I'll add minimal context if needed, but keep it tight.
   Ready.✅
   Output matches response. 
   Self-Correction/Verification during thought: 
   - "Prompt eval is much slower than generation" is actually the opposite of typical behavior (prefill is usually faster than decode per-token, but total prefill time can be high). If it's *slower*, it's almost always VRAM spilling, small batch size, or disabled flash attention. I'll make sure the checklist targets these.
   - I'll explicitly mention `--cache-type-k f16` as it's a known `llama.cpp` optimization for prefill.
   - I'll keep it to exactly 8-10 actionable items.
   - All good.✅
   Output generation. 
   [Proceeds]