## Turn 1

Here's a thinking process:

1.  **Analyze User Input:**
   - **Problem:** Local model server responding slowly
   - **Symptom:** Prompt evaluation (prefill) is much slower than token generation (decode)
   - **Context:** llama.cpp on a single GPU
   - **Request:** Concise troubleshooting checklist

2.  **Identify Key Concepts:**
   - Prompt evaluation (prefill) vs. generation (decode) in LLM inference
   - llama.cpp architecture and GPU offloading
   - Common bottlenecks for prefill on GPU
   - Single GPU constraints
   - llama.cpp specific flags/configurations

3.  **Brainstorming Potential Causes & Solutions:**
   - *GPU Offloading:* Not fully offloaded? Layers on CPU? Need `--ngl` or `--gpu-layers` to max.
   - *Batch Size/Context:* Prefill batch size too small? `--batch` or `--ubatch` settings.
   - *Memory/Bandwidth:* VRAM bandwidth bottleneck? Using slower memory? Check VRAM usage.
   - *Model Format/Quantization:* GGUF format correct? Quantization type affecting compute? (Usually less impact on prefill vs decode, but still relevant)
   - *CUDA/Backend:* Correct backend selected? `--flash-attn` enabled? (Flash attention speeds up prefill significantly)
   - *Context Length:* Very long prompts? Prefill scales O(N) or O(N^2) depending on attention. Flash attention helps.
   - *Threading:* CPU threads interfering? `--threads` or `--threads-batch` settings.
   - *Driver/Environment:* Outdated CUDA drivers? ROCm issues? (Assume CUDA for single GPU, but keep it general)
   - *Server Config:* `--host`, `--port`, `--parallel`, `--slots`? Not directly related to prefill speed.
   - *llama.cpp specific flags:* `--flash-attn`, `--ngl`, `--batch`, `--ubatch`, `--mlock`, `--no-mmap`

4.  **Structure the Checklist (Concise & Actionable):**
   Group by category for clarity:
   - GPU Offloading & Backend
   - Attention & Batch Settings
   - Memory & Threading
   - Model & Environment
   - Verification Steps

   Draft:
   1. **Full GPU Offload:** Ensure `--ngl` (or `--gpu-layers`) equals total model layers. CPU offload kills prefill speed.
   2. **Enable Flash Attention:** Add `--flash-attn` (requires CUDA/ROCm). Dramatically speeds up long-context prefill.
   3. **Batch Size Tuning:** Increase `--batch` (prefill batch) and `--ubatch` (max batch). Default is often too low (e.g., 512). Try 2048-4096.
   4. **VRAM & Memory Bandwidth:** Verify model fits entirely in VRAM. Check `nvidia-smi`/`rocm-smi`. Swap/pagefile usage will tank prefill.
   5. **Thread Configuration:** Set `--threads` to physical cores. Use `--threads-batch` for prefill parallelism (often matches `--threads`).
   6. **Disable MMAP if Needed:** If prefill is inconsistent/slow, try `--no-mmap` to force direct memory load (avoids page faults).
   7. **Check Quantization/Format:** Use Q4_K_M or Q5_K_M for balance. Avoid exotic quants that lack GPU kernels. Ensure GGUF is v3+.
   8. **Verify Backend:** Confirm correct CUDA/ROCm version. Run `llama-cli -m model.gguf -p "test" -n 1 -v` to check backend detection.
   9. **Context Length Impact:** Prefill scales with prompt length. Test with short vs long prompts to isolate. Flash attention mitigates quadratic scaling.
   10. **Update llama.cpp:** Use latest `master` or `release` branch. GPU prefill optimizations are frequent.

5.  **Refine & Condense (Self-Correction/Verification):**
   - *Check accuracy:* `--ngl` is correct for llama.cpp. `--flash-attn` is correct and crucial for prefill. `--batch`/`--ubatch` are correct. `--threads-batch` exists. `--no-mmap` can help with I/O bottlenecks. VRAM check is standard. Backend verification is good.
   - *Conciseness:* Keep it tight. Use bullet points. Remove fluff.
   - *Single GPU focus:* Emphasize VRAM limits, full offload, flash attention.
   - *llama.cpp specific:* Flags are accurate for current versions (v2.0+). Note that `--ngl` might be `--gpu-layers` in some wrappers, but `--ngl` is standard CLI. I'll stick to standard CLI flags.
   - *Structure:* Group logically. Add quick verification command.

   Revised Checklist:
   **GPU & Backend**
   - [ ] **Full Offload:** Set `--ngl` to total layers (e.g., `--ngl 99`). Any CPU layers bottleneck prefill.
   - [ ] **Flash Attention:** Enable `--flash-attn` (CUDA/ROCm). Critical for long-context prefill speed.
   - [ ] **Backend Verification:** Run `llama-cli -m model.gguf -n 1 -p "test" -v` to confirm GPU backend is active and no fallback warnings.

   **Batch & Threading**
   - [ ] **Increase Batch Size:** Set `--batch 2048` (or higher) and `--ubatch 2048`. Default (512) limits prefill throughput.
   - [ ] **Thread Tuning:** Set `--threads` to physical cores. Use `--threads-batch` (same value) for prefill parallelism.

   **Memory & I/O**
   - [ ] **VRAM Check:** Ensure entire model + KV cache fits in VRAM. Use `nvidia-smi`/`rocm-smi`. Swap/pagefile kills prefill.
   - [ ] **Disable MMAP:** Add `--no-mmap` if prefill is inconsistent or disk-bound. Forces direct RAM/VRAM load.

   **Model & Environment**
   - [ ] **Quantization:** Use standard quants (Q4_K_M, Q5_K_M). Exotic quants may lack optimized GPU kernels.
   - [ ] **Update llama.cpp:** Pull latest `master`. GPU prefill optimizations are frequent.
   - [ ] **Context Length Test:** Benchmark short vs. long prompts. Prefill scales linearly (or quadratically without flash-attn).

   **Quick Validation Command:**
   `llama-server -m model.gguf --ngl 99 --flash-attn --batch 2048 --ubatch 2048 --threads 8 --threads-batch 8 --no-mmap`

6.  **Final Review against User Prompt:**
   - "prompt eval is much slower than generation" -> Addressed (prefill bottlenecks: batch size, flash attn, offload, VRAM, mmap)
   - "concise troubleshooting checklist" -> Formatted as checklist, grouped, actionable
   - "llama.cpp on a single GPU" -> Tailored flags, VRAM focus, single-GPU constraints noted
   - Output format: Just the checklist as requested. I'll ensure it's tight.

   Everything looks solid. I'll output it.✅
   Self-Correction/Note during thought: 
   - `--threads-batch` is valid in recent llama.cpp versions.
   - `--ubatch` is the max batch size, `--batch` is the prefill batch size. I'll clarify briefly.
   - Flash attention is definitely the #1 fix for slow prefill on GPU in llama.cpp.
   - I'll keep it strictly concise. No extra fluff.
   - Format: Markdown checklist. Ready. 
   Output matches response. 
   Proceeds. 
   [Output Generation] -> *Proceeds*
   *(Self-Correction/Refinement during drafting)*: 
   I should verify flag names for latest llama.cpp (v2.0+). 
   `--ngl` is correct. 
   `--flash-attn` is correct. 
   `--batch` and `--ubatch` are correct. 
   `--threads` and `--threads-batch` are correct. 
   `--no-mmap` is correct. 
   All good. I'll present it cleanly. 
   Structure: 
   - GPU/Backend
   - Batch/Threading
   - Memory/I/O
   - Model/Env
   - Quick test command
   Concise. Direct. Matches request.✅
   Output matches the refined version. 
   [Done.] 
   *Output Generation* (matches the final refined version)
   "Here’s a concise troubleshooting checklist for llama.cpp on a single GPU when prompt evaluation (prefill) is bottlenecking:
   **GPU & Backend**
   - [ ] **Full GPU Offload:** Set `--ngl` to the model’s total layer count (e.g., `--ngl 99`). Any CPU layers severely throttle prefill.
   - [ ] **Enable Flash Attention:** Add `--flash-attn` (requires CUDA/ROCm). This is the single biggest prefill speedup for long contexts.
   - [ ] **Verify Backend:** Run `llama-cli -m model.gguf -n 1 -p "test" -v` to confirm GPU backend is active and no fallback/warning messages appear.
   **Batch & Threading**
   - [ ] **Increase Prefill Batch:** Set `--batch 2048` (or higher) and `--ubatch 2048`. The default (512) caps prefill throughput.
   - [ ] **Thread Configuration:** Set `--threads` to physical cores. Mirror with `--threads-batch` for prefill parallelism.
   **Memory & I/O**
   - [ ] **VRAM Capacity:** Ensure the full model + KV cache fits in VRAM. Check `nvidia-smi`/`rocm-smi`. Hitting system RAM/pagefile tanks prefill.
   - [ ] **Disable MMAP:** Add `--no-mmap` if prefill is inconsistent or disk-bound. Forces direct memory load, avoiding page faults.
   **Model & Environment**
   - [ ] **Quantization Check:** Use standard quants (Q4_K_M, Q5_K_M). Exotic quants may lack optimized GPU kernels.
   - [ ] **Update llama.cpp:** Pull the latest `master` branch. GPU prefill optimizations are frequent.
   - [ ] **Context Length Test:** Benchmark short vs. long prompts. Prefill scales with token count; flash-attn mitigates quadratic attention overhead.
   **Quick Validation Command:**
   `llama-server -m model.gguf --ngl 99 --flash-attn --batch 2048 --ubatch 2048 --threads 8 --threads-batch 8 --no-mmap`
   *(Adjust `--ngl`, `--threads`, and batch sizes to your hardware.)*"
   This matches all constraints. Ready. 
   Output matches response.✅
   [Final Check of the Prompt]: "Give a concise troubleshooting checklist for llama.cpp on a single GPU." -> Met.
   All good.✅
   Output Generation. 
   *(Self-Correction/Note during output gen prep)*: I'll make sure the formatting is exactly as requested. No extra text. Just the checklist. The prompt says "concise troubleshooting checklist", so I'll keep it tight. I'll output exactly that.✅
   Proceeds. 
   [Output] -> *Proceeds* (matches the refined version)
   *(Done.)*