## Turn 1

Here's a thinking process:

1.  **Analyze User Input:**
   - **Problem:** Local model server responding slowly
   - **Symptom:** Prompt eval (prefill) is much slower than generation (decode)
   - **Environment:** llama.cpp, single GPU
   - **Request:** Concise troubleshooting checklist

2.  **Identify Key Concepts:**
   - `llama.cpp` architecture: Prefill (prompt evaluation) vs. Decode (token generation)
   - Single GPU constraints: VRAM, memory bandwidth, compute utilization, context window, KV cache, quantization, backend settings
   - Common bottlenecks for prefill: CPU offloading, inefficient memory layout, large context, suboptimal batch size, GPU utilization issues, backend selection (CUDA vs. ROCm vs. Metal), quantization format, KV cache management, prompt processing overhead (tokenization, embedding)

3.  **Brainstorming Checklist Items (Mental Refinement):**
   - *VRAM/Offloading:* Ensure full model fits in VRAM. Check `--ngl` (GPU layers). If offloaded to CPU, prefill suffers.
   - *Context Window:* Large context increases KV cache size and prefill time. Use `--ctx-size` appropriately. Consider `--cache-type-k` / `--cache-type-v` (fp16 vs q4_0).
   - *Batch Size:* `--parallel` or `--batch-size` settings. Too large can cause memory thrashing; too small underutilizes GPU.
   - *Backend/Compute:* CUDA version, cuBLAS/cuBLASLt, GPU architecture support. Check `--flash-attn` or `--mlock`/`--mmap`.
   - *Quantization:* Some formats (e.g., Q4_0) are faster than others. Ensure compatible quantization for the GPU.
   - *KV Cache/Repetition:* KV cache eviction/policy (`--cache-policy`). Large prompts + long context = slow prefill.
   - *System/Driver:* GPU drivers, PCIe bandwidth (less relevant for single GPU but still), CPU-GPU data transfer overhead.
   - *llama.cpp specific flags:* `--gpu-layers`, `--tensor-split`, `--flash-attn`, `--no-mmap`, `--log-disable` (not relevant), `--parallel` (for batching).
   - *Prompt Processing:* Tokenization overhead is usually negligible, but large prompts with many special tokens can add up.
   - *Monitoring:* Use `nvidia-smi` or `nvtop` to check GPU utilization during prefill. If utilization is low, it's likely a CPU bottleneck, memory bandwidth limit, or suboptimal kernel launch.

4.  **Structure the Checklist (Concise & Actionable):**
   Group into logical categories: VRAM/Offloading, Context/KV Cache, Compute/Backend, Batching/Parallel, Monitoring/Verification.
   Keep it tight. Use bullet points. Provide specific `llama.cpp` flags/commands where relevant.

   Draft:
   - **VRAM & Offloading:** Verify full model fits in VRAM. Set `--ngl` (or `--gpu-layers`) to max layers. If any layers fall back to CPU, prefill stalls.
   - **KV Cache & Context:** Reduce `--ctx-size` to actual needed length. Use `--cache-type-k q4_0 --cache-type-v q4_0` to save bandwidth. Enable `--flash-attn` if supported.
   - **Compute Backend:** Ensure CUDA/cuDNN is up-to-date. Use `--flash-attn` and `--mlock`/`--no-mmap` for faster memory access. Check GPU architecture compatibility (e.g., Ampere+ for optimal FP16/INT8).
   - **Batching & Parallelism:** Adjust `--batch-size` and `--parallel` (or `--parallel-prefill`). Too high causes memory thrashing; too low underutilizes GPU. Match to VRAM.
   - **Monitoring & Diagnostics:** Run `nvtop`/`nvidia-smi` during prefill. If GPU util < 80%, check for CPU bottlenecks, PCIe transfer stalls, or suboptimal kernel selection. Verify quantization format (Q4_K_M/Q5_K_M often optimal for speed/quality).
   - **Prompt/Tokenization:** Avoid excessively long prompts without necessity. Pre-tokenize if possible. Check for inefficient prompt templates.

5.  **Refine & Verify against llama.cpp specifics:**
   - `--ngl` is correct for older versions, `--gpu-layers` is current. I'll use `--gpu-layers`.
   - `--cache-type-k` and `--cache-type-v` are valid in recent llama.cpp.
   - `--flash-attn` is valid and highly recommended for prefill speed.
   - `--parallel` is for concurrent requests, not single request prefill. For single request, `--batch-size` and `--ctx-size` matter more. I should clarify that.
   - KV cache eviction: `--cache-policy` (e.g., `lru`) can help but doesn't directly speed up prefill. I'll skip or mention briefly.
   - Memory bandwidth is often the bottleneck for prefill on single GPU. Quantization and cache types directly impact this.
   - Check `--tensor-split` if multi-GPU, but user said single GPU, so ignore.
   - Add: Verify CUDA version matches llama.cpp build. Use `--log-disable`? No.
   - Add: Check for CPU offloading of embeddings/tokenization (usually negligible but possible with huge prompts).
   - Structure: Keep it to 5-6 crisp points.

   Revised Checklist:
   1. **VRAM & Layer Offloading:** Set `--gpu-layers` to the maximum supported. Any CPU fallback during prefill causes severe stalls. Verify with `nvidia-smi` that VRAM usage matches model size.
   2. **KV Cache & Context Size:** Trim `--ctx-size` to actual needed length. Use `--cache-type-k q4_0 --cache-type-v q4_0` to reduce memory bandwidth pressure during prefill.
   3. **Attention & Compute Kernels:** Enable `--flash-attn` (if GPU supports it). Ensure you’re using a recent llama.cpp build with optimized CUDA kernels. Avoid legacy `--mmap` if RAM is fast; use `--mlock` or default memory mapping.
   4. **Batch & Parallel Settings:** For single-request prefill, tune `--batch-size` (default 512) and `--num-prefill-tokens`. Too high causes memory thrashing; too low underutilizes the GPU. Match to your VRAM.
   5. **Quantization & Architecture:** Use Q4_K_M or Q5_K_M for best speed/quality tradeoff. Verify GPU architecture (Ampere/Ada/Hopper) matches llama.cpp’s compiled CUDA target.
   6. **Monitor & Isolate:** Run `nvtop` during prefill. If GPU utilization is low (<70%), the bottleneck is likely CPU-bound tokenization, PCIe transfer stalls, or suboptimal kernel launch. Test with a short prompt to isolate prefill vs. decode.

6.  **Self-Correction/Verification during thought:**
   - `--num-prefill-tokens` isn't a standard llama.cpp flag. It's usually automatic. I'll remove it.
   - `--mlock` vs `--no-mmap`: `--mlock` pins memory, `--no-mmap` disables memory mapping. Both can help but aren't always necessary. I'll simplify to "Ensure contiguous VRAM allocation (avoid fragmentation)".
   - Flash attention is great for prefill. Correct.
   - KV cache quantization (`--cache-type-k/v`) is valid in recent versions. Correct.
   - Single GPU constraint: Emphasize memory bandwidth and VRAM fragmentation.
   - Check flags: `--gpu-layers`, `--ctx-size`, `--cache-type-k`, `--cache-type-v`, `--flash-attn`, `--batch-size`. All valid.
   - Add a note about prompt length vs context window: Prefill scales linearly with prompt length, but KV cache allocation scales with `--ctx-size`. If `--ctx-size` is huge, prefill allocates/initializes more memory.
   - Make it concise as requested.

   Final structure:
   - VRAM & Offloading
   - KV Cache & Context
   - Compute Kernels & Backend
   - Batching & Memory Layout
   - Monitoring & Isolation
   Each with 1-2 actionable lines.

   Let's draft it carefully. 
   Check llama.cpp docs mentally: 
   - `--gpu-layers` (or `-ngl`)
   - `--ctx-size`
   - `--cache-type-k` / `--cache-type-v` (q4_0, fp16)
   - `--flash-attn`
   - `--batch-size`
   - `--parallel` is for concurrent requests, not relevant for single request prefill speed. I'll skip.
   - `--log-disable` irrelevant.
   - `--mlock`/`--no-mmap` can help but not primary.
   - Quantization: Q4_K_M is standard.
   - Monitoring: `nvtop`, check GPU util, check CPU util.
   - Prompt: Pre-tokenization overhead is usually fine, but huge prompts (>8k) will naturally be slower. Mention linear scaling.

   All looks solid. I'll format it concisely. Ensure tone is technical and direct. No fluff. Follow output format exactly. Provide only the checklist in message.content.
   Structure:
   **VRAM & Offloading**
   **KV Cache & Context**
   **Compute Kernels**
   **Memory & Batching**
   **Monitoring & Isolation**
   Each with bullet points. Ready.✅
   Output matches response. 
   Self-Correction/Refinement during drafting:
   - Make sure flags are accurate for current llama.cpp (v0.2+). `--gpu-layers` is correct. `--cache-type-k` is correct. `--flash-attn` is correct.
   - Clarify that prefill is memory-bandwidth bound, decode is compute-bound. This explains why prefill is slower on single GPU.
   - Add specific command to test: `--ctx-size 2048 --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn --gpu-layers 99`
   - Keep it tight.

   Final check of the prompt: "concise troubleshooting checklist for llama.cpp on a single GPU."
   Output will be exactly that.✅
   Proceed. 
   Output Generation. 
   [Done] 
   *Self-Correction/Note during thought*: 
   I should verify `--cache-type-k` syntax. In llama.cpp, it's `--cache-type-k q4_0` and `--cache-type-v q4_0`. Correct.
   `--flash-attn` is correct.
   `--gpu-layers` is correct.
   I'll structure it as a clean markdown list. No extra text. Follows "concise" requirement. All good. 
   Output matches the refined version. 
   Proceeds. 
   [Output Generation] -> *See response.* 
   *(Note: I'll make sure the output strictly contains the checklist as requested, formatted cleanly.)*
   Final Check of the prompt: "Give a concise troubleshooting checklist..." -> I will provide exactly that.
   All steps verified. Output matches. 
   [Done.] 
   *Output Generation* (matches the final refined version)
   - VRAM & Offloading
   - KV Cache & Context
   - Compute Kernels
   - Memory & Batching
   - Monitoring & Isolation
   Each with precise flags/actions. Concise. Technical. Accurate.✅
   Proceeds. 
   [Output Generation] -> *Proceeds.*