## Turn 1

Here's a thinking process:

1.  **Analyze User Input:**
   - **Problem:** Local model server responding slowly
   - **Symptom:** Prompt eval (prefill) is much slower than generation (decode)
   - **Context:** llama.cpp, single GPU
   - **Request:** Concise troubleshooting checklist

2.  **Identify Key Concepts:**
   - `llama.cpp` architecture: Prefill (prompt evaluation) vs. Decode (token generation)
   - Single GPU constraints: Memory bandwidth, compute utilization, KV cache, offloading, context window, batch size, quantization, GPU architecture compatibility
   - Common bottlenecks for prefill: Large context, inefficient batch processing, CPU/GPU sync, memory bandwidth limits, suboptimal compilation flags, KV cache fragmentation, prompt processing overhead

3.  **Brainstorm Troubleshooting Steps (Mental Refinement):**
   - *Check GPU utilization & memory:* Is the GPU fully utilized during prefill? Is VRAM maxed out causing swapping?
   - *Context window & batch size:* Large context + large batch = memory bandwidth bottleneck. Reduce `--batch` or `--ctx-size`.
   - *KV cache & offloading:* Ensure full model is on GPU (`-ngl 100` or similar). Check for CPU offload causing sync delays.
   - *Compilation/Build flags:* Was it built with proper GPU support (CUDA/Metal/Vulkan)? Check for fallback to CPU or inefficient kernels.
   - *Prompt processing:* Is the prompt unusually long? Tokenization overhead? Consider `--prompt-cache` or pre-tokenizing.
   - *Server/Server config:* Check `--threads`, `--parallel`, `--log-disable`, server-specific settings (e.g., Ollama, text-generation-webui, vLLM-like settings if applicable).
   - *Quantization/Model format:* GGUF version? Some older GGUFs have slower prefill. Ensure latest llama.cpp.
   - *Memory bandwidth vs compute:* Prefill is memory-bound, decode is compute-bound. On single GPU, memory bandwidth often limits prefill. Check if GPU is high-bandwidth (H100/A100 vs consumer GPU).
   - *Check logs/metrics:* Use `--log-level debug` or profiling tools to pinpoint exact bottleneck.

4.  **Structure as Concise Checklist:**
   Need to be direct, actionable, and specific to llama.cpp + single GPU + prefill bottleneck.
   Group logically: Configuration, Hardware/VRAM, Build/Software, Prompt/Workload, Diagnostics.

   Draft:
   - [ ] Verify full GPU offload (`-ngl 100` or `--gpu-layers all`). Partial offload causes CPU-GPU sync stalls during prefill.
   - [ ] Reduce batch/context size: Prefill is memory-bound. Lower `--batch` and `--ctx-size` to match VRAM bandwidth limits.
   - [ ] Check build flags: Ensure compiled with correct backend (`-DGGML_CUDA=ON` or Metal/Vulkan). Fallback to CPU or unoptimized kernels kills prefill.
   - [ ] Update llama.cpp & GGUF: Use latest release. Older GGUFs or builds lack fused prefill kernels.
   - [ ] Disable unnecessary logging/overhead: `--log-disable` or `--log-level warn`. Server wrappers may add serialization delays.
   - [ ] Check VRAM fragmentation/swap: `nvidia-smi` or `rocm-smi`. If VRAM >95%, prefill stalls due to paging or cache thrashing.
   - [ ] Test with short prompt: If prefill normalizes, bottleneck is context-window scaling (memory bandwidth limit).
   - [ ] Verify parallelism settings: `--parallel 1` (single GPU). Higher values cause thread contention without benefit.
   - [ ] Profile with `--log-level debug` or `--verbose`: Look for `ggml_backend_cuda` fallbacks, CPU sync waits, or KV cache reallocation.

5.  **Refine & Check against Constraints:**
   - Concise? Yes, bullet checklist format.
   - Specific to llama.cpp? Yes, uses exact flags and concepts.
   - Single GPU focus? Yes, addresses VRAM, parallelism, offloading.
   - Addresses prefill vs decode? Yes, explicitly notes memory-bound nature of prefill and bandwidth limits.
   - Actionable? Yes, each item has a clear check/fix.

   Self-Correction/Improvement during drafting:
   - "ngl 100" is specific to llama-server/llama-cli. Better to say `--gpu-layers all` or `-ngl <total_layers>`.
   - Add note about memory bandwidth being the fundamental limit for prefill on single GPU.
   - Clarify that prefill being slower than decode is *normal* for large prompts on memory-bound GPUs, but can be optimized.
   - Keep it tight. Remove fluff.

   Final structure:
   - [ ] Verify full GPU offload (`--gpu-layers all` / `-ngl <N>`)
   - [ ] Reduce `--batch` & `--ctx-size` (prefill is memory-bandwidth bound)
   - [ ] Confirm correct backend build (`-DGGML_CUDA=ON` / Metal / Vulkan)
   - [ ] Update to latest `llama.cpp` & GGUF format
   - [ ] Check VRAM usage & paging (`nvidia-smi` / `rocm-smi`)
   - [ ] Set `--parallel 1` (single GPU)
   - [ ] Disable verbose logging (`--log-disable` or `--log-level warn`)
   - [ ] Test with short prompt to isolate context scaling
   - [ ] Profile with `--log-level debug` for CPU sync or KV cache reallocation warnings

   Add brief context note: Prefill is inherently memory-bound; on a single GPU, bandwidth often limits it vs compute-bound decode. This checklist targets configuration/build bottlenecks.

   Matches requirements. Output matches exactly. Proceed. 
   Self-Correction/Verification during thought:
   - Check llama.cpp flag names: `--gpu-layers` is correct for newer versions. `-ngl` is legacy but still works. I'll use `--gpu-layers all`.
   - `--parallel` is correct.
   - `--batch` and `--ctx-size` are correct.
   - Build flag: `-DGGML_CUDA=ON` is correct for cmake.
   - All accurate.
   - Format: Just the checklist as requested. I'll keep it tight.
   - Output matches response.✅
   Proceed. 
   Output Generation. 
   [Done] 
   *Self-Correction/Refinement during generation prep*: 
   I'll make sure it's strictly a checklist format as requested, with minimal fluff. I'll add a one-line context note if needed, but keep it tight. The prompt says "Give a concise troubleshooting checklist", so I'll deliver exactly that.
   Structure:
   - [ ] ...
   - [ ] ...
   etc.
   Ready. 
   Output matches the refined version.✅
   *Output Generation* (matches the final refined version)
   "[ ] Verify full GPU offload...
   [ ] Reduce batch/context...
   etc." 
   All good. 
   Proceeds. 
   [Output Generation] -> *See response.* 
   *(Note: I'll ensure the formatting strictly follows the request. No extra text.)*
   Actually, the prompt says "Give a concise troubleshooting checklist for llama.cpp on a single GPU." I will provide exactly that. I'll add a brief introductory sentence if needed, but keep it tight. The output format requirement is just "message.content". I'll follow standard markdown checklist.
   Final check of the checklist items for accuracy:
   - Prefill is memory-bound, decode is compute-bound. This is fundamental.
   - Single GPU means no multi-GPU offload tricks.
   - Flags are correct for modern llama.cpp (v0.3+).
   - All actionable.
   Ready. 
   Output matches response. 
   [Proceeds]