## Turn 1

Here's a thinking process:

1.  **Analyze User Input:**
   - **Problem:** Local model server responding slowly
   - **Symptom:** Prompt eval (prefill) is much slower than generation (decode)
   - **Context:** Using `llama.cpp` on a single GPU
   - **Request:** Concise troubleshooting checklist

2.  **Identify Key Concepts in llama.cpp:**
   - `llama.cpp` splits inference into two phases: prefill (prompt evaluation) and decode (token generation)
   - Prefill is typically memory-bandwidth bound and parallelizable across the prompt length
   - Decode is typically compute-bound and sequential
   - Single GPU setup implies potential bottlenecks: VRAM capacity, memory bandwidth, compute utilization, context window size, batch size, quantization, offloading, kernel selection, etc.

3.  **Brainstorm Troubleshooting Areas for Prefill Slowness:**
   - **VRAM/Offloading:** Is the model fully on GPU? Partial offloading causes CPU-GPU transfers during prefill.
   - **Context Window:** Large context windows increase prefill time significantly. Is it necessary?
   - **Batch Size:** `--batch` or `--parallel` settings. Too large can cause memory fragmentation or inefficient kernel launches.
   - **Quantization/Format:** Some quant formats (e.g., Q4_K_M) might have slower prefill kernels compared to others. Check if using optimal format for the GPU.
   - **Kernel/Compute Backend:** CUDA vs ROCm vs Metal. CUDA usually best for NVIDIA. Check if using optimized kernels (e.g., `llama.cpp`'s CUDA kernels are enabled).
   - **Memory Bandwidth:** Prefill is bandwidth-bound. Is the GPU memory saturated? Check for other processes using VRAM.
   - **Prompt Length/Tokenizer:** Very long prompts or inefficient tokenization can slow prefill.
   - **llama.cpp Version/Flags:** Outdated version? Missing optimization flags like `--flash-attn`, `--mlock`, `--no-mmap`, `--tensor-split` (though single GPU, still relevant for internal routing).
   - **Server/HTTP Overhead:** Is the slowness actually in the server wrapper (e.g., Ollama, LM Studio, vLLM, custom FastAPI) rather than llama.cpp core? Check raw `llama.cpp` server vs wrapper.
   - **GPU Utilization/Monitoring:** Use `nvidia-smi` or `nvtop` to check if GPU is actually being utilized during prefill. Low utilization = bottleneck elsewhere.

4.  **Structure as a Concise Checklist:**
   Need to organize logically, keep it concise, and focus on single-GPU llama.cpp specifics.
   Categories:
   - VRAM & Offloading
   - Context & Batch Settings
   - Kernel & Backend Optimization
   - Monitoring & Isolation
   - Server/Wrapper Check

   Draft checklist:
   - [ ] Verify full GPU offload: `--ngl 999` (or `--tensor-split 1.0`). Partial offload causes CPU↔GPU transfers during prefill.
   - [ ] Check VRAM usage: Ensure no other processes consume VRAM. Prefill is memory-bandwidth bound; fragmentation or shared VRAM causes thrashing.
   - [ ] Reduce context window: Use `--ctx-size` only as large as needed. Prefill scales linearly with prompt length.
   - [ ] Optimize batch/parallel: Lower `--batch` and `--parallel` (or `--threads`). High values can cause inefficient kernel launches or memory overhead on single GPU.
   - [ ] Use optimal quantization: Prefer Q4_K_M or Q5_K_M for NVIDIA. Avoid older/slower formats like Q2_K or Q8_0 unless necessary.
   - [ ] Enable flash attention: Add `--flash-attn` if supported by your GPU/architecture.
   - [ ] Verify backend: Ensure CUDA (NVIDIA) or ROCm (AMD) is active. Check `llama.cpp` build flags for `GGML_CUDA` or `GGML_HIP`.
   - [ ] Monitor GPU utilization: Run `nvtop`/`nvidia-smi` during prefill. <70% utilization indicates CPU bottleneck, memory bandwidth limit, or inefficient kernels.
   - [ ] Test raw llama.cpp server: Bypass wrappers (Ollama, LM Studio, etc.) with `./server -m model.gguf -c 2048 --port 8080`. Isolate server overhead from inference.
   - [ ] Update llama.cpp: Pull latest `main` branch. Prefill kernels and memory allocators are frequently optimized.

5.  **Refine & Check Against Requirements:**
   - Concise? Yes, bullet/checklist format, direct actions.
   - Single GPU focus? Yes, emphasizes VRAM, offloading, batch/parallel tuning, bandwidth limits.
   - Addresses prompt eval vs generation? Yes, explicitly targets prefill bottlenecks (bandwidth, context size, offloading, kernel efficiency).
   - llama.cpp specific? Yes, uses correct flags (`--ngl`, `--ctx-size`, `--batch`, `--flash-attn`, `GGML_CUDA`, etc.).
   - Format: "final answer in message.content" - I'll output just the checklist as requested.

   Self-Correction/Verification during drafting:
   - `--ngl 999` is standard for full GPU offload in llama.cpp.
   - `--flash-attn` is valid in recent llama.cpp versions.
   - Batch/parallel: In llama.cpp server, `--batch` and `--parallel` control KV cache and prefill batching. High values can hurt single-GPU prefill due to memory fragmentation and kernel launch overhead. Lowering them often helps.
   - Quantization: Q4_K_M is generally best for NVIDIA. Q8_0 is slower due to higher bandwidth demand.
   - Monitoring: `nvtop` is perfect.
   - Wrapper isolation: Critical because many "llama.cpp servers" are actually wrappers with extra latency.
   - All points are accurate and concise.

   Structure:
   - Use markdown checklist format
   - Keep descriptions brief
   - Group logically if needed, but keep it tight
   - Ensure technical accuracy for llama.cpp v24+ (current era)

   Final check of the prompt: "Give a concise troubleshooting checklist for llama.cpp on a single GPU."
   I'll output exactly that. No extra fluff. Follows format. Ready. 
   Output matches the refined draft.✅
   Proceeds. 
   Output generation. 
   [Self-Correction/Note during thought]: 
   I should make sure flags match current llama.cpp conventions. `--ngl` is correct. `--ctx-size` is correct. `--flash-attn` is correct. `--batch` and `--parallel` are correct. I'll add a note about `--mlock`/`--no-mmap` if relevant, but maybe keep it tight. The checklist is solid. I'll present it clearly.✅
   Output matches response. 
   Proceeds. 
   [Output Generation] -> *Proceeds*
   *(Self-Correction/Verification during output prep)*: 
   Wait, `--flash-attn` might not be enabled by default in all builds, but it's a valid flag. I'll phrase it as "Enable if supported". 
   Also, `--tensor-split` is for multi-GPU, so I'll remove it to avoid confusion. 
   I'll keep it strictly single-GPU focused. 
   All good. 
   Output matches the refined version.✅
   [Final Check of the Prompt]: "My local model server is responding slowly. Logs show prompt eval is much slower than generation. Give a concise troubleshooting checklist for llama.cpp on a single GPU."
   Matches perfectly. 
   Proceeds. 
   [Output Generation] -> *Proceeds*