## Turn 1

Local model server (llama.cpp) is slow.
Prompt evaluation (prefill) is much slower than generation (decoding).
Single GPU.
Concise troubleshooting checklist.

    *   *Prompt Eval (Prefill):* Processing the input tokens. This is compute-bound (matrix multiplications).
    *   *Generation (Decoding):* Producing tokens one by one. This is memory-bandwidth bound.
    *   *The Problem:* If prefill is slow, it usually means the GPU isn't being saturated, the KV cache is mismanaged, the context length is too high, or there's a bottleneck in data transfer/memory.

    *   *Hardware/Driver:* Is the GPU actually being used? Is it overheating?
    *   *Model/Quantization:* Is the model too big for VRAM? Is it a weird format?
    *   *llama.cpp Parameters:* `n_batch`, `n_ubatch`, `n_ctx`, `threads`.
    *   *Memory:* Is it spilling to System RAM (GGUF specific)?
    *   *Context:* Is the prompt extremely long?

    *   *Category 1: GPU Utilization & Offloading*
        *   Check `n_gpu_layers`. Is it -1 (all)?
        *   Check VRAM usage. Is it hitting the limit? (Spilling to RAM is a killer).
        *   Check `n_batch` and `n_ubatch`. (Crucial for prefill).

    *   *Category 2: Context & Memory*
        *   `n_ctx`: Is it unnecessarily large?
        *   Flash Attention: Is it enabled? (Huge for prefill speed).

    *   *Category 3: System/Driver*
        *   CUDA/ROCm versions.
        *   Power limits/Thermal throttling.

    *   *Checklist Item 1: GPU Offloading.* Ensure all layers are on GPU.
    *   *Checklist Item 2: Batch Sizes.* `n_batch` and `n_ubatch` are the primary knobs for prefill.
    *   *Checklist Item 3: Flash Attention.* Must-have for long prompts.
    *   *Checklist Item 4: VRAM Overflow.* If it spills to CPU, prefill dies.
    *   *Checklist Item 5: Context Window.* `n_ctx` size vs. actual usage.
    *   *Checklist Item 6: Hardware.* Thermal throttling/Power limits.

    *   *Refining "n_batch":* For prefill, higher is usually better (up to a point).
    *   *Refining "Flash Attention":* Mention `-fa` or `--flash-attn`.
    *   *Refining "VRAM":* Mention `n_gpu_layers`.