Okay, so I need to figure out why my local model server is slow, specifically that prompt evaluation is taking much longer than generation. Let me start by recalling what happens in models like Phi's (assuming Phi is similar to LLaMA or something). First, generation usually involves feeding input and getting output through the model's decoder. Prompt eval might be a different process, maybe evaluating the model's performance on a specific prompt, which could involve more steps. Let me break down the possible areas to check. Hardware-wise, maybe the GPU is being underutilized. Check if the server has enough GPU memory. If there's OOM (out-of-memory) issues during prompt eval, that could slow things down. Also, check the GPU utilization. Is the generation and eval using the same GPU? Maybe there's contention or the model isn't optimized for the GPU architecture. Looking at the code in llama.cpp, prompt eval might be doing more computations. For example, maybe it's running the model in evaluation mode with certain settings that are less optimized. Check if there's any difference in how the model is loaded, like using half-precision vs full. Maybe prompt eval requires more memory or different tensor operations that are slower. Another angle: data transfer. Are there unnecessary data transfers between CPU and GPU during prompt eval? That could add latency. Also, check if the input data for eval is being preprocessed differently, like larger batch sizes or more tokens, which could slow down per-example processing. Profiling is key here. Use a profiler to see where the time is spent during prompt eval. Maybe the attention mechanisms are taking longer. Look into the implementation of the eval function—are there any loops or operations that are inefficient? Also, check if there's any logging or debugging code that's only enabled during eval, adding overhead. Concurrency might be an issue. If the server is handling multiple requests, maybe the GPU is context switching a lot. Check if the server is using asynchronous operations or streams efficiently. For example, overlapping data loading with generation could help, but if eval is blocking, that's a problem. Thermal throttling could be a factor. If the GPU gets too hot during eval, it might slow down. Monitor temperatures to see if that's happening. Also, check for any system-level issues like low disk I/O if data is being read from a slow storage medium during eval. Software updates: Is the model running on the latest version of the framework? Maybe there's a known issue with prompt eval in the current version. Check the changelogs for optimizations related to eval procedures. Caching: Are there opportunities to cache certain computations? For example, if the same prompts are used repeatedly, caching parts of the computation could speed things down. Also, check if the model's parameters are being accessed efficiently—maybe parameter server optimizations are missing. Batch size: Maybe the batch size during eval is smaller than during generation, leading to inefficiencies. Or perhaps the eval requires a different batch handling that's less optimal. I should also consider the specific parts of llama.cpp. For instance, if the eval is using a different tokenizer or preprocessing pipeline than generation, that could add time. Verify that both processes use the same optimized paths. Another thought: some models have separate handling for evaluation, like not using gradient checkpoints or other optimizations that are active during training. If prompt eval is running in a mode that's less optimized, that would explain the slowdown. Check for any explicit loops in the code that could be vectorized or batched. Python loops over examples in eval might be slower than processing them in batches. Also, look at the model's configuration. Is the prompt eval using the same model instance as generation, or is there a separate instance that's less optimized? Sometimes, having redundant model instances can add overhead. Lastly, check for any external dependencies or integrations that might be affecting performance during eval. For example, if the eval process is writing results to a database or file system, I/O bottlenecks could occur. So, putting this all together, the checklist should cover hardware checks (GPU utilization, memory, thermal), code-level optimizations (efficient data handling, avoiding loops, using batched operations), profiling to identify bottlenecks, ensuring optimized model configurations, and checking for any external system issues. **Consolidated Troubleshooting Checklist for Llama.cpp on Single GPU** 1. **Hardware & Resource Checks** - Verify GPU utilization and memory usage during prompt eval vs. generation. - Check for thermal throttling or CPU/GPU contention. - Ensure sufficient GPU memory allocation to prevent OOM errors. 2. **Code-Level Analysis** - **Model Loading**: Confirm model is loaded in optimized mode (e.g., FP16/FP32, mixed precision). - **Data Pipelines**: - Validate tokenizer/preprocessing consistency between eval and gen. - Check for redundant data transfers between CPU/GPU. - **Batched Operations**: Ensure eval uses optimal batch sizes (no sub-optimizations per batch). - **Loop Efficiency**: Replace explicit Python loops with batched tensor operations. 3. **Profiling & Bottlenecks** - Use nvprof/nanopython profiling or PyTorch/CUDA profilers to pinpoint slow kernels (e.g., attention, LM steps). - Disable extraneous logging/debugging code in eval mode. 4. **Algorithmic & Architectural Considerations** - Compare eval mode implementation to gen mode for inefficiencies (e.g., redundant gradient computations). - Optimize attention mechanisms or use sparse implementations if applicable. 5. **System-Level Checks** - Validate disk I/O for cached data access during eval. - Check for synchronization delays (e.g., CPU-GPU overlap). 6. **Configuration & Best Practices** - Ensure model configs (batch size, seq_len) match generation optimizations. - Use CUDA streams for concurrent data loading/generation.eval. - Update framework/optimizations (e.g., cuDNN, PyTorch updates). 7. **External Dependencies** - Confirm no redundant model instances or code paths in eval. - Check for slow integrations (e.g., result logging, external APIs). **Final Answer** Systematically validate hardware resources, optimize data pipelines, profile CUDA kernels, and align eval/generation code paths. Focus on batched tensor operations, mixed-precision, and eliminating CPU-GPU overhead. Update configs to match gen optimizations and use async streams for concurrency.