Okay, so I need to figure out why my local model server is slow, specifically that prompt evaluation is much slower than generation. The user mentioned logs showing this discrepancy, and they want a troubleshooting checklist focused on llama.cpp on a single GPU. Let me start by breaking down the problem. First, I should recall what prompt evaluation and generation entail in the context of models like Phi (assuming Phi is using LLaMA or a similar model). Evaluation usually involves processing the input prompt through the model's encoder or some intermediate steps to get a representation, while generation is producing the output text. The fact that evaluation is slower suggests there's a bottleneck there. Let me think about the steps involved in both processes. For generation, the model might be using a decoder RNN or a transformer decoder, which involves multiple layers of attention and feed-forward networks. Generation could be optimized with techniques like beam search or sampling, which might take time but perhaps not as much as evaluation if the latter involves more complex computations. Prompt evaluation might involve more steps. Maybe it's using a different decoding strategy, like greedy decoding without the optimizations used in generation. Or perhaps it's using a different part of the model, such as an encoder-only approach for evaluation, which could be computationally intensive if the model is large. Alternatively, evaluation might require more tokens to be processed, or there could be additional logging, validation steps, or checks that slow things down. Now, considering the checklist for troubleshooting: 1. **Check GPU Utilization**: Ensure that the GPU is being used efficiently. Maybe the generation is offloaded to the GPU, but evaluation is using CPU or not utilizing GPU properly. Use tools like nvidia-smi to monitor GPU usage. 2. **Batch Sizes**: Evaluate if batch sizes are the same for both generation and evaluation. Maybe generation is using a larger batch to hide latency, but evaluation is done per-example, leading to inefficiency. 3. **Model Parallelism**: Since it's a single GPU, check if the model is split across multiple GPUs. If llama.cpp is incorrectly configured for data parallelism, that could affect performance. Ensure all operations are on the GPU and fused where possible. 4. **Memory Constraints**: Check for memory issues. If evaluation is causing out-of-memory errors, it might be slower due to spilling or delays. Monitor GPU memory usage during both processes. 5. **I/O Bottlenecks**: If prompts are being loaded from disk during evaluation, that could slow things down. Check if there's unnecessary I/O during prompt evaluation. 6. **Code Paths**: Compare the code paths for generation and evaluation in llama.cpp. Maybe evaluation has extra logging, synchronization points, or uses a different implementation that's less optimized. Look for any conditional code that only runs during evaluation. 7. **Attention Mechanisms**: Evaluate if the attention computations in prompt evaluation are more complex. For example, if causal attention is used in generation but not in evaluation, or vice versa. Or if there's a bottleneck in the attention implementation. 8. **Mixed Precision**: Check if mixed precision (like FP16) is used. If generation is using it but evaluation isn't, that could affect speed. Auto-regressive generation often benefits from FP16 for speed. 9. **Kernel Launch Overhead**: Ensure that there are no unnecessary kernel launches. For example, if the evaluation is processing each token sequentially without batching, that could add overhead. 10. **PyTorch/TensorFlow Version**: If using a framework like PyTorch, ensure that the version is up-to-date with optimizations for the model architecture in use. Deprecated functions could slow down execution. 11. **Caching Mechanisms**: Check if there's caching of computations between generation and evaluation. For instance, if some intermediate results are reused during generation but not during evaluation, that could be a point of inefficiency. 12. **Thread Management**: Verify that data loading and GPU computation are properly pipelined. If evaluation is using a different thread setup that's less efficient, that might cause delays. 13. **Profiling**: Use a profiler like nvprof, PyTorch's profiler, or TensorFlow's profiler to pinpoint where the time is being spent in evaluation. Look for hotspots in specific layers or operations. 14. **Activation Management**: Check for redundant activation computations. For example, if gradients are being tracked during evaluation but not needed, that could add overhead. 15. **Model Architecture**: Ensure that the model architecture in llama.cpp is correctly implemented. Any deviations from the intended architecture could lead to performance issues. 16. **Activation Thresholds**: In auto-regressive generation, sometimes there's a threshold for stopping when the probability is low. If evaluation uses a different threshold that's more lenient, it might process more tokens than necessary. 17. **Synchronization Points**: Look for any explicit synchronization points (like CUDA streams or events) that might be introduced in the evaluation code but not in generation. 18. **Logging and Debug Information**: Excessive logging during evaluation could slow things down. Check if the evaluation phase has more verbose logging than generation. 19. **Data Transfer Overhead**: If data is being moved between CPU and GPU during evaluation, that could add latency. Ensure minimal data transfers. 20. **Check for Redundant Operations**: In evaluation, maybe there are operations that aren't needed, like additional tokenization steps or post-processing that's more thorough. 21. **Beam Search vs. Greedy**: If generation uses beam search and evaluation uses greedy, the beam search might be slower. Or vice versa, depending on implementation. 22. **Tokenization Differences**: Ensure that the tokenization process for evaluation isn't more complex or computationally intensive than for generation. 23. **Dynamic Batching**: Check if dynamic batching is enabled for generation but not for evaluation, leading to inefficiencies. 24. **Check for Deprecated Functions**: Ensure that the code isn't using any deprecated functions that have less optimized implementations. 25. **Hardware Issues**: Rule out any hardware problems with the GPU, like overheating or insufficient power, which could cause throttling. So, compiling all these points into a concise checklist, focusing on the most likely culprits: code paths in llama.cpp, GPU utilization, memory usage, attention mechanisms, mixed precision, and I/O operations. The checklist should guide the user through checking each potential bottleneck step by step. **Concise Troubleshooting Checklist for Llama.cpp on Single GPU** 1. **GPU Utilization & Profiling** - Use `nvidia-smi` to confirm GPU usage during both phases. - Profile with PyTorch/CUDA profilers (e.g., `torch.profiler`) to identify bottlenecks in prompt evaluation. 2. **Batch Size Consistency** - Ensure generation and evaluation use identical batch sizes to leverage parallelism. 3. **Code Path Differences** - Compare `llama.cpp` code paths for generation vs. evaluation. Look for: - Extra logging, synchronization, or validation steps in evaluation. - Use of causal vs. non-causal attention (if applicable). - Redundant tokenization/post-processing. 4. **Mixed Precision** - Verify if FP16 (e.g., `torch.cuda.amp`) is enabled in evaluation but not generation. 5. **Memory Usage** - Monitor for out-of-memory (OOM) errors during evaluation. Check for memory leaks or excessive allocations. 6. **Attention & Computation** - Check if evaluation requires more tokens, deeper layers, or complex computations (e.g., reranking). 7. **I/O Bottlenecks** - Ensure prompts are preloaded into GPU memory to avoid disk/CPU transfers during evaluation. 8. **Kernel Overhead** - Optimize loops in evaluation to minimize per-token kernel launches (e.g., batch inference). 9. **Dynamic Batching/Parallelism** - Confirm no unintended model parallelism or thread contention in evaluation. 10. **Activation Management** - Disable gradient tracking (e.g., `torch.no_grad()`) in evaluation to save compute. 11. **Logging Overhead** - Reduce verbosity in evaluation to avoid I/O delays. 12. **Hardware & Thermal Checks** - Ensure GPU is not throttled due to overheating or power limits. **Example Fixes**: - Enable mixed precision in evaluation: ```python with autocast(): output = model(eval_prompt) ``` - Batch evaluation tokens to GPU: ```python outputs = model(eval_prompt_batch) ``` - Remove unnecessary logging/validation in evaluation code. By systematically addressing these areas, you can isolate and resolve the slowdown in prompt evaluation.