# Short Diagnosis

Your model is likely experiencing token exhaustion during reasoning phases, causing it to get stuck in infinite loops or overly complex inference paths. This is common with local LLMs using llama.cpp behind proxies, where computational resources are limited and token budgets are easily consumed by complex reasoning chains.

# Why This Happens

The core issue stems from how local LLMs process prompts and generate responses. When you present a complex query, the model's attention mechanism can get trapped in recursive reasoning loops, especially when dealing with:

1. **Ambiguous or open-ended prompts** that don't provide clear boundaries for reasoning
2. **Recursive problem-solving** where the model keeps generating new sub-problems instead of answering the main question
3. **Proxy overhead** that adds computational latency, making the model more prone to getting "stuck" in token-consuming reasoning paths
4. **Inadequate stopping criteria** that don't properly detect when the model has completed its reasoning or reached an answer

The llama.cpp backend, while efficient, doesn't have the same memory management and optimization as cloud-based systems. When your model's context window fills up with intermediate reasoning steps, it may begin to loop or fail to terminate properly, consuming all available tokens without producing a final answer.

# Settings To Check

First, examine these critical llama.cpp parameters:

**Context Window Settings:**
- `--ctx-size` (default 2048): Ensure this is sufficient for your prompt + reasoning + answer
- `--rope-freq-base` and `--rope-freq-scale`: These affect attention patterns and can cause instability with complex prompts
- `--memory-fraction`: Set to 0.8-0.9 to leave room for computation

**Generation Parameters:**
- `--temp` (temperature): Set to 0.2-0.4 for more deterministic responses, 0.7+ for creativity
- `--top-p` (nucleus sampling): 0.9 is a good default
- `--top-k`: 40-100 range works well
- `--repeat-penalty`: 1.1-1.3 to prevent repetition
- `--presence-penalty`: 0.0-0.5 to encourage new topics
- `--frequency-penalty`: 0.0-0.5 to discourage repetition

**Stopping Criteria:**
- `--stop` parameter: Define explicit stop sequences like `["\n\n", "\n\n\n"]`
- `--max-tokens`: Set to 512-2048 depending on complexity
- `--ignore-eos`: Set to true if you want to allow longer responses

**Proxy-Related Settings:**
- `--port` and `--host` should be properly configured
- `--timeout` settings to prevent hanging
- `--n-gpu-layers` should be set appropriately for your hardware

# Prompt Design

**Good Prompt Shapes:**
```
"Explain quantum computing in 300 words or less. Focus on the core principles of superposition and entanglement, and provide one practical application."
```

```
"Given the following data points, calculate the compound annual growth rate (CAGR) and explain the formula used:
- Initial value: $10,000
- Final value: $15,000
- Time period: 3 years"
```

```
"Summarize the key points from this technical document in bullet format:
[insert document here]"
```

**Bad Prompt Shapes:**
```
"Tell me everything you know about quantum computing."
```

```
"Explain the concept of quantum computing and then explain it again but with more detail and then again with even more detail."
```

```
"Analyze this complex problem step by step, considering all possible angles, implications, and related concepts."
```

The key difference is specificity. Good prompts provide clear boundaries, explicit output formats, and concrete constraints. Bad prompts invite open-ended exploration that can cause the model to spiral into infinite reasoning loops.

# Benchmark Design

Design your benchmarks with these considerations:

**Test Cases:**
1. **Simple factual questions** (100-200 tokens)
2. **Multi-step reasoning** (300-500 tokens)  
3. **Code generation** (400-800 tokens)
4. **Complex analysis** (600-1200 tokens)

**Metrics to Track:**
- **Token utilization**: % of context window used
- **Response time**: Wall clock time to completion
- **Success rate**: % of prompts that complete with valid answers
- **Answer quality**: Human evaluation of relevance and accuracy
- **Token efficiency**: Tokens per meaningful output word

**Control Variables:**
- Fixed context window size
- Consistent temperature settings
- Standardized proxy configurations
- Equal hardware resources for all tests

**Sample Benchmark Protocol:**
```
Test 1: "What is the capital of France?" (Simple)
Test 2: "Calculate the area of a circle with radius 5 using π = 3.14159" (Multi-step)
Test 3: "Write a Python function to sort an array using merge sort" (Code)
Test 4: "Analyze the impact of climate change on agriculture in developing countries" (Complex)
```

# What The Flight Recorder Should Show

A proper flight recorder for your local LLM should capture:

**Pre-Generation Metrics:**
- Input token count
- Context window utilization percentage
- Prompt complexity score (based on nesting depth, question types)
- Proxy latency measurements
- Hardware resource usage (CPU, GPU, memory)

**Generation Process:**
- Token-by-token generation trace
- Attention pattern changes
- Memory allocation during generation
- Stop sequence detection timing
- Temperature and sampling parameter evolution

**Post-Generation Metrics:**
- Final token count used
- Response quality score
- Time to first token vs. total generation time
- Reasoning depth (measured by recursive sub-questions)
- Success/failure classification

**Example Flight Recorder Output:**
```
Input: "Explain quantum computing in 300 words or less"
Tokens: 120 input, 250 output
Context utilization: 75%
Generation time: 1.2s
Success: Yes
Token efficiency: 0.85
```

# Practical Defaults

For most home-lab scenarios, start with these settings:

**Context Window:**
- `--ctx-size 2048` (minimum 1024 for complex prompts)
- `--memory-fraction 0.85`

**Generation Parameters:**
- `--temp 0.3` (balanced reasoning)
- `--top-p 0.9`
- `--top-k 40`
- `--repeat-penalty 1.1`
- `--presence-penalty 0.2`
- `--frequency-penalty 0.2`

**Stopping Criteria:**
- `--max-tokens 1024`
- `--stop ["\n\n", "\n\n\n"]`
- `--ignore-eos true`

**Proxy Configuration:**
- `--port 8080`
- `--host 0.0.0.0`
- `--timeout 300`

**Hardware Optimization:**
- `--n-gpu-layers 35` (for 13B models on 12GB GPUs)
- `--n-thread 8` (for multi-core CPUs)
- `--n-batch 512` (batch size for inference)

# When To Increase Budgets

Increase your token budgets when:

1. **Complex multi-step reasoning** is required:
   ```
   "Explain the process of photosynthesis step by step, including the chemical equations and the role of chlorophyll in each stage."
   ```

2. **Code generation** with multiple functions:
   ```
   "Write a Python script that implements a binary search tree with insert, delete, and search operations, including error handling."
   ```

3. **Data analysis** with multiple calculations:
   ```
   "Analyze the following sales data and provide: 1) Total revenue, 2) Average monthly sales, 3) Top performing product category, 4) Year-over-year growth rate."
   ```

4. **Creative writing** with structured output:
   ```
   "Write a 500-word short story in the style of Edgar Allan Poe about a detective solving a mysterious disappearance in a Victorian mansion."
   ```

In these cases, increase `--max-tokens` to 2048-4096 and ensure `--ctx-size` is at least 4096.

# When To Stop A Run

Implement these stopping criteria:

**Automatic Termination:**
- If no output generated within 30 seconds
- If token usage exceeds 90% of context window
- If the model generates the same token sequence 3+ times in a row
- If the model enters a recursive loop (detecting repeated sub-problems)
- If the response contains more than 50% of tokens that are stop sequences

**Manual Intervention:**
- When response quality degrades significantly
- When computational resources are maxed out
- When the model appears to be stuck in reasoning loops
- When proxy connections become unstable

**Example Stop Conditions:**
```
# Stop if:
# 1. More than 1000 tokens used
# 2. No output for 60 seconds
# 3. Repeating token pattern detected
# 4. Context window usage exceeds 95%
```

**Monitoring Tools:**
Use tools like `htop`, `nvidia-smi`, or custom logging to monitor resource usage and implement automatic timeouts. Set up alerts when token consumption exceeds expected thresholds, and ensure your proxy configuration has appropriate timeout settings to prevent indefinite hanging.

The key is to balance computational efficiency with response quality, using these parameters to create a stable, predictable environment for your local LLM while maintaining its capability to handle complex tasks.

# When To Stop A Run (continued)

**Implementation Examples:**

For llama.cpp specifically, you can implement these stopping conditions programmatically:

```
# In your Python wrapper script:
import time
import signal

class LLMStopper:
    def __init__(self, max_tokens=2048, timeout=60):
        self.max_tokens = max_tokens
        self.timeout = timeout
        self.start_time = None
        self.token_count = 0
        
    def check_stop_conditions(self):
        if self.token_count > self.max_tokens * 0.9:
            return True, "Token limit reached"
        if time.time() - self.start_time > self.timeout:
            return True, "Timeout reached"
        return False, None

# Usage in generation loop:
stopper = LLMStopper(max_tokens=2048, timeout=30)
while not stopper.check_stop_conditions():
    # Generate tokens
    token = model.generate()
    stopper.token_count += 1
```

**Advanced Monitoring:**
Implement token usage tracking that logs every 100 tokens to detect patterns:
```
# Token usage monitoring
def monitor_tokens(token_count, context_size):
    usage_percent = (token_count / context_size) * 100
    if usage_percent > 80:
        print(f"Warning: {usage_percent:.1f}% context usage")
    elif usage_percent > 90:
        print("Critical: Approaching context limit")
```

**Proxy-Specific Considerations:**
When using proxies, ensure your timeout settings account for network latency:
```
# Proxy timeout configuration
proxy_settings = {
    "request_timeout": 300,  # 5 minutes
    "connection_timeout": 60,  # 1 minute
    "retry_attempts": 3,
    "retry_delay": 5  # seconds
}
```

**Resource Management:**
Monitor GPU memory usage to prevent crashes:
```
# Memory monitoring
import psutil
import GPUtil

def check_resources():
    cpu_percent = psutil.cpu_percent()
    memory_percent = psutil.virtual_memory().percent
    gpus = GPUtil.getGPUs()
    gpu_memory = sum([gpu.memoryUtil for gpu in gpus]) * 100
    
    if cpu_percent > 90 or memory_percent > 90 or gpu_memory > 90:
        return False, "Resource limit exceeded"
    return True, "Resources OK"
```

**Error Recovery:**
Implement graceful degradation when stopping conditions are met:
```
def handle_stop_condition(reason):
    print(f"Stopping generation: {reason}")
    # Save partial results
    # Log the stop condition
    # Return partial response if available
    # Clean up resources
    return {"status": "stopped", "reason": reason, "partial_response": current_response}
```

**Performance Optimization:**
Use these settings to optimize for your specific hardware:
```
# For 12GB GPU:
--n-gpu-layers 35
--ctx-size 2048
--temp 0.3
--max-tokens 1024

# For 24GB GPU:
--n-gpu-layers 40
--ctx-size 4096
--temp 0.2
--max-tokens 2048

# For CPU-only:
--n-gpu-layers 0
--ctx-size 1024
--temp 0.4
--max-tokens 512
```

**Prompt Engineering Best Practices:**

**Structured Prompt Templates:**
```
"Act as a {role} who specializes in {domain}. 
Your task is to {specific_task}.
Provide your response in the following format:
- Main answer: [brief summary]
- Supporting details: [3-5 bullet points]
- Key takeaways: [2-3 bullet points]

Question: {question}"
```

**Constraint-Based Prompts:**
```
"Answer the following question with exactly 150 words. 
Do not use the words 'therefore', 'thus', or 'consequently'. 
Include at least one numerical example. 
End your response with a brief conclusion."

"Generate a technical specification document for a {product}. 
The document must include:
1. Overview section (100 words)
2. Technical requirements (200 words)
3. Implementation steps (150 words)
4. Risk assessment (100 words)
5. Timeline (50 words)
Format: Use clear section headings and bullet points."
```

**Iterative Prompt Refinement:**
```
# Initial prompt
"Explain quantum computing"

# Refined prompt
"Explain quantum computing in simple terms for someone with basic science knowledge. 
Include:
1. Core principles (superposition, entanglement)
2. Key differences from classical computing
3. One practical application
4. Limitations
Keep response under 300 words."

# Further refined
"Explain quantum computing in 200 words maximum. 
Use analogies to help understanding. 
Include: 1) What makes it different from classical computers, 2) One real-world application, 3) Current limitations. 
End with a brief conclusion."
```

**Testing and Validation:**

**Comprehensive Test Suite:**
```
# Test categories
1. Factual accuracy
2. Response length consistency
3. Token efficiency
4. Reasoning depth
5. Error handling
6. Timeout behavior
7. Resource utilization
8. Proxy stability
9. Output format compliance
10. Multi-language support
```

**Validation Metrics:**
```
# Quality scoring system
def evaluate_response(response, prompt, expected_length):
    score = 0
    
    # Length compliance (20%)
    if len(response) > expected_length * 0.8 and len(response) < expected_length * 1.2:
        score += 20
    
    # Relevance (30%)
    # Check if response addresses all key points from prompt
    
    # Completeness (30%)
    # Verify all required elements are present
    
    # Clarity (20%)
    # Assess readability and logical flow
    
    return score
```

**Continuous Improvement:**
Implement feedback loops to improve prompt effectiveness:
```
# Feedback collection
def collect_feedback(prompt, response, user_rating):
    feedback = {
        "prompt": prompt,
        "response": response,
        "rating": user_rating,
        "improvement_suggestions": analyze_response_quality(response),
        "token_usage": calculate_token_efficiency(response),
        "time_to_complete": measure_generation_time()
    }
    
    # Store feedback for future prompt optimization
    store_feedback(feedback)
```

**Hardware-Specific Optimization:**

**GPU Configuration:**
```
# For NVIDIA GPUs:
--n-gpu-layers 35  # Optimal for 12GB VRAM
--tensor-parallel  # Enable for multi-GPU setups
--flash-attn  # Enable if supported

# For AMD GPUs:
--n-gpu-layers 30  # Adjust based on VRAM
--rocm  # Enable ROCm support
--fp16  # Use half-precision for better performance
```

**CPU Optimization:**
```
# For multi-core CPUs:
--n-thread 8  # Match your CPU cores
--n-batch 512  # Optimize batch size
--no-mmap  # Disable memory mapping for better control
--use-mlock  # Lock memory to prevent swapping
```

**Memory Management:**
```
# Memory allocation strategy
def optimize_memory(context_size, available_memory):
    if available_memory < 8 * 1024 * 1024 * 1024:  # < 8GB
        return context_size * 0.5  # Reduce context
    elif available_memory < 16 * 1024 * 1024 * 1024:  # < 16GB
        return context_size * 0.75  # Moderate reduction
    else:
        return context_size  # Full context
```

**Network and Proxy Integration:**

**Load Balancing:**
```
# Multi-proxy setup
proxies = [
    {"host": "proxy1.local", "port": 8080, "weight": 0.4},
    {"host": "proxy2.local", "port": 8081, "weight": 0.3},
    {"host": "proxy3.local", "port": 8082, "weight": 0.3}
]

# Round-robin selection with health checks
def select_proxy():
    healthy_proxies = [p for p in proxies if is_healthy(p)]
    if not healthy_proxies:
        return proxies[0]  # Fallback to first proxy
    return weighted_random_selection(healthy_proxies)
```

**Connection Pooling:**
```
# Connection management
connection_pool = {
    "max_connections": 10,
    "connection_timeout": 30,
    "retry_attempts": 3,
    "retry_delay": 1,
    "keep_alive": True
}

# Reuse connections to reduce overhead
def get_connection():
    # Check pool for available connection
    # Create new if none available
    # Return connection object
    pass
```

**Security Considerations:**

**Input Sanitization:**
```
# Prevent prompt injection
def sanitize_prompt(prompt):
    # Remove dangerous characters
    dangerous_patterns = [
        r'--\s*.*',  # Command line arguments
        r'\b(quit|exit|stop)\b',  # Termination commands
        r'\b(import|exec|eval|open|file)\b',  # Code execution
        r'\b(ssh|telnet|nc|curl|wget)\b',  # Network commands
    ]
    
    for pattern in dangerous_patterns:
        prompt = re.sub(pattern, '[REDACTED]', prompt, flags=re.IGNORECASE)
    
    return prompt
```

**Rate Limiting:**
```
# Prevent abuse
def rate_limit_check(user_id):
    # Check recent request frequency
    # Implement sliding window
    # Return True if within limits
    pass

# Example rate limiting
rate_limits = {
    "requests_per_minute": 60,
    "tokens_per_minute": 10000,
    "concurrent_requests": 5
}
```

**Monitoring and Logging:**

**Comprehensive Logging:**
```
# Log structure
log_entry = {
    "timestamp": datetime.now(),
    "prompt_length": len(prompt),
    "response_length": len(response),
    "tokens_used": token_count,
    "generation_time": time_taken,
    "hardware_stats": {
        "cpu_percent": cpu_percent,
        "memory_percent": memory_percent,
        "gpu_utilization": gpu_utilization,
        "gpu_memory": gpu_memory
    },
    "proxy_stats": {
        "latency": proxy_latency,
        "status": proxy_status,
        "errors": error_count
    },
    "success": True/False,
    "error_message": error_msg if any
}
```

**Alerting System:**
```
# Automated alerts
def check_alert_conditions(log_entry):
    alerts = []
    
    if log_entry["tokens_used"] > log_entry["context_size"] * 0.95:
        alerts.append("Token usage warning")
    
    if log_entry["generation_time"] > 30:
        alerts.append("Slow generation detected")
    
    if log_entry["hardware_stats"]["gpu_utilization"] > 95:
        alerts.append("GPU utilization high")
    
    if log_entry["error_message"]:
        alerts.append("Error occurred")
    
    return alerts
```

**Performance Tuning:**

**Model Selection:**
```
# Model size vs. performance
small_models = ["7B", "13B"]  # Good for home labs
medium_models = ["33B", "65B"]  # Better performance, higher resource usage
large_models = ["70B", "175B"]  # Maximum performance, requires significant resources

# Recommendation based on hardware
def recommend_model(hardware_specs):
    if hardware_specs["gpu_memory"] < 12:
        return "7B or 13B model"
    elif hardware_specs["gpu_memory"] < 24:
        return "13B or 33B model"
    else:
        return "33B or 65B model"
```

**Batch Processing:**
```
# Efficient batch handling
def process_batch(prompts, batch_size=4):
    results = []
    for i in range(0, len(prompts), batch_size):
        batch = prompts[i:i+batch_size]
        # Process batch
        batch_results = process_prompts(batch)
        results.extend(batch_results)
    return results
```

This comprehensive approach to managing local LLMs ensures stable, efficient operation while maintaining the capability to handle complex tasks. The key is to balance resource utilization with performance requirements, implement robust monitoring, and continuously optimize based on real-world usage patterns.