# Quick Diagnosis  

| What to look for | Why it matters | How to check | Typical signs of trouble |
|------------------|----------------|--------------|--------------------------|
| **First‑request latency** | The first answer is often the slowest because the model must load weights, compile kernels, or prime the cache. | `curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8000/api/v1/chat/completions` (see *Curl Examples* below) | 1‑3 s vs 50 ms for subsequent requests |
| **GPU memory spikes** | A sudden jump indicates a new tensor allocation or a cache flush. | `nvidia-smi --query-gpu=timestamp,utilization.gpu,utilization.memory,memory.used,memory.free --format=csv` | Memory jumps from 4 GB to 8 GB on a single request |
| **CPU/GPU utilisation** | If the GPU sits idle after a request, the model might be waiting on CPU‑side preprocessing or on the network. | `watch -n0.5 nvidia-smi` | GPU utilisation < 10 % for > 1 s |
| **LLama.cpp logs** | Look for “warm‑up”, “tensor‑alloc”, “batch‑size”, “MTP” flags. | `tail -f /var/log/llama.log` | “MTP enabled” vs “MTP disabled” messages |
| **Flight Recorder proxy stats** | The proxy can add latency if it buffers or retries. | `curl http://localhost:9000/metrics` | 10 ms extra latency on every request |
| **SQLite query latency** | If the DB is slow, the server will stall waiting for token counts or usage stats. | `sqlite3 /var/lib/llama.db "EXPLAIN QUERY PLAN SELECT * FROM usage WHERE user_id=1;"` | Query > 50 ms on a small table |

**Quick sanity check**  
1. Run a single request with `curl` and note the latency.  
2. Immediately run `nvidia-smi` and `top` to see GPU/CPU utilisation.  
3. Look at the first few lines of `llama.log`.  
4. If you see a spike in memory or a long “warm‑up” message, you’re likely hitting a cache miss or a MTP boundary.

---

# Data Collection Plan  

Collecting data systematically is the key to diagnosing intermittent behaviour. Below is a step‑by‑step plan that covers all the tools you mentioned: Flight Recorder, ServerTop, SQLite, and the llama.cpp logs.

## 1. Set up a baseline environment

| Tool | Command | Output to capture |
|------|---------|-------------------|
| **Flight Recorder** | `curl http://localhost:9000/metrics` | JSON with `latency_ms`, `throughput_rps`, `error_rate` |
| **ServerTop** | `top -b -n 1 -p $(pidof llama)` | CPU %, MEM, I/O |
| **SQLite** | `sqlite3 /var/lib/llama.db ".schema"` | Table definitions |
| **llama.cpp logs** | `tail -n 20 /var/log/llama.log` | Recent events |

## 2. Create a workload script

```bash
#!/usr/bin/env bash
# run_workload.sh
set -euo pipefail

API_URL="http://localhost:8000/api/v1/chat/completions"
PROMPT="Explain the difference between MTP and non‑MTP in llama.cpp."
TOKEN=1

for i in {1..100}; do
  # Send request
  curl -s -w "@curl-format.txt" -o /dev/null \
    -X POST "$API_URL" \
    -H "Content-Type: application/json" \
    -d '{"model":"llama-7b","prompt":"'"$PROMPT"'","max_tokens":256}'
  # Log GPU stats
  nvidia-smi --query-gpu=timestamp,utilization.gpu,utilization.memory,memory.used,memory.free --format=csv >> gpu_stats.csv
  # Log CPU stats
  top -b -n 1 -p $(pidof llama) >> cpu_stats.txt
  # Log SQLite usage
  sqlite3 /var/lib/llama.db "SELECT * FROM usage WHERE user_id=1 ORDER BY timestamp DESC LIMIT 1;" >> db_usage.txt
  sleep 0.5
done
```

**curl-format.txt**

```
time_namelookup:  %{time_namelookup}\n
time_connect:    %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_pretransfer:%{time_pretransfer}\n
time_redirect:   %{time_redirect}\n
time_starttransfer:%{time_starttransfer}\n
time_total:      %{time_total}\n
```

## 3. Run the script with and without MTP

| Mode | Command |
|------|---------|
| **MTP enabled** | `export LLM_MTP=1; ./run_workload.sh > mtp.log 2>&1` |
| **MTP disabled** | `export LLM_MTP=0; ./run_workload.sh > no_mtp.log 2>&1` |

> **Tip**: Keep the same prompt, batch size, and token limit for both runs to ensure comparability.

## 4. Aggregate the data

```bash
# Parse GPU stats
awk -F, '{print $1","$4","$5}' gpu_stats.csv > gpu_summary.csv
# Parse CPU stats
awk '/%Cpu/{print $2","$3","$4","$5}' cpu_stats.txt > cpu_summary.csv
# Parse SQLite logs
awk -F'|' '{print $1","$2","$3}' db_usage.txt > db_summary.csv
```

## 5. Visualise

Use a lightweight tool like `plotly` in Python or `gnuplot`. Example Python snippet:

```python
import pandas as pd
import plotly.express as px

gpu = pd.read_csv('gpu_summary.csv', names=['time','used','free'])
cpu = pd.read_csv('cpu_summary.csv', names=['user','nice','system'])
db = pd.read_csv('db_summary.csv', names=['timestamp','token_count','latency'])

fig = px.line(gpu, x='time', y='used', title='GPU Memory Usage Over Time')
fig.show()
```

---

# MTP Comparison Plan  

The goal is to isolate the effect of Multi‑Threaded Prompt (MTP) on latency, memory, and throughput. Follow this structured approach.

## 1. Define the metrics

| Metric | Definition | Unit |
|--------|------------|------|
| **First‑request latency** | Time from request sent to first token received. | ms |
| **Average latency** | Mean of all request latencies. | ms |
| **Throughput** | Number of requests processed per second. | rps |
| **GPU memory peak** | Highest memory usage during a request. | MB |
| **CPU utilisation** | Average CPU % over the test period. | % |
| **Error rate** | Number of failed requests / total. | % |
| **Token per second** | Tokens generated per second. | tps |

## 2. Run controlled experiments

| Experiment | Setup | Expected outcome |
|------------|-------|------------------|
| **Baseline** | MTP=0, single request | Low latency, low memory |
| **MTP on** | MTP=1, same request | Slightly higher latency, higher memory |
| **Batching** | MTP=1, batch size 4 | Throughput ↑, latency ↑ |
| **Large prompt** | MTP=1, 10k tokens | Memory spike, latency ↑ |

Run each experiment 30 times to get a distribution.

## 3. Capture data

Use the same script as in the Data Collection Plan, but add a `mode` column:

```bash
echo "mode,latency_ms" >> mtp_comparison.csv
```

## 4. Statistical analysis

```python
import pandas as pd
import scipy.stats as st

df = pd.read_csv('mtp_comparison.csv')
print(df.groupby('mode')['latency_ms'].describe())

# T-test between MTP and non-MTP
t, p = st.ttest_ind(df[df.mode=='MTP']['latency_ms'],
                    df[df.mode=='noMTP']['latency_ms'])
print(f"T={t:.2f}, p={p:.4f}")
```

A p‑value < 0.05 indicates a statistically significant difference.

## 5. Visualise the comparison

```python
import plotly.express as px
fig = px.box(df, x='mode', y='latency_ms', title='Latency by MTP Mode')
fig.show()
```

---

# Reasoning Budget Plan  

When you’re using llama.cpp for different use‑cases (coding, RAG, agentic work, chat, creative writing), the *reasoning budget*—the amount of compute you allocate per request—must be tuned.

| Use‑case | Typical token budget | Reasoning strategy | Notes |
|----------|----------------------|--------------------|-------|
| **Coding** | 256–512 tokens | Focus on syntax, error detection | Keep `max_tokens` low to avoid hallucinations |
| **RAG** | 512–1024 tokens | Retrieval + summarisation | Use `temperature=0` for deterministic answers |
| **Agentic** | 1024–2048 tokens | Multi‑step planning | Enable `stop_sequences` to enforce plan boundaries |
| **Chat** | 256–512 tokens | Conversational flow | Use `top_p=0.9`, `temperature=0.7` |
| **Creative Writing** | 512–1024 tokens | Narrative generation | Use higher `temperature` (0.9–1.0) |

**Practical tip**: start with a *token budget* that is 1.5× the expected output length. If you hit the limit, increase the budget by 25 % and re‑run.

---

# Long Output Test Plan  

Long outputs stress GPU memory and can expose hidden bugs. Use the following steps.

## 1. Define a long prompt

```text
Write a 2000‑word essay on the impact of quantum computing on cryptography, including historical context, technical details, and future outlook. Use citations in APA format.
```

## 2. Run the test

```bash
curl -s -w "@curl-format.txt" -o /dev/null \
  -X POST http://localhost:8000/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-7b","prompt":"'"$LONG_PROMPT"'","max_tokens":4096}'
```

## 3. Monitor

| Metric | Tool | Command |
|--------|------|---------|
| **Memory** | nvidia-smi | `nvidia-smi --query-gpu=memory.used --format=csv` |
| **Latency** | curl | `time_total` from `curl-format.txt` |
| **Token count** | llama.log | `grep "generated tokens" /var/log/llama.log` |
| **CPU** | top | `top -b -n 1 -p $(pidof llama)` |

## 4. Record the results

```bash
echo "long_output,latency_ms,memory_used_mb,tokens_generated" >> long_output.csv
```

## 5. Analyse

- If memory usage > 90 % of GPU capacity, consider reducing `max_tokens` or enabling *paged offloading*.
- If latency > 5 s, check if the model is swapping tensors to CPU.

---

# SQLite Queries  

Your SQLite database holds usage stats, token counts, and possibly user quotas. Below are ready‑to‑run queries for common diagnostics.

## 1. Token usage per user

```sql
SELECT user_id,
       SUM(token_count) AS total_tokens,
       COUNT(*) AS request_count,
       AVG(token_count) AS avg_tokens,
       MAX(token_count) AS max_tokens
FROM usage
GROUP BY user_id
ORDER BY total_tokens DESC;
```

## 2. Latency distribution

```sql
SELECT latency_ms,
       COUNT(*) AS cnt
FROM usage
GROUP BY latency_ms
ORDER BY latency_ms;
```

## 3. Peak memory usage per request

```sql
SELECT request_id,
       MAX(memory_used_mb) AS peak_memory
FROM request_stats
GROUP BY request_id;
```

## 4. Error rate

```sql
SELECT status,
       COUNT(*) AS cnt,
       ROUND(100.0 * COUNT(*) / (SELECT COUNT(*) FROM usage), 2) AS pct
FROM usage
GROUP BY status;
```

## 5. Time‑series of token counts

```sql
SELECT datetime(timestamp, 'unixepoch') AS ts,
       SUM(token_count) AS tokens
FROM usage
WHERE timestamp BETWEEN strftime('%s','now','-1 hour') AND strftime('%s','now')
GROUP BY ts
ORDER BY ts;
```

---

# Dashboard Screenshots To Capture  

When you’re ready to present your findings, capture the following dashboards. They give a holistic view of performance.

| Dashboard | What to capture | Why it matters |
|-----------|-----------------|----------------|
| **Latency Heatmap** | `latency_ms` vs `timestamp` | Spot spikes and patterns |
| **Memory Usage Curve** | `memory_used_mb` over time | Detect leaks or cache misses |
| **Throughput Bar** | `requests_per_second` | Compare MTP vs non‑MTP |
| **Error Rate Gauge** | `error_pct` | Ensure reliability |
| **Token per Second** | `tokens_per_second` | Evaluate model efficiency |
| **CPU/GPU Utilisation** | `cpu_pct`, `gpu_pct` | Check resource utilisation |
| **SQLite Query Latency** | `query_time_ms` | Verify DB performance |

> **Screenshot tip**: Use a high‑resolution capture (≥ 1920×1080) and annotate key points (e.g., “MTP enabled”, “Memory spike”).

---

# Weekend Checklist  

| Day | Task | Time | Notes |
|-----|------|------|-------|
| **Friday** | 1. Install/upgrade llama.cpp, Flight Recorder, SQLite. <br>2. Verify all services start. <br>3. Run baseline test (no MTP). | 2 h | Capture baseline metrics. |
| **Saturday** | 1. Run MTP test (same workload). <br>2. Run long‑output test. <br>3. Collect all logs. | 4 h | Focus on consistency. |
| **Sunday** | 1. Analyse data (Python scripts). <br>2. Generate charts. <br>3. Draft report. | 3 h | Prepare presentation. |

**Daily sanity checks**  
- After each test, run `nvidia-smi` to ensure memory is freed.  
- Check `llama.log` for “OOM” or “tensor‑alloc” errors.  
- Verify SQLite integrity: `sqlite3 /var/lib/llama.db "PRAGMA integrity_check;"`.

---

# Interpreting Results  

1. **Latency vs MTP**  
   - If the first‑request latency increases by < 10 % with MTP, the overhead is negligible.  
   - A > 30 % increase suggests MTP is causing thread contention or cache thrashing.

2. **Memory usage**  
   - A steady memory usage of ~70 % is healthy.  
   - Sudden spikes > 90 % indicate that the model is allocating large tensors (e.g., for a long prompt).  
   - If memory never frees after a request, you have a leak.

3. **Throughput**  
   - Throughput should increase with MTP if the GPU is under‑utilised.  
   - If throughput drops, MTP may be over‑parallelising and causing context switches.

4. **Error rate**  
   - A spike in errors when MTP is enabled points to race conditions or resource exhaustion.

5. **SQLite latency**  
   - If DB queries take > 50 ms, consider indexing `timestamp` or `user_id`.  
   - A slow DB can throttle the entire pipeline.

6. **Use‑case suitability**  
   - **Coding**: Low latency, moderate token budget. If latency > 500 ms, consider a smaller model.  
   - **RAG**: Requires deterministic output; if MTP introduces variability, disable it.  
   - **Agentic**: Needs higher token budgets; ensure memory can handle 2k tokens.  
   - **Chat**: Latency < 300 ms is ideal; if not, reduce `max_tokens`.  
   - **Creative Writing**: Accept higher latency; focus on temperature tuning.

---

# Common Mistakes  

| Mistake | Why it hurts | Fix |
|---------|--------------|-----|
| **Running tests on a busy host** | Background processes skew CPU/GPU metrics. | Use a dedicated VM or container. |
| **Ignoring GPU memory fragmentation** | Fragmentation can cause OOM even if free memory looks high. | Restart the server between tests. |
| **Using the same prompt for all tests** | Some prompts trigger special token patterns that inflate memory. | Use a diverse set of prompts. |
| **Not resetting the cache** | Warm‑up effects persist across runs. | Clear cache: `nvidia-smi --gpu-reset` or restart. |
| **Over‑optimising for latency only** | May sacrifice throughput or model quality. | Balance latency, throughput, and token budget. |
| **Assuming SQLite is fast** | Complex queries can slow down the whole pipeline. | Profile queries with `EXPLAIN QUERY PLAN`. |
| **Not accounting for network latency** | External API calls can mask model latency. | Run tests locally, or mock network. |
| **Using a single metric to decide** | Latency alone doesn’t capture memory or error rate. | Use a composite score: `score = latency + 0.5*memory + 2*error_rate`. |

---

## Final Thought

By following the structured data collection, MTP comparison, and analysis steps above, you’ll be able to pinpoint whether the inconsistencies you’re seeing are due to MTP, GPU memory, or other system factors. The key is to keep the experiments repeatable, capture all relevant metrics, and interpret them in the context of your specific use‑case. Happy benchmarking!

## Advanced GPU Profiling  

While `nvidia-smi` gives you a snapshot, deeper insight comes from NVIDIA’s profiling stack.  
Below is a step‑by‑step guide that will let you see *exactly* where the GPU spends its time, how many warps are idle, and whether memory bandwidth is saturated.

### 1. Install the NVIDIA profiling tools

```bash
sudo apt-get update
sudo apt-get install -y nvprof nvprof-tools
```

> **Tip**: If you’re on a Jetson or a newer RTX, you may want `nsight-systems` instead of `nvprof`. The commands are very similar.

### 2. Run a single request with profiling

```bash
nvprof --log-file=nvprof.log \
  curl -s -w "@curl-format.txt" -o /dev/null \
  -X POST http://localhost:8000/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-7b","prompt":"Explain MTP in llama.cpp","max_tokens":256}'
```

### 3. Inspect the log

```bash
nvprof --print-gpu-trace nvprof.log | less
```

Key sections to look for:

| Section | What it shows | What to look for |
|---------|---------------|------------------|
| **Kernel launch** | Which kernels were executed | Are there many small kernels? |
| **Memory copy** | Host‑to‑device / device‑to‑host transfers | Large transfers can dominate latency |
| **Occupancy** | Warp utilisation | < 30 % occupancy may indicate a bottleneck |
| **Memory bandwidth** | Bytes/sec | If close to the GPU’s theoretical peak, you’re bandwidth‑bound |

### 4. Visualise with Nsight Systems

```bash
nsys profile -o nsys_report \
  --trace=cuda,osrt,api \
  --capture-range=cpu \
  --capture-range-end=cpu \
  --capture-range-scope=process \
  --duration=10 \
  curl -s -w "@curl-format.txt" -o /dev/null \
  -X POST http://localhost:8000/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama-7b","prompt":"Explain MTP in llama.cpp","max_tokens":256}'
```

Open `nsys_report.qdrep` in Nsight Systems.  
Look for:

- **CPU‑to‑GPU latency**: the time between the request arriving and the kernel launch.
- **GPU‑to‑CPU latency**: the time to copy the output back.
- **Kernel execution time**: the actual compute time.

If you see a large gap between the request and the kernel launch, the problem is likely in the *pre‑processing* or *batch scheduling* logic.

---

## CPU Profiling with `perf`

Even if the GPU is the bottleneck, CPU stalls can still cause jitter. Use `perf` to see where the llama.cpp process spends its time.

```bash
sudo perf record -p $(pidof llama) -e cycles,cache-misses,branch-misses -o perf.data
sudo perf report -g
```

Key metrics:

| Metric | What it indicates | Typical threshold |
|--------|-------------------|-------------------|
| **Cycles** | CPU cycles spent in the process | < 1 % of total cycles |
| **Cache‑misses** | L1/L2 cache misses | < 5 % of total accesses |
| **Branch‑misses** | Branch prediction failures | < 1 % of total branches |

If you see a high number of cache misses, consider:

- **Increasing the thread count**: more threads can keep the CPU busy.
- **Adjusting the batch size**: too small batches lead to frequent context switches.
- **Enabling `--numa`**: bind threads to the same NUMA node as the GPU.

---

## Memory Leak Detection with `valgrind`

Occasionally, the llama.cpp process may leak GPU memory or CPU memory. `valgrind` can help detect CPU leaks, while `cuda-memcheck` can detect GPU leaks.

```bash
valgrind --leak-check=full --log-file=valgrind.log \
  ./llama --model-path /models/llama-7b.gguf --port 8000
```

After a few requests, inspect `valgrind.log` for “definitely lost” or “indirectly lost” bytes. A persistent leak indicates a bug in the binding layer.

For GPU leaks:

```bash
cuda-memcheck --log-file=cuda.log \
  ./llama --model-path /models/llama-7b.gguf --port 8000
```

Look for “CUDA_ERROR_OUT_OF_MEMORY” or “CUDA_ERROR_INVALID_VALUE” messages.

---

## Advanced Log Parsing with `jq` and `awk`

Your llama.cpp logs are plain text. Use `jq` to parse JSON payloads and `awk` to aggregate metrics.

```bash
# Extract latency from logs
grep "latency_ms" /var/log/llama.log | awk -F':' '{print $2}' > latencies.txt

# Compute average latency
awk '{sum+=$1} END {print "Avg:", sum/NR}' latencies.txt
```

If you want to correlate GPU memory with latency:

```bash
paste gpu_stats.csv latencies.txt | awk -F',' '{print $1","$4","$5","$7}' > gpu_vs_latency.csv
```

Now you can feed `gpu_vs_latency.csv` into a Python script to plot a scatter plot.

---

## Prometheus + Grafana Integration  

For continuous monitoring, push metrics to Prometheus and visualize them in Grafana.

### 1. Export metrics from Flight Recorder

Create a simple exporter:

```python
# flight_recorder_exporter.py
from flask import Flask, jsonify
import requests

app = Flask(__name__)

@app.route('/metrics')
def metrics():
    resp = requests.get('http://localhost:9000/metrics')
    data = resp.json()
    # Convert to Prometheus format
    out = []
    for k, v in data.items():
        out.append(f'{k} {v}')
    return '\n'.join(out) + '\n'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=9100)
```

Run it:

```bash
python3 flight_recorder_exporter.py
```

### 2. Configure Prometheus

```yaml
scrape_configs:
  - job_name: 'flight_recorder'
    static_configs:
      - targets: ['localhost:9100']
```

### 3. Grafana Dashboards

Create panels for:

- **Latency**: `avg_over_time(flight_recorder_latency_ms[5m])`
- **Throughput**: `rate(flight_recorder_requests_total[1m])`
- **GPU Memory**: `max_over_time(nvidia_smi_memory_used_mb[5m])`
- **CPU Utilisation**: `avg_over_time(cpu_utilisation_percent[5m])`

Add alert rules:

| Alert | Condition | Frequency |
|-------|-----------|-----------|
| **High Latency** | `avg_over_time(flight_recorder_latency_ms[5m]) > 200` | Every 5 min |
| **Memory Spike** | `max_over_time(nvidia_smi_memory_used_mb[5m]) > 90%` | Every 5 min |
| **Error Rate** | `rate(flight_recorder_errors_total[1m]) > 0.01` | Every 5 min |

---

## Advanced MTP Tuning  

MTP (Multi‑Threaded Prompt) is a trade‑off between parallelism and cache locality. Fine‑tuning it can yield significant gains.

| Parameter | Default | Effect | Suggested Range |
|-----------|---------|--------|-----------------|
| `--mtp` | 1 | Enables MTP | 0–1 |
| `--mtp-batch-size` | 4 | Number of prompts processed in parallel | 1–8 |
| `--mtp-thread-count` | 8 | Number of CPU threads used | 4–16 |
| `--mtp-cache-size` | 256 MiB | Size of the per‑thread cache | 128–512 MiB |

### 1. Auto‑tuning script

```bash
#!/usr/bin/env bash
# autotune_mtp.sh
for batch in 1 2 4 8; do
  for threads in 4 8 12 16; do
    echo "Testing batch=$batch, threads=$threads"
    ./llama --model-path /models/llama-7b.gguf \
            --mtp 1 \
            --mtp-batch-size $batch \
            --mtp-thread-count $threads \
            --port 8000 &
    pid=$!
    sleep 5
    # Run a quick request
    curl -s -w "@curl-format.txt" -o /dev/null \
      -X POST http://localhost:8000/api/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{"model":"llama-7b","prompt":"Hello","max_tokens":64}'
    kill $pid
  done
done
```

After running, parse the `curl-format.txt` output to find the configuration that gives the lowest latency while keeping GPU memory < 80 % of total.

### 2. Cache‑size experiments

If you notice that the GPU memory spikes when the prompt length exceeds 4 k tokens, try increasing `--mtp-cache-size` to 512 MiB. Monitor the `nvprof` logs to see if the number of memory copies decreases.

---

## Long‑Output Stress Tests  

Long outputs are a common source of memory churn. Below is a systematic approach to stress‑testing the model for up to 8 k tokens.

| Test | Prompt | Max Tokens | Expected GPU Memory | Expected Latency |
|------|--------|------------|---------------------|------------------|
| **A** | “Explain the history of quantum computing.” | 4 k | 70 % | 1.5 s |
| **B** | “Generate a 2000‑word essay on climate change.” | 8 k | 90 % | 3.5 s |
| **C** | “Write a 500‑line Python script that solves the traveling salesman problem.” | 6 k | 80 % | 2.2 s |

Run each test 10 times, collect `nvidia-smi` and `curl` metrics, and compute the mean and standard deviation. A high variance indicates instability.

---

## Advanced SQLite Tuning  

SQLite is lightweight, but poorly indexed queries can become a bottleneck when you have thousands of requests per second.

### 1. Enable WAL mode

```sql
PRAGMA journal_mode=WAL;
```

WAL allows concurrent reads and writes, reducing lock contention.

### 2. Index the `timestamp` and `user_id` columns

```sql
CREATE INDEX IF NOT EXISTS idx_usage_timestamp ON usage(timestamp);
CREATE INDEX IF NOT EXISTS idx_usage_user ON usage(user_id);
```

### 3. Use `PRAGMA` for performance

```sql
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 10000;  -- 10 k pages (~40 MB on 4 kB pages)
```

### 4. Batch inserts

If you’re inserting many rows per request, batch them:

```sql
BEGIN TRANSACTION;
INSERT INTO usage (user_id, token_count, latency_ms, timestamp) VALUES (1, 256, 120, strftime('%s','now'));
-- repeat
COMMIT;
```

### 5. Vacuum periodically

```sql
VACUUM;
```

This rebuilds the database and defragments it, improving read speed.

---

## Use‑Case‑Specific Guidance  

| Use‑Case | Recommended Model | Token Budget | Temperature | Prompt Engineering Tips |
|----------|-------------------|--------------|-------------|------------------------|
| **Coding** | Llama‑7B (quantized 4‑bit) | 256–512 | 0.3 | Use “You are a senior software engineer” style prompt. |
| **RAG** | Llama‑13B (quantized 4‑bit) | 512–1024 | 0.0 | Append retrieved passages verbatim; use “Answer the following question based on the context.” |
| **Agentic** | Llama‑30B (quantized 4‑bit) | 1024–2048 | 0.5 | Use a step‑by‑step plan prompt; add `stop_sequences=["END"]`. |
| **Chat** | Llama‑7B (quantized 4‑bit) | 256–512 | 0.7 | Keep context < 2 k tokens; use “You are a friendly assistant.” |
| **Creative Writing** | Llama‑13B (quantized 4‑bit) | 512–1024 | 0.9 | Use “Write a short story about…”; allow higher temperature for more variety. |

**Why quantization matters**  
Quantizing to 4‑bit reduces memory usage by ~75 % and speeds up inference on GPUs that support INT4 kernels. However, it can increase the variance in latency due to the need for dequantization. Test both 4‑bit and 8‑bit to see which gives the best trade‑off for your use‑case.

---

## Advanced Weekend Checklist (Extended)  

| Time | Task | Tool | Expected Output |
|------|------|------|-----------------|
| **Friday 18:00** | Install dependencies (llama.cpp, nvidia‑profiler, Prometheus, Grafana) | `apt`, `pip` | All services running |
| **Friday 19:00** | Baseline run (no MTP) | `run_workload.sh` | `baseline.log` |
| **Friday 20:00** | MTP run (batch = 4, threads = 8) | `run_workload.sh` | `mtp.log` |
| **Friday 21:00** | Long‑output test A | `curl` + `nvidia-smi` | `long_a.csv` |
| **Friday 22:00** | Long‑output test B | `curl` + `nvidia-smi` | `long_b.csv` |
| **Saturday 08:00** | Auto‑tune MTP | `autotune_mtp.sh` | `mtp_tuning.csv` |
| **Saturday 10:00** | Prometheus scrape test | `curl http://localhost:9100/metrics` | `prometheus_metrics.txt` |
| **Saturday 12:00** | Grafana dashboard review | Web UI | Screenshots of latency, memory, throughput |
| **Saturday 14:00** | SQLite tuning | `sqlite3` | `sqlite_tuning.sql` |
| **Saturday 16:00** | CPU profiling | `perf` | `perf.data` |
| **Saturday 18:00** | GPU profiling | `nvprof` | `nvprof.log` |
| **Sunday 08:00** | Analyze data with Python | `pandas`, `plotly` | `analysis.ipynb` |
| **Sunday 10:00** | Draft report | Markdown | `report.md` |
| **Sunday 12:00** | Review alert rules | Grafana | Confirm thresholds |
| **Sunday 14:00** | Final sanity check | `curl` + `nvidia-smi` | Confirm no spikes |
| **Sunday 16:00** | Archive logs | `tar czf` | `archive.tar.gz` |

---

## Interpreting Results (Deep Dive)  

### 1. Latency vs. Throughput Trade‑off  

| Scenario | Latency | Throughput | Recommendation |
|----------|---------|------------|----------------|
| **Low latency** (≤ 200 ms) | ✅ | ❌ | Use for chat or coding assistants. |
| **Balanced** (200–500 ms) | ✅ | ✅ | Good for RAG or agentic tasks. |
| **High latency** (≥ 500 ms) | ❌ | ✅ | Acceptable for creative writing or batch jobs. |

If you see a *U‑shaped* latency curve when increasing batch size, the GPU is saturating. Reduce batch size or increase `--mtp-thread-count`.

### 2. Memory Utilisation Patterns  

- **Steady 70–80 %**: Good, the GPU is fully utilized without hitting the limit.  
- **Fluctuating 60–90 %**: Likely due to varying prompt lengths. Consider normalising prompt length or using a *prompt cache*.  
- **Spikes > 95 %**: Indicates a memory leak or an unusually long prompt. Check `nvprof` for large memory copies.

### 3. CPU Utilisation  

- **< 20 %**: CPU is idle; you might be able to increase batch size.  
- **20–50 %**: Balanced; CPU is not a bottleneck.  
- **> 70 %**: CPU is saturated; consider increasing `--mtp-thread-count` or moving to a higher‑core CPU.

### 4. Error Rate  

A sudden jump in error rate often correlates with GPU memory exhaustion. If you see `CUDA_ERROR_OUT_OF_MEMORY`, reduce `max_tokens` or enable `--offload` to CPU.

### 5. Prometheus Alert Patterns  

| Alert | What it means | Action |
|-------|---------------|--------|
| **High Latency** | Average latency > 200 ms | Check MTP settings, GPU utilisation. |
| **Memory Spike** | Memory > 90 % | Restart server, check for leaks. |
| **Error Rate** | > 1 % errors | Inspect logs, check GPU health. |

---

## Common Mistakes (Extended)  

| Mistake | Why it hurts | How to avoid |
|---------|---------------|--------------|
| **Running tests on a shared host** | Background processes skew CPU/GPU metrics. | Use a dedicated VM or container with `--cpuset-cpus` and `--memory` limits. |
| **Ignoring NUMA effects** | Threads may cross NUMA nodes, causing memory latency. | Bind threads to the same NUMA node as the GPU: `numactl --cpunodebind=0 --membind=0 ./llama`. |
| **Using default `max_tokens` for all use‑cases** | Some tasks need fewer tokens, leading to wasted GPU time. | Tailor `max_tokens` to the use‑case. |
| **Not resetting the cache** | Warm‑up effects persist, making results inconsistent. | Restart the server between runs or use `nvidia-smi --gpu-reset`. |
| **Over‑optimising for latency only** | Ignoring throughput can lead to under‑utilised GPU. | Use a composite score: `score = latency + 0.5*memory + 2*error_rate`. |
| **Assuming SQLite is always fast** | Complex queries can lock the DB. | Use WAL mode, proper indexes, and batch writes. |
| **Not monitoring GPU temperature** | Over‑clocking can cause throttling. | Add `temperature.gpu` to Prometheus and set alerts > 85 °C. |
| **Using 8‑bit quantization on a GPU that only supports 4‑bit** | Incompatibility leads to slower inference. | Verify GPU compute capability and kernel support. |
| **Ignoring the impact of `stop_sequences`** | Long prompts without proper stop tokens can cause runaway generation. | Always set a reasonable `stop_sequences` for safety. |
| **Not validating the model after quantization** | Quantization can introduce errors. | Run a validation set and compare perplexity. |

---

## Sample Python Analysis Notebook (`analysis.ipynb`)  

```python
import pandas as pd
import plotly.express as px

# Load data
latency = pd.read_csv('baseline.log', sep='\t', names=['time','latency_ms'])
gpu = pd.read_csv('gpu_stats.csv', names=['time','util','mem_used','mem_free'])
db = pd.read_csv('db_usage.csv', names=['timestamp','token_count','latency_ms'])

# Merge on time
merged = pd.merge_asof(latency, gpu, on='time', direction='nearest')
merged['timestamp'] = pd.to_datetime(merged['time'], unit='s')

# Latency vs GPU memory
fig = px.scatter(merged, x='mem_used', y='latency_ms',
                 title='Latency vs GPU Memory Usage',
                 labels={'mem_used':'GPU Memory Used (MiB)','latency_ms':'Latency (ms)'})
fig.show()

# Throughput over time
throughput = latency.resample('1T', on='time').size()
fig2 = px.line(throughput, title='Throughput (requests per minute)')
fig2.show()

# DB token distribution
fig3 = px.histogram(db, x='token_count', nbins=50,
                    title='Token Count Distribution')
fig3.show()
```

---

## Final Checklist for Production Deployment  

1. **Model Validation**  
   - Run a sanity test set (e.g., 100 prompts) and compute perplexity.  
   - Verify that the output matches the reference for a subset of prompts.

2. **Resource Limits**  
   - Set `ulimit -n 65536` for file descriptors.  
   - Use `cgroups` to limit CPU and memory usage.

3. **Health Checks**  
   - Expose `/health` endpoint that returns `200 OK` if the model is loaded.  
   - Add a `/metrics` endpoint for Prometheus.

4. **Logging**  
   - Rotate logs daily (`logrotate`).  
   - Store logs in a central location (e.g., ELK stack) for long‑term analysis.

5. **Security**  
   - Run the server behind a reverse proxy (NGINX) with TLS.  
   - Use API keys or JWT for authentication.

6. **Backup**  
   - Snapshot the model weights and SQLite DB weekly.  
   - Store backups in an off‑site location.

7. **Scaling**  
   - If throughput > 100 rps, consider sharding the model across multiple GPUs or using a model server like Triton.  
   - Use a load balancer (HAProxy) to distribute traffic.

---

## Summary of Key Takeaways  

- **Consistent latency** is a symptom of cache misses, thread contention, or GPU memory fragmentation.  
- **MTP** can improve throughput but may increase latency if not tuned.  
- **GPU profiling** (`nvprof`, `nsight`) reveals whether the kernel launch or memory copy is the bottleneck.  
- **CPU profiling** (`perf`) helps identify stalls that cause jitter.  
- **SQLite tuning** (WAL, indexes, batch writes) is essential when you have high request rates.  
- **Prometheus + Grafana** give you real‑time visibility and alerting.  
- **Use‑case‑specific settings** (token budget, temperature, prompt style) are critical for delivering the right experience.  
- **Avoid common pitfalls** such as shared hosts, ignoring NUMA, or over‑optimising for latency alone.

With the data collection plan, MTP comparison plan, and the detailed profiling steps above, you should be able to pinpoint the root cause of the inconsistencies you’re seeing and decide whether MTP is truly beneficial for your workload. Happy benchmarking!