Datadog Trained a 9B Model for $0.003 Per Alert Investigation — Here's the Architecture

The standard argument for fine-tuning small models over running large models in production is cost and latency. Datadog’s change attribution model adds a third argument: behavioral alignment. The fine-tuned Qwen3.5-9B doesn’t just cost less — it investigates differently than the large model it learned from, in ways that make it more useful for production incident work.

The Architecture

The task: given a production alert, identify which recent change (deployment, feature flag, configuration update) most likely caused the incident.

Teacher-student framework: GLM-5.3, a large model, generated investigation traces for training. Qwen3.5-9B learned from those traces through supervised fine-tuning. The “rejection sampling” step is important — only traces where the teacher’s conclusion matched extracted proxy labels (drawn from existing Bits Investigation conclusions) were kept for training. This filtered 186 high-quality examples from a larger set, rather than training on all available traces regardless of quality.

LoRA fine-tuning: 16-bit low-rank adaptation on those 186 examples across two cycles (May–June 2026).

Infrastructure: A single NVIDIA A100 40GB GPU supports approximately 100,000 investigations per week at 280 output tokens per second. AWS p4d.24xlarge pricing puts this at roughly $1.74/GPU-hour — translating to $0.003 per investigation.

The Numbers

87% of the teacher’s internal recall at 5% of the cost. The customer recall result (0.62 vs teacher’s likely-higher but unmeasured number) shows the model generalizes beyond the training distribution — customer incidents were not in the training set.

The Behavioral Shift

Fine-tuning didn’t just compress the teacher’s behavior — it changed it. After training:

• Average search calls dropped from 6.8 to 5.8
• Log/span/metric calls increased from 4.7 to 8.3
• Total tokens: 52,900 vs the teacher’s 109,000

The fine-tuned model learned to spend less time casting wide search nets and more time gathering dense evidence once a hypothesis formed. That’s a more expert investigative pattern — one that emerges from seeing how 186 high-quality investigations actually resolved, rather than from general instruction-following.

Integration and Roadmap

The model integrates via MCP into Bits Investigation and Bits Chat. Change Tracking data surfaces through the same interfaces, and 20% of existing Bits Investigation conclusions already reference captured changes — the model has a real signal to work with rather than speculative coverage.

The planned next step is reinforcement learning using Bits Investigation feedback signals as automatic reward mechanisms, which would enable the model to improve from production failures without explicit human labeling.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.