Datadog Tap to Parse: Click a Log Field, Get a Grok Parser — No Regex Required

Log parsing has been a quiet friction point in observability for a long time. Most logs arrive as unstructured strings. To query specific fields — an error code buried in a message, a request ID, a duration value — you need to write a Grok pattern, which requires knowing Grok syntax and having the time to test it. Datadog’s Tap to Parse removes that requirement.

What Tap to Parse Does

The workflow: select a log event in Datadog, click on the field in the message you want to extract as a structured attribute, and the system analyzes a sample of similar logs, identifies the pattern, and generates a parsing rule. You review the rule before it’s applied — the generation is AI-assisted, the approval is human.

The output is a working Grok parser, not a suggestion you have to implement yourself. For teams that have avoided parsing specific fields because writing Grok felt like a detour during an incident, this changes the calculus: you can parse on the fly during an investigation rather than scheduling it as a separate task.

Three Points in the Log Lifecycle

Tap to Parse operates at different stages depending on where you need structure:

Log Explorer (Generally Available): Creates calculated fields for the current investigation session. Useful when you encounter an unstructured field during active incident triage — parse it now, use it immediately, without affecting ingestion or future storage.

Log Pipelines (Generally Available): Generates a Grok parser that applies to future ingested logs matching the pattern. This is the path to permanent structured extraction — logs arriving after the rule is added come in already parsed. Appropriate for fields you know you’ll query repeatedly.

Observability Pipelines (Available on Request): Structures logs before they reach Datadog or other downstream destinations. Useful for teams preprocessing high-volume log streams before routing to multiple destinations, where parsing upstream reduces cost at every destination.

The three stages serve different operational needs. Log Explorer helps you move faster during an investigation. Log Pipelines reduces ongoing query complexity. Observability Pipelines reduces cost and improves consistency at the routing layer.

What This Removes From the Developer’s Path

Grok is powerful but specific — it has its own syntax, its own matching primitives, and edge cases that trip up even experienced users when dealing with unusual log formats. The practical effect has been that parsing gets deferred: teams query logs with text search instead of structured filters, or they build the Grok rule eventually but not when they need it most.

Tap to Parse converts a skill-gated task into an approval workflow. The AI does the pattern inference; the developer reviews and confirms. That’s a better division of labor for a feature that’s mostly mechanical pattern matching on sample data.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.