Build an Observability Knowledge Graph: Netflix's 4-Hour RCA

TL;DR — An observability knowledge graph fixes slow incident RCA by giving your tools one shared vocabulary, so a single query replaces a four-hour tab-hopping session. Netflix does it with an ontology, a catalog-gated Application type, and SHACL-style validation so LLM output becomes deterministic triples. I rebuilt the core in 60 lines of Python: validation caught both bad writes, but the same join took 42 seconds on 360,000 triples, which is where toy graphs stop.

An observability knowledge graph is a store of typed, machine-readable facts about your systems (applications, owners, deployments, alerts, incidents) where every entity follows one schema and every fact is a subject-predicate-object triple, so cross-tool questions become one query.

Why Does Incident RCA Take Four Hours?

Netflix’s engineers describe a normal regression: the home screen starts rendering about 1.5 seconds slower, an alert fires, and the investigation takes roughly four hours. In their Netflix TechBlog write-up the point is that very little of that time went into hard computation. Most of it went into reconstructing relationships their systems already knew but nobody had written down in a form a human or machine could read.

The scale explains why. Netflix reports roughly 2 billion requests on an ordinary day and more than 38 million real-time logging events every second. Metrics, events, logs and traces each answer about a quarter of “what caused this?”, and the answer lives in the joins between them.

The joins are the problem. Atlas knows time series, Radar knows alert history, incident.io knows severities, PagerDuty knows rotations, Spinnaker knows what shipped, and Slack knows who solved something similar last quarter. A thing that is an application in one tool is a service in another and a cluster in a third, with names that differ by a suffix, a casing rule or a rename two people remember.

I think this is the most useful reframe in the post: end-to-end observability is a data modelling problem before it is a tooling problem. More dashboards do not help when the bottleneck is the human who carries the name mapping in their head. If your own incidents stall on “who owns this?” and “what shipped?”, the same logic applies at 100x smaller scale, as I found when writing about telling a traffic anomaly from an outage.

What Is an Observability Knowledge Graph?

The unit is the triple: subject, predicate, object. Netflix’s example encodes six facts about one application and one incident, and then answers “which incidents affected applications owned by Team Bedrock in us-east-1?” in a single query instead of three tools.

Two modelling decisions from the post are worth stealing. First, they refused a generic Service type, because “service” means four things across their tooling. Applications resolve to exactly one type, ops:Application, validated against a software catalog of roughly 49,000 entries before the entity exists. Usernames, team names and scraped dependency names never become applications, and dependency edges are capped in both directions so one popular service cannot dominate the graph.

Second, they modelled human reasoning, not only telemetry. A 200-message, 50-person, 15-service incident channel becomes ordered triples, because the hypotheses tried and the mitigation that worked otherwise evaporate when the channel goes quiet. That pairs well with the idea in building an AI agent knowledge base: durable, structured memory beats re-reading raw history.

How to Build an Observability Knowledge Graph in 5 Steps

This is the smallest version I could build that keeps Netflix’s key properties: one catalog-gated type, typed writes, validation at the boundary, and a cross-domain query.

1. Pick one catalog per entity kind. For applications, that is your service catalog. Nothing becomes an Application unless it is in the catalog.
2. Define the types and the relations you will actually query. Start with Application, Incident, Deployment and three predicates: ownedBy, deployedIn, affects.
3. Write through a gate. Every writer calls one function that checks the catalog and the type before adding a triple.
4. Validate with SHACL after each batch. Shapes encode “an incident must affect at least one application and carry a severity from a closed list”.
5. Query across domains with SPARQL. Prove the graph earns its keep by asking the question that used to span tools.

from rdflib import Graph, Namespace, Literal, RDF
from pyshacl import validate

OPS = Namespace("http://example.org/ops#")
CATALOG = {"api-gateway", "graphql-edge", "home-feed", "playback-api"}
g = Graph()

def app(name):
    if name not in CATALOG:          # the catalog gate: unknown strings never become apps
        return None
    g.add((OPS[name], RDF.type, OPS.Application))
    return OPS[name]

def add_incident(inc, application, severity):
    g.add((OPS[inc], RDF.type, OPS.Incident))
    g.add((OPS[inc], OPS.affects, OPS[application]))
    g.add((OPS[inc], OPS.severity, Literal(severity)))

SHAPES = """
@prefix sh: <http://www.w3.org/ns/shacl#> . @prefix ops: <http://example.org/ops#> .
ops:IncidentShape a sh:NodeShape ; sh:targetClass ops:Incident ;
  sh:property [ sh:path ops:affects ; sh:minCount 1 ; sh:class ops:Application ] ;
  sh:property [ sh:path ops:severity ; sh:minCount 1 ; sh:in ("SEV1" "SEV2" "SEV3") ] .
"""
ok, _, report = validate(g, shacl_graph=Graph().parse(data=SHAPES, format="turtle"))

QUERY = """PREFIX ops: <http://example.org/ops#>
SELECT ?inc ?app WHERE {
  ?inc a ops:Incident ; ops:affects ?app .
  ?app ops:ownedBy "Bedrock" ; ops:deployedIn "us-east-1" .
} ORDER BY ?inc"""

I loaded four applications, three incidents and one deployment, then ran it. Validation passed, and the query returned INC-5377 on api-gateway and INC-5379 on graphql-edge: two incidents, one query, no tool hopping. I then wrote one deliberately bad incident, with severity "catastrophic" pointing at an application called jsmith. SHACL failed validation with two violations: the value was not an ops:Application, and the severity was not in the allowed list. That is the whole trick in miniature: the boundary rejects what the type system forbids.

How Does Netflix Make LLM Output Deterministic?

Building the graph means using tools that are inherently stochastic: LLMs that summarize discussions, fuzzy matching that resolves names, a probabilistic engine that proposes causal chains. Netflix’s answer is to constrain the output space tightly enough that a stochastic step yields a deterministic contract.

Their example is blunt. The model may produce any prose, but its output must fit fixed predicates, for instance a thread category drawn only from incident, question, discussion or decision, and a sentiment from a closed list. The result is a typed triple, not free text. Any single step is non-deterministic, but the chain as a whole gives a graph that answers a SPARQL query the same way every time.

The same idea shows up in grounding AI agent findings in evidence: let the model interpret, but make code verify what is allowed to be recorded. Netflix packages the pattern as enrichers, with a deliberately narrow contract.

The correlation engine sits on top. It reads thirteen signal types, from log errors and metric anomalies to deployments, config changes and experiments. Some ordered pairs, such as a deployment before a log error, carry a high causal prior. Chains are scored on temporal proximity and severity inside a ten-minute window, and anything under 0.6 confidence is discarded.

Netflix is explicit about the role: the engine proposes with evidence and does not conclude. The graph stores the confidence next to the claim, so a 0.87 chain is a strong hypothesis for a human or agent to check. That restraint matters more than the algorithm, and it is the same lesson as auditing an LLM judge: a verdict without its reasons cannot be trusted.

What Breaks When You Scale a Knowledge Graph?

My demo was tiny, so I tested the obvious failure. I generated 20,000 applications and 100,000 incidents, 360,000 triples in total, and ran the same cross-domain join against rdflib’s in-memory store. It returned 142 rows after 42 seconds. That is one run on a cloud container with a naive pattern order and no tuning, so treat it as an order-of-magnitude warning rather than a benchmark.

The lesson is not that graphs are slow. It is that modelling and serving are separate problems. Netflix persists its graph in QuipuDB, an in-house distributed RDF quad store based on redb, and its post notes a full Turtle serialization is archived on every harvest because it is diff-friendly. For your own build:

When Is This Worth Building?

Build it when your incidents stall on assembling context, not on computing anything: finding the owner, reconstructing what shipped, remembering that something similar happened last quarter. Skip it when you have one team and a handful of services, where a shared spreadsheet and a naming convention will beat an ontology.

Netflix’s own eight principles reduce to three I would defend. Model identity first, with one authoritative catalog per entity kind. Treat the ontology as code, versioned and reviewed through pull requests. Constrain every stochastic tool with a type system, and keep humans as the approvers of agent-proposed changes. The authors also admit their ranked hypotheses are sometimes wrong, which is exactly why each one carries its evidence. Measure your own time-to-context before and after, since the post publishes no accuracy numbers.

If you are giving agents more operational responsibility, read when to give AI agents autonomy and the LLM engineering hub next, and keep skill decay in AI-assisted incident response in mind: a graph that hands engineers the picture should not erase their ability to build it.

FAQ

Answers to the most common questions are in the frontmatter FAQ above and mirrored here for readers.

What is an observability knowledge graph? A graph of typed facts about applications, owners, deployments, alerts and incidents, where one query can join data that used to sit in separate tools.

Do I need RDF? No, but SHACL validation and SPARQL joins come free with it, and Netflix chose it for portability across graph stores.

How do I keep an LLM from polluting the graph? Constrain its output to a closed vocabulary, gate identities against a catalog, and validate every write.

Sources

• Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale, Netflix TechBlog
• W3C SHACL specification
• rdflib documentation
• My own demo: Python 3.13, rdflib 7.6.0, pyshacl 0.40.1, run 2026-10-19