/articlesIT operations
Finding an incident’s root cause, with evidence you can check
How AI root cause analysis works in Botify: gather problems, deployments, logs and changes, put them on one timeline, match names, then rank causes with evidence.
- Updated
- Published
- min read
- 06
- On this page
- 09
On this page09
- What can AI do in root cause analysis, and what can it not?
- Step 1: Gather problems, deployments, logs and changes
- Step 2: Put everything on one time window
- Step 3: Match the same service across tools
- Step 4: Rank likely causes instead of naming one
- Step 5: Tie every finding to its evidence
- Common mistakes: clock drift, timing taken as cause and more
- How Botify helps with root cause analysis
- Common questions
01
What can AI do in root cause analysis, and what can it not?
Most of a root cause analysis (RCA) goes on collecting evidence: opening the problem in Dynatrace, finding the deployments, searching Splunk logs, checking ServiceNow changes and working out which server each tool means. Botify does that collection in minutes, and it does not forget to check the second system.
AI cannot prove a cause on its own. The Google SRE book describes troubleshooting as forming a guess from what you see, then testing it until the cause is found. Botify is strong at the first half and useful in the second. The final call, and any test that touches production, stays with an engineer.
So the goal is not an AI that announces the root cause. It is a short, ranked list of possible causes, each with its sources, that an engineer can confirm or rule out quickly.
02
Step 1: Gather problems, deployments, logs and changes
Start from the symptom and collect four kinds of evidence for the affected service. Each answers a different question. Skipping one is the most common reason an investigation blames the wrong thing.
- Problems and alerts: what broke, when, and which services are affected. In Dynatrace this is the problem and its root-cause evidence; in Splunk, fired alerts or ITSI notable events.
- Deployments: what software changed. Deployment events in Dynatrace are the fastest source, if your release pipeline records them.
- Logs, traces and metrics: what the system actually did. Look for new error types, slow operations and overloaded dependencies.
- Change records: what people changed outside deployments, such as a firewall rule, a database setting or a certificate approved in ServiceNow.
03
Step 2: Put everything on one time window
Before comparing anything, pick one window and use it for every query: from shortly before the first symptom (often one to two hours for a sudden failure, longer for a slow decline) to now or to recovery. Every query uses the same start and end, in UTC, so results line up.
Systems also disagree about time. Clocks drift, some tools stamp an event when they receive it rather than when it happened, and ServiceNow changes often show the planned window, not the real one. Allow a few minutes of slack when comparing times, and prefer the real time over the planned one.
04
Step 3: Match the same service across tools
The same service rarely has the same name everywhere. Dynatrace gives it an entity ID, Splunk a host or source field, and ServiceNow a configuration item with its own sys_id. Until these are matched, Botify cannot know that a change on one item and an error spike on one host are about the same thing.
Keep the matches as data, not guesses. Look the service up by name in each tool, confirm with host names, tags or links, and save confirmed matches so the next investigation starts with them. One wrong match spoils every comparison after it, so a person confirms each new one.
05
Step 4: Rank likely causes instead of naming one
With evidence on one timeline and names matched, Botify builds a short list of possible causes and ranks them by the same tests an experienced engineer uses. The table shows the result for a sample checkout incident.
- Timing: the change happened before the first symptom, not after.
- Reach: the change touched the failing service or something it depends on.
- Fit: the kind of change matches the kind of failure, such as a connection-pool setting and a spike in connection timeouts.
- Spread: the affected services match what the change could reach.
- History: a similar change caused a similar incident before.
| Possible cause | Evidence for | Evidence against | Next check |
|---|---|---|---|
| Release 4.12 of the checkout service | Deployed 6 minutes before the error spike; new timeout errors in the logs | None found yet | Compare error rates on hosts still on 4.11 |
| Database setting change | Approved change in the same window on the database item | Database response time stayed flat | Check connection-pool metrics |
| Outside payment provider | Some errors mention the provider | Errors started before calls to the provider rose | Low priority unless the first two are ruled out |
06
Step 5: Tie every finding to its evidence
An AI-written RCA is useful only if every sentence can be checked. Each finding points to the query that produced it: which system, which window and what came back. Anything the model inferred without a direct source is labelled as a guess, not stated as fact.
This matters in long investigations too. When Botify shortens a large result to keep working, the full result stays available, so the final report quotes the original record, not a summary of a summary.
07
Common mistakes: clock drift, timing taken as cause and more
AI-assisted RCA makes the same mistakes people make, only faster. Watch for these.
- Clock drift: comparing exact times across systems can make a cause look like it came after the effect. Allow some slack, and say how much in the report.
- Timing taken as cause: the SRE book tells of a slowdown that lined up with one database call, until unrelated static-content requests turned out to be slow too. A match in time is a lead to test, not a verdict.
- Blaming the last deployment: recent releases are easy to find, so they get blamed first. Give infrastructure and settings changes the same weight.
- No data read as no problem: “no errors in the logs” may mean the wrong index or too narrow a window. Botify should say where it searched.
- Open-ended searches: a search with no time window or result cap can time out or cost a lot. Limit every query.
08
How Botify helps with root cause analysis
Botify has four actions built for these steps. One finds the same service, host or application across your connected systems. One saves a confirmed match between names, and always waits for a person to approve. One checks whether two time windows overlap, with slack for clocks that disagree. And one reads back the full evidence collected earlier in the same investigation, so every finding leads to its source.
They work with the read-only actions in Botify’s Dynatrace, Splunk and ServiceNow integrations: problems and deployments, log searches and ITSI notable events, incidents and nearby changes. Botify can then deliver the result as a PDF, Excel, Word or CSV report, in Arabic or English. The IT teams section of the Solutions page shows a sample timeline.
09
Common questions
01Can AI find the root cause of an incident automatically?
AI gathers evidence and ranks likely causes much faster than a person, but it cannot prove a cause alone. Treat its output as ranked guesses with evidence, and let an engineer confirm.
02What data does AI need for incident root cause analysis?
At least the problem or alert, deployment events, logs or traces, and change records for the affected service and time window. Without change records, AI tends to blame the most visible deployment.
03How do you handle clock drift between monitoring tools?
Use one window in UTC for every query and compare times with a few minutes of slack, not exact timestamps. Prefer when an event happened over when it was planned or received.
04Is this the same as Dynatrace Davis or AIOps root cause analysis?
No. The root cause analysis built into a monitoring platform looks at that platform’s data. Botify works across platforms and adds change records, logs from other tools and past incidents, often starting from the platform’s own problem as the first clue.
05How do you check that an AI root cause analysis is right?
Every finding links to the exact query or record behind it. Reviewers check that evidence, look for evidence against the top cause, and run a test that could rule it out before accepting it.
10
Sources
11