Skip to content

/articlesIT operations

Finding an incident’s root cause, with evidence you can check

How AI root cause analysis works in Botify: gather problems, deployments, logs and changes, put them on one timeline, match names, then rank causes with evidence.

Updated
Published
min read
06
On this page
09
On this page09

What can AI do in root cause analysis, and what can it not?

Most of a root cause analysis (RCA) goes on collecting evidence: opening the problem in Dynatrace, finding the deployments, searching Splunk logs, checking ServiceNow changes and working out which server each tool means. Botify does that collection in minutes, and it does not forget to check the second system.

AI cannot prove a cause on its own. The Google SRE book describes troubleshooting as forming a guess from what you see, then testing it until the cause is found. Botify is strong at the first half and useful in the second. The final call, and any test that touches production, stays with an engineer.

So the goal is not an AI that announces the root cause. It is a short, ranked list of possible causes, each with its sources, that an engineer can confirm or rule out quickly.

Step 1: Gather problems, deployments, logs and changes

Start from the symptom and collect four kinds of evidence for the affected service. Each answers a different question. Skipping one is the most common reason an investigation blames the wrong thing.

  • Problems and alerts: what broke, when, and which services are affected. In Dynatrace this is the problem and its root-cause evidence; in Splunk, fired alerts or ITSI notable events.
  • Deployments: what software changed. Deployment events in Dynatrace are the fastest source, if your release pipeline records them.
  • Logs, traces and metrics: what the system actually did. Look for new error types, slow operations and overloaded dependencies.
  • Change records: what people changed outside deployments, such as a firewall rule, a database setting or a certificate approved in ServiceNow.

Step 2: Put everything on one time window

Before comparing anything, pick one window and use it for every query: from shortly before the first symptom (often one to two hours for a sudden failure, longer for a slow decline) to now or to recovery. Every query uses the same start and end, in UTC, so results line up.

Systems also disagree about time. Clocks drift, some tools stamp an event when they receive it rather than when it happened, and ServiceNow changes often show the planned window, not the real one. Allow a few minutes of slack when comparing times, and prefer the real time over the planned one.

Step 3: Match the same service across tools

The same service rarely has the same name everywhere. Dynatrace gives it an entity ID, Splunk a host or source field, and ServiceNow a configuration item with its own sys_id. Until these are matched, Botify cannot know that a change on one item and an error spike on one host are about the same thing.

Keep the matches as data, not guesses. Look the service up by name in each tool, confirm with host names, tags or links, and save confirmed matches so the next investigation starts with them. One wrong match spoils every comparison after it, so a person confirms each new one.

Step 4: Rank likely causes instead of naming one

With evidence on one timeline and names matched, Botify builds a short list of possible causes and ranks them by the same tests an experienced engineer uses. The table shows the result for a sample checkout incident.

  • Timing: the change happened before the first symptom, not after.
  • Reach: the change touched the failing service or something it depends on.
  • Fit: the kind of change matches the kind of failure, such as a connection-pool setting and a spike in connection timeouts.
  • Spread: the affected services match what the change could reach.
  • History: a similar change caused a similar incident before.
Possible causeEvidence forEvidence againstNext check
Release 4.12 of the checkout serviceDeployed 6 minutes before the error spike; new timeout errors in the logsNone found yetCompare error rates on hosts still on 4.11
Database setting changeApproved change in the same window on the database itemDatabase response time stayed flatCheck connection-pool metrics
Outside payment providerSome errors mention the providerErrors started before calls to the provider roseLow priority unless the first two are ruled out

Step 5: Tie every finding to its evidence

An AI-written RCA is useful only if every sentence can be checked. Each finding points to the query that produced it: which system, which window and what came back. Anything the model inferred without a direct source is labelled as a guess, not stated as fact.

This matters in long investigations too. When Botify shortens a large result to keep working, the full result stays available, so the final report quotes the original record, not a summary of a summary.

Common mistakes: clock drift, timing taken as cause and more

AI-assisted RCA makes the same mistakes people make, only faster. Watch for these.

  • Clock drift: comparing exact times across systems can make a cause look like it came after the effect. Allow some slack, and say how much in the report.
  • Timing taken as cause: the SRE book tells of a slowdown that lined up with one database call, until unrelated static-content requests turned out to be slow too. A match in time is a lead to test, not a verdict.
  • Blaming the last deployment: recent releases are easy to find, so they get blamed first. Give infrastructure and settings changes the same weight.
  • No data read as no problem: “no errors in the logs” may mean the wrong index or too narrow a window. Botify should say where it searched.
  • Open-ended searches: a search with no time window or result cap can time out or cost a lot. Limit every query.

How Botify helps with root cause analysis

Botify has four actions built for these steps. One finds the same service, host or application across your connected systems. One saves a confirmed match between names, and always waits for a person to approve. One checks whether two time windows overlap, with slack for clocks that disagree. And one reads back the full evidence collected earlier in the same investigation, so every finding leads to its source.

They work with the read-only actions in Botify’s Dynatrace, Splunk and ServiceNow integrations: problems and deployments, log searches and ITSI notable events, incidents and nearby changes. Botify can then deliver the result as a PDF, Excel, Word or CSV report, in Arabic or English. The IT teams section of the Solutions page shows a sample timeline.

Common questions

01

Can AI find the root cause of an incident automatically?

AI gathers evidence and ranks likely causes much faster than a person, but it cannot prove a cause alone. Treat its output as ranked guesses with evidence, and let an engineer confirm.

02

What data does AI need for incident root cause analysis?

At least the problem or alert, deployment events, logs or traces, and change records for the affected service and time window. Without change records, AI tends to blame the most visible deployment.

03

How do you handle clock drift between monitoring tools?

Use one window in UTC for every query and compare times with a few minutes of slack, not exact timestamps. Prefer when an event happened over when it was planned or received.

04

Is this the same as Dynatrace Davis or AIOps root cause analysis?

No. The root cause analysis built into a monitoring platform looks at that platform’s data. Botify works across platforms and adds change records, logs from other tools and past incidents, often starting from the platform’s own problem as the first clue.

05

How do you check that an AI root cause analysis is right?

Every finding links to the exact query or record behind it. Reviewers check that evidence, look for evidence against the top cause, and run a test that could rule it out before accepting it.

Sources

  1. [1]Google SRE Book, Chapter 12: Effective Troubleshooting

Where you’ll find this in Botify

Start with one assistant on WhatsApp or your website.

Book a demo and we’ll show you an assistant answering questions like your customers’. We set up the first one with you and review the results together before you add more.