IT operations
Ask Dynatrace and Splunk in one conversation
How Botify works with Dynatrace and Splunk: the questions it answers, the DQL, SPL and ITSI capabilities it uses, and why it stays read-only by default.
Updated · 6 min read
Why connect one assistant to both Dynatrace and Splunk?
Many enterprises run both platforms for good reasons: Dynatrace for application and infrastructure observability, with automatic problem detection and topology, and Splunk for log search, security and long-retention event data. During an incident, engineers end up switching between the two, rewriting the same question in DQL and then in SPL, and matching hosts and services by eye.
Botify removes that switching. It keeps one question and one time window, calls each platform through its API, and brings the results together in one answer. It does not copy your telemetry anywhere; it queries the platforms you already pay for, with the permissions you grant.
What questions can you answer with Dynatrace and Splunk in one place?
The best questions are the ones engineers already ask in incident channels, where the answer sits in two or more tools.
- What problems are open on the payments service right now, and what does Dynatrace say is the root cause?
- Was anything deployed to checkout in the two hours before the latency spike?
- Show error logs from the order-api hosts since 09:30, grouped by error message.
- Which Splunk alerts fired overnight, and do any of them match an open Dynatrace problem?
- Are there ITSI notable events for the same service, and when did they start?
- Which traces through the checkout service are slowest in the last hour, and which downstream call dominates?
Which Dynatrace capabilities does Botify use?
A useful Dynatrace connector exposes the environment as a set of narrow tools instead of one generic API call, so each tool can have its own permissions and risk level. Botify ships 14 Dynatrace tools. Thirteen are read-only and run directly; one writes and always waits for approval.
Dynatrace Query Language (DQL) is the pipeline-based query language for data stored in Grail, and a DQL query is a read-only request. Dynatrace also offers generative AI, historically branded Davis CoPilot and now part of Dynatrace Intelligence, that translates plain-language questions into DQL and explains DQL back in plain language.
| Tool | What it does | Approval |
|---|---|---|
| list_problems | Lists active and resolved problems in a time window | Runs directly |
| get_problem | Fetches one problem’s root-cause evidence and affected entities | Runs directly |
| list_entities | Lists monitored hosts, services, containers or databases matching a selector | Runs directly |
| get_entity | Fetches one entity’s properties, relationships and tags | Runs directly |
| get_entity_type_schema | Returns the schema of an entity type, for valid selectors | Runs directly |
| query_metrics | Queries a timeseries metric over a time window | Runs directly |
| search_logs | Runs a DQL log search against Grail | Runs directly |
| query_traces | Runs a DQL span search against Grail | Runs directly |
| verify_dql | Checks a DQL query for syntax errors without running it | Runs directly |
| davis_nl_to_dql | Turns a plain-language question into DQL with Davis CoPilot | Runs directly |
| davis_explain_dql | Explains a DQL query in plain language with Davis CoPilot | Runs directly |
| list_events | Lists events, including deployment and change events, in a window | Runs directly |
| list_deployments | Lists deployment and change events to correlate with a release | Runs directly |
| ingest_deployment_event | Records a deployment event in Dynatrace | Always waits for approval |
Which Splunk capabilities does Botify use?
Splunk searches are written in the Search Processing Language (SPL). Botify’s Splunk tools in Botify are all read-only, and the search tool refuses to run without a time window and a result cap, so an ambiguous question cannot turn into an expensive all-time search.
| Tool | What it does | Approval |
|---|---|---|
| run_spl_search | Runs an SPL search with a mandatory time window and result cap, and returns the first page | Runs directly |
| get_search_results | Pages through the results of an earlier search job without re-running it | Runs directly |
| list_saved_searches | Lists the saved searches the token can see | Runs directly |
| run_saved_search | Runs an existing saved search by name | Runs directly |
| list_fired_alerts | Lists alerts that fired in a bounded time window | Runs directly |
| list_indexes | Lists the indexes visible to the token | Runs directly |
| get_field_summary | Summarises the fields available in an index | Runs directly |
| list_itsi_notable_events | Lists IT Service Intelligence notable events (ITSI licence required) | Runs directly |
Why read-only by default, and why do deployment events need approval?
Reading from Dynatrace and Splunk cannot break production, so those tools can run without slowing the engineer down. Writing is different even when it looks harmless. A deployment event posted to the Dynatrace Events API v2 becomes part of the timeline Dynatrace and your team use to correlate problems with releases; a wrong one, on the wrong entity or at the wrong time, misleads every later investigation.
That is why the deployment-event tool in Botify always waits for a person to approve the exact event, and why the permission to write events (the events:ingest scope) is not part of the default Dynatrace grant. Splunk tokens are role-based, so the token’s Splunk role decides what it can reach; give it a role that can search only the indexes Botify needs.
Example: investigating a checkout latency spike across both platforms
This is how a single question, “Why did checkout get slow at 14:05?”, can flow through both platforms. Botify chooses each step from the previous result.
- list_problems for 12:00 to now finds an open problem on the checkout service; get_problem returns the affected entities and Dynatrace’s root-cause evidence.
- list_deployments for the same window shows a release to the checkout service at 13:58.
- query_traces with a DQL span query shows that most of the added latency is in calls to the inventory service.
- run_spl_search on the application index, bounded to 13:30 to 14:30, finds a new connection-timeout error from the inventory client starting at 14:04.
- list_itsi_notable_events confirms an ITSI notable event on the inventory service at 14:06, and list_fired_alerts shows the related Splunk alert.
- Botify writes a short summary: the likely cause is the 13:58 release’s inventory client change, with each claim linked to the query that produced it, and a suggested next step for the on-call engineer.
How do you keep AI-written DQL and SPL correct?
Language models write plausible queries that are sometimes wrong in quiet ways: the wrong field name, the wrong index, a filter that returns nothing. Build the safeguards into the tool flow rather than hoping the model is careful.
- Discover before querying: list indexes and summarise fields in Splunk, and fetch the entity type schema in Dynatrace, before writing the query.
- Validate before running: check DQL with verify_dql, and use the explain step when an engineer needs to read what a query does.
- Prefer vetted queries: a saved search your team already trusts is better than a new SPL query written under pressure.
- Bound everything: one time window for every call, and a result cap on every search.
- Show the query: every finding in the answer should include the DQL or SPL that produced it, so an engineer can rerun it in the platform’s own UI.
Frequently asked questions
Can Botify write Splunk SPL and Dynatrace DQL?
Yes. It can write both, and for DQL it can also use Dynatrace’s own natural-language-to-DQL capability. Queries should be bounded by a time window and a result cap, validated where the platform allows it, and shown with the results so engineers can check them.
Does the assistant need write access to Dynatrace or Splunk?
No, not for investigation. Everything in a typical investigation is read-only. Write access is only needed for actions such as recording a deployment event, which should require approval each time.
Does Botify replace Davis AI or Splunk ITSI?
No. Dynatrace’s built-in AI and Splunk ITSI detect problems and group events within their own data. Botify reads their output and correlates it with the other platform, change records and incidents.
What permissions should the Dynatrace and Splunk connections have?
For Dynatrace, grant read scopes for problems, entities, metrics, events, logs and spans, and add the event-ingest scope only if you want deployment events written. For Splunk, use a token whose role can search only the indexes Botify needs.
Can Botify read ITSI notable events without ITSI?
No. Notable events are part of Splunk IT Service Intelligence, so the tool only applies to Splunk deployments with an ITSI licence. Without ITSI, Botify can still use fired alerts, saved searches and SPL searches.