Back to Blog

Your Observability Stack Just Became an Attack Surface

GrafanaGhost chains three bypasses to silently exfiltrate enterprise data through Grafana's AI assistant. No credentials, no trace, no SIEM alerts. Here's why traditional tools miss it entirely.

Reversed telescope leaking data representing observability tools weaponized for exfiltration

Grafana is the platform your engineering team trusts with everything. Real-time financial metrics. Infrastructure health. Customer activity logs. Private operational telemetry. For most organizations, Grafana is not just a dashboard. It is the most complete, live picture of how your systems work and what they contain.

On April 7, Noma Security disclosed a vulnerability they call GrafanaGhost. The attack uses indirect prompt injection to exfiltrate that data silently, without login credentials, and without leaving any trace in the audit logs your security team monitors.

The mechanisms are not novel. The combination is.

What GrafanaGhost Actually Does

The attack chains three bypasses to move data from inside a Grafana environment to an attacker-controlled server, using Grafana's own AI assistant as the transport mechanism.

Bypass one: defeat domain validation. Grafana restricts outbound requests to approved domains. The attacker formats a URL in a way that Grafana's security check reads as internal, while the browser treats it as a request to an external server. The gap between what the validator expects and what actually executes is the opening.

Bypass two: disable AI guardrails. When researchers first tried to inject malicious instructions into the AI model, it recognized the pattern and refused. After studying how the model processed different input types, they found a specific keyword that caused it to override its own safety logic and treat the injection as a legitimate request.

Bypass three: exfiltrate via image tag. With the domain check and the guardrails bypassed, the AI initiates an outbound request to the attacker's server. The sensitive data rides along in an image tag embedded in the HTTP request. The AI made the call. To any observer, it looks like routine AI behavior.

No credentials needed to start the attack. No user needs to click a link. No error messages surface. No access-denied entries appear in logs. From the inside, nothing happened.

Why Traditional Tools Don't Catch This

The attack is invisible to conventional defenses by design.

Vulnerability scanners test for known CVEs and patch levels. GrafanaGhost is not a missing patch. The AI model's behavior is the vulnerability.

SIEM rules fire on anomalous events: unexpected logins, privilege escalation, unusual network destinations. The exfiltration in GrafanaGhost is initiated by the AI assistant itself, through channels and calling patterns indistinguishable from its normal operation.

DLP tools monitor for specific data patterns leaving known endpoints. They are not designed to inspect whether an AI's outbound network request was triggered by a user prompt or by an injected instruction embedded in a log file.

Sasi Levi, vulnerability research lead at Noma Labs, described the gap directly: "Without runtime protection that understands AI-specific behavior, monitoring what the model was asked, what it retrieved, and what actions it took, this attack would be effectively invisible."

This is not a gap in the implementation of existing security tools. It is a gap in the threat models those tools were built to address.

The Structural Problem Underneath

Grafana patched GrafanaGhost. But the underlying issue is larger than one fix.

AI features are being integrated into enterprise platforms at a pace that outstrips the corresponding security work. Platforms add AI because users want it. The AI gets access to whatever data the platform already holds. The trust model for that data, built when AI was not in scope, does not get revisited.

In Grafana's case, the platform was designed to ingest and display data from a wide range of sources. That design assumption, that ingested data is content to render rather than instructions to evaluate, is exactly what indirect prompt injection exploits. The AI processes everything it receives as potential input. An injected instruction in a log file is indistinguishable from a legitimate user request.

Grafana is not unique in this exposure. Any enterprise platform that added AI features as a layer on top of existing architecture, gave the AI access to user-generated or externally-sourced data, and assumed existing trust controls were sufficient for the new capability, is carrying this same class of risk.

The list is long: observability tools, customer success platforms, support ticket systems, code review tools, CRM products. Anywhere an AI reads data that someone outside the organization can influence.

Grafana's Disputed Assessment

Grafana Labs disputes portions of Noma's characterization. The company's CISO stated that any successful execution of the exploit would have required significant user interaction, specifically that the end user would have to repeatedly instruct the AI assistant to follow malicious instructions after being warned.

Noma's published research describes a bypass path that removes the need for the user to manually override the model's warnings. The dispute centers on whether that bypass functioned in production or only in a research environment. What both sides agree on: the vulnerability existed, Grafana fixed it, and no data was leaked from Grafana Cloud.

The disagreement is worth noting, but it does not change the structural lesson. Whether this specific attack required one user action or zero, the category of vulnerability is real. Indirect prompt injection through enterprise AI features is not theoretical.

What To Do

The response operates at two levels.

For Grafana specifically: Update to the patched version. Review which data sources the AI assistant can read, and audit whether any of those sources contain user-controlled or externally-influenced content. If log data from external sources is accessible to the AI, treat it as a potential injection surface.

For the broader pattern:

  • Audit every AI-integrated platform in your stack. For each one, ask: what data can the AI read? Can any of that data be influenced by parties outside the organization? If yes, what prevents that data from being treated as instructions?

  • Do not assume AI features inherit the security properties of the platform they run on. They do not. They introduce a new class of trust boundary that the original platform was not designed to enforce.

  • Build a threat model for AI-native attacks. The frameworks are still early, but the core questions are the same as any other: who can put data into this system, what can the system do with that data, and where can that data go?

The Bigger Picture

What makes GrafanaGhost worth understanding beyond the specific fix is what it reveals about where enterprise security is heading.

Traditional application vulnerabilities live in the gap between expected and actual behavior of code. AI-native attacks live in the gap between expected and actual behavior of the AI reasoning layer. The application code may be perfectly implemented. The AI's response to a carefully crafted input may still leak everything behind it.

Security teams are well-equipped to defend the first surface. Most organizations have no tooling, no detection rules, and no playbooks for the second.

The AI features in your enterprise stack are not passive. They read data, take actions, and make outbound calls. When an attacker can influence what they read, they can influence what they do.

The question worth asking now, before the next GrafanaGhost, is not whether your AI features are useful. It is whether you know what they trust.