Waxell

MCP Server Evaluation: 5 Checks Before You Connect

Most teams connect MCP servers like npm packages—fast and unvetted. Here are 5 checks to evaluate a server's auth, scope, and tools before you connect.

Logan KellyUpdated 17 min read

Most teams install MCP servers the same way they used to install npm packages: find one that does what you need, copy the config, move on. That was already a risky habit in the npm ecosystem. With MCP, the consequences are different in kind, not just in degree. A bad npm package breaks your build. A bad MCP server runs inside your agents — with your credentials, your data, and your users on the other end.

MCP server evaluation is the process of assessing a third-party server's authentication model, tool definition integrity, permission scope, and governance compatibility before connecting it to your agents. It is a point-in-time judgment made once, at connection time — which is what separates it from MCP governance, the runtime layer that decides what an agent may do with a connected server in every session afterward. Evaluation tells you whether a server is safe to connect. It cannot tell you whether the server is still the same server tomorrow.

Here's how fast that bill comes due. On March 10, 2026, NIST's NVD published CVE-2026-26118, a server-side request forgery flaw in Microsoft's own Azure MCP Server, filed by Microsoft as CNA with a CVSS 3.1 base score of 8.8 (High). NVD's description is one line: SSRF in Azure MCP Server "allows an authorized attacker to elevate privileges over a network" — the vector string requires low privileges, not none. Microsoft's advisory is the place to go for mechanism detail. But the shape of the bug is exactly what pre-connection evaluation is meant to surface: a tool that accepts a user-supplied URL parameter, in a server that holds its own credentials, running inside an agent that holds yours. This was a first-party server from a hyperscaler, documented and patched. The lesson isn't that Microsoft was careless; it's that which tools a server exposes, and what those tools will do with your input, is something you have to look at before you wire it in.

Why MCP Is a Different Kind of Supply Chain Risk

The npm comparison isn't a metaphor. Security researchers watching the MCP ecosystem describe it explicitly: rapid adoption, minimal vetting, growing enterprise dependency, and the same underlying pattern that made npm a regular source of supply chain incidents.

But there's a meaningful difference. When a malicious npm package runs, it runs in a build environment — usually isolated, usually reviewable, with blast radius limited to what the build process touches. When a malicious MCP server runs, it runs inside an agent that's connected to your email, your database, your file system, your GitHub. The agent doesn't know the difference between a tool doing what you expected and a tool doing something it was quietly updated to do.

Early security assessments make the scale concrete, with the caveat that most of them are vendor research rather than peer-reviewed work, and several do not publish a denominator:

  • Equixly's assessment of "some of the most popular MCP server implementations" found command injection flaws in 43% of those tested, path traversal in 22%, and SSRF in 30%. Equixly does not state how many servers that covered.
  • Endor Labs, drawing on its 2025 Dependency Management Report, found that of 2,614 MCP implementations, 82% use file system operations prone to path traversal (CWE-22), 67% use APIs associated with code injection, and 34% use APIs associated with command injection. Note the framing: this is risky API usage, not confirmed exploitability.
  • Astrix Security's audit of 5,205 open-source MCP server repositories found 88% require credentials, 53% rely on long-lived static API keys or personal access tokens, only 8.5% use OAuth, and 79% pass keys through plain environment variables.
  • Enkrypt AI's scan of 1,000 servers reports 32% with at least one critical vulnerability and 5.2 vulnerabilities per server on average. (The post's headline says 33%; its body says 32%. Enkrypt sells an MCP scanner.)
  • Pynt's compositional risk modelling across 281 real-world MCP configurations puts the probability that a system contains an exploitable chain at 36% with two plugins, 52% with three, and 92% with ten. This is modelled risk from combining untrusted-input plugins with privileged-execution plugins, not an observed exploit rate.

The most rigorous look at what is actually reachable came in July 2026. In Exposed by Design — a preprint, not yet peer-reviewed — Nicolás Padilla reports what the paper describes as the first dynamic behavioral security assessment of internet-facing MCP servers: passive discovery across eleven data sources, then active testing with a purpose-built framework covering ten MCP-specific vulnerability classes. Across four measurement runs in July 2026 the study confirmed 640 production MCP servers, dynamically audited 414, and reported 68 vulnerabilities including SQL injection, SSRF against cloud metadata services, prompt template injection, and path traversal. Of the servers audited, 91.8% lacked OAuth authentication, and 687 tool instances across the confirmed set exposed shell execution capabilities with no access controls.

Regulators have noticed. On May 20, 2026 the NSA's Artificial Intelligence Security Center published a Cybersecurity Information Sheet, Model Context Protocol (MCP): Security Design Considerations for AI-Driven Automation, which states that the current protocol specification "requires careful and cautious implementation for security" and names serialization risks, trust boundaries and agent misuse among its concerns. Its central argument is that these are not interface-level problems you can patch at an endpoint — securing MCP means treating the agentic environment as a continuum, because inconsistencies at any stage propagate.

Not every problem requires an attacker. In research Backslash Security published in June 2025, thousands of publicly available locally-executed MCP servers were analyzed and hundreds were found bound to 0.0.0.0 — the "NeighborJack" pattern, which makes a server reachable by anyone on the same local network. Backslash's own illustration is a shared coworking space, not the open internet, and the team was explicit that its analysis "did not yield obviously malicious MCPs." The exposure at internet scale showed up separately in early 2026 with the OpenClaw incident, where large numbers of agent runtimes — many with MCP enabled by default and no authentication — were found reachable on the public internet.

The rug pull attack is the most dramatic version of this failure mode — a server that behaves correctly for weeks, then changes. But the rug pull requires an attacker. Most of the risk in MCP deployments today doesn't require anyone malicious. It requires teams that never evaluated the server they connected.

The Five Checks Before You Connect

This isn't a comprehensive security audit. It's the minimum viable evaluation — the five questions that surface the most risk before you've written a single line of agent code.

1. What's the authentication model?

The first thing to check is how the server handles credentials. Does it require a long-lived static API key? A personal access token? Or OAuth 2.1 with proper scopes and refresh flow?

Astrix found that 53% of open-source MCP servers rely on long-lived static secrets and only 8.5% use OAuth; Padilla's dynamic audit found 91.8% of internet-facing servers running without OAuth at all. That's not any individual server's fault; it reflects where the ecosystem is, and the fact that MCP made authentication optional by design. But it's your call whether to accept that model in your stack. If you're connecting to a server that asks for a broad API key "to get started," ask what it would take to use scoped, rotatable credentials instead. If the answer is "it doesn't support that," weight it accordingly.

2. Can you read the tool definitions before you connect?

MCP servers expose tools to your agent. The tool descriptions — what each tool is named, what parameters it accepts, what it claims to do — are the interface your agent reasons about. If you can't read those descriptions before connecting, you can't know what your agent is being asked to trust.

This matters because tool descriptions are also an attack surface. Tool poisoning embeds malicious instructions directly in tool descriptions, exploiting the fact that agents process them as context — the same class of indirect-injection problem that lets a poisoned input hijack an agent with no CVE to point at. And the parameters a tool accepts matter as much as its description: CVE-2026-26118 was an SSRF reached through a URL parameter on a first-party server. Before you connect any MCP server, open the source, find the tool definitions, and read them — names, descriptions, and the inputs each tool will act on. Not skim — read. If the server is closed-source or the definitions aren't accessible before installation, that's a reason to pause.

3. What permissions does it actually need — and what does it ask for?

Most MCP servers ask for more access than the job requires. This isn't necessarily malicious — it's the path of least resistance at setup time — but it's worth examining. If a documentation server needs read access to your GitHub repos, that's reasonable. If it also requests write access because "some users find it useful," that's scope creep baked into the default config.

Before connecting, map the tasks your agent will actually perform with this server against the permissions it requests. Any gap between what the task needs and what the server asks for is a scope you're accepting by default. The narrower you can make that gap upfront, the less you'll need to fix later. This is the foundational principle of MCP least privilege, and unenforced, it's exactly how agents drift into scope violations — acting outside their intended boundary because nothing structurally stopped them. It starts at evaluation, not after deployment.

4. What's the version history and update behavior?

A server that's been through twenty stable releases with documented changes is a different risk profile than one published three weeks ago by an account with no other activity. Check both.

This is the check where evaluation and governance visibly diverge, and Padilla's study is the sharpest available evidence for why. Across consecutive measurement runs, 41.6% of confirmed production MCP servers disappeared within three days — which the paper reads as rapid deployment cycles without security review. Sit with that number as an operator. An ungoverned agent connected to a server in that population keeps issuing calls against whatever definition the endpoint returns, and the first signal that anything changed is a behavior nobody asked for. A governed one has a recorded baseline of the tool's name, description and input schema, so a changed definition surfaces as a change rather than as a silent new instruction. Neither posture prevents the maintainer from shipping; only one of them tells you they did.

Version history tells you how the maintainer handles changes — whether updates are documented, whether breaking changes are flagged, whether the tool definitions have been stable or volatile. An MCP server that silently updates its tool definitions is exactly the pre-condition for a rug pull. If you can't see a clear history of what changed and when, you have no baseline to detect when something changes after you've connected.

5. What's the multi-server blast radius?

If you're running multiple MCP servers — which most non-trivial agent deployments do — evaluate each one in context of the others. A server with modest permissions on its own may compound significantly with another server that shares authentication context or accesses the same underlying system.

Pynt's compositional modelling above comes from exactly this scenario: each additional plugin widens the attack surface, and a compromise in one can propagate because the adjacent servers were never evaluated for combined risk. Shared credentials make it worse — as the Composio key exposure showed, a single leaked integration-platform credential can return live tokens for every downstream service it fronted. Before connecting a new server, ask: if this server were compromised tomorrow, what else in our current setup would it have reach toward?

Red Flags Worth Stopping For

Some things are blockers. If you see these, the evaluation isn't over — it just became a longer conversation.

The server requires broad OAuth scopes and frames them as necessary for basic functionality. A calendar server that needs access to your entire Google account to read calendar events isn't scoped correctly. If the documentation doesn't explain why a given scope is required, that's the question to ask before connecting.

The server has no mechanism for you to be notified when tool definitions change. This is the structural condition that makes rug pulls possible. If the server doesn't version its tools or notify connected clients of definition changes, you're accepting continuous invisible exposure to whatever the maintainer ships next.

The maintainer is unknown, the package is recent, and the download counts spiked suddenly. This is the typosquatting / name-squatting pattern that the npm ecosystem learned to recognize. The MCP ecosystem is still learning.

There's no security contact, no disclosure policy, and no issue history. Open-source doesn't automatically mean trustworthy, but a project with years of public issue history and an active maintainer has a track record you can read. A project with none of that is asking you to trust without evidence — and the track record matters because disclosure often goes nowhere. Of the maintainers Equixly notified about the flaws it found, it reports that 30% acknowledged and shipped fixes, 45% characterized the risks as theoretical or acceptable, and 25% never responded at all.

Evaluation Is Necessary. It's Not Sufficient.

The five checks above tell you whether a server is safe to connect. They don't tell you whether it stays safe after you've connected it.

Tool definitions change. Credentials rotate — or don't. Servers that were scoped correctly at evaluation drift over time as maintainers add features and teams change. The evaluation gets you to first connection; governance is what carries you through every session after that.

The gap is concrete, not hypothetical. You can read every tool definition on a Monday, confirm the parameters look safe, and connect with confidence — and then the maintainer ships an update on Tuesday that adds a URL parameter behaving exactly like the one in CVE-2026-26118. Nothing about your Monday evaluation tells you Tuesday happened.

What comes after evaluation is a registry — a governed record of which MCP servers are approved, what tools each one is permitted to expose to each agent, and what policies govern those tool calls at runtime. Alongside it, change detection: when a tool definition updates server-side, that change should surface for review rather than pass unremarked because the server still responds. A controlled interface pattern, where tool call boundaries are explicit and validated at runtime, is one of the mechanisms that closes the gap.

Evaluation answers "is this safe to connect?" Governance answers "is this still safe, right now, in this session?"

How Waxell handles this: Waxell's MCP Gateway is one URL per tenant that replaces every upstream MCP config, and it keeps running the parts of this checklist that don't hold still. It fingerprints every tool on every connected upstream — hashed against name, description and input schema — and tracks each through 5 trust states: Pending review, Drift detected, Trusted, Blocked, Removed. New tools arrive as Pending review, surfaced for an admin before they accumulate quiet, load-bearing use. When an already-approved definition changes server-side, the tool moves to Drift detected and the change is recorded and surfaced for review — fingerprinting catches the change, and a policy decides what happens next: add a deny_drift rule (Waxell's starter rule set recommends one everywhere) and the gateway blocks the changed tool at call time until a human re-approves it. A prompt injection scanner runs against tool descriptions at fingerprint time, and each call that traverses the gateway is identity-resolved and policy-checked before the upstream sees it — coverage is what you route through it, so agents holding direct upstream credentials still bypass it. How many upstream connectors any one tenant may connect is set by plan. What the gateway will not do is judge a maintainer's track record, decide which permissions your task actually needs, or reason about your multi-server blast radius. Those stay editorial.

Start free with the Waxell MCP Gateway and point one client at it before you connect anything else. Fingerprint the upstreams you already run, and see which tools you have been calling on trust. Get started →

FAQ

What is MCP server evaluation? MCP server evaluation is the process of assessing a third-party MCP server's authentication model, tool definition transparency, permission requirements, maintainer credibility, and multi-server blast radius before connecting it to your agents. It's the due diligence step between finding a server that does what you need and confirming it's safe to run in your stack. Most teams skip it because there's no immediate failure signal when a server is over-privileged or inadequately maintained — the risk is invisible until it isn't.

How do I vet an MCP server before connecting it? Five checks cover the most common failure modes: verify the authentication model (prefer OAuth 2.1 over long-lived static tokens), read the tool definitions before connecting (not after), audit whether the permissions requested match the permissions the task actually requires, review the version history for how changes are handled, and assess the combined blast radius with other MCP servers already in your environment. These don't replace a full security audit, but they surface the risks that cause the most incidents.

What makes an MCP server trustworthy? Open-source code you can read, tool definitions that are versioned and auditable, an authentication model that supports scoped and rotatable credentials, a maintainer with a verifiable track record, and a clear process for security disclosure and change notification. Trustworthiness isn't binary — it's a risk profile you assess relative to what the server has access to and what your agents will do with it.

What are red flags in an MCP server? Requests for broad OAuth scopes framed as necessary for basic functionality, no versioning or notification mechanism for tool definition changes, an unknown or recently created maintainer account with a sudden download spike, and no security contact or public issue history. Any of these individually warrants a longer look. Multiple of them together is a reason to evaluate alternatives.

Can I just trust a server listed in an official MCP registry? Not on its own. As of 2026 the registry landscape has no single source of truth — different directories use different inclusion criteria, quality filters, and update cadences, and a curated listing tells you a server exists, not that it's correctly scoped for your use case. A listing is a starting point, not a substitute for reading the tool definitions and auditing the permissions yourself. Treat "it's in a registry" the way you'd treat "it's on npm": necessary context, not a security guarantee.

Is evaluating MCP servers the same as having MCP governance? No. Evaluation is the pre-adoption step — it determines whether a server is safe to connect. Governance is the runtime layer — it determines what agents are allowed to do with connected servers in every session after that. A server that passes evaluation can still expose you to risk if tool definitions drift, permissions expand, or your agents operate under policies that were never formally defined. Evaluation and governance are both necessary; they answer different questions.

How do I know if an MCP server changed after I connected it? Only if something recorded what it looked like when you connected. MCP has no built-in change-notification contract, so a maintainer can alter a tool's description or input schema and connected clients will keep calling it with no signal. Padilla's July 2026 measurement study found 41.6% of confirmed production servers disappearing within three days between runs, which gives a rough sense of how volatile that population is. The practical answer most teams land on is fingerprinting: hash each tool's name, description and input schema at approval time and compare on subsequent listings. Waxell's MCP Gateway does this across connected upstreams and moves a changed tool into a Drift state, recorded and surfaced for review; a deny_drift policy rule blocks the changed tool at call time until it's re-approved.

Does every MCP server need to go through this evaluation? Yes, including internal ones. First-party MCP servers you build yourself still need tool definitions reviewed, permission scopes audited, and policies defined for which agents can call them. CVE-2026-26118 is the reminder here: it was a first-party server from Microsoft, and it still shipped an SSRF-prone tool. The failure modes from over-privileged or under-reviewed internal servers are the same as for third-party ones — and internal servers often get less scrutiny precisely because they feel more trusted.

Sources