tris.fyi > blog >

Insecure remote management tooling is a policy choice

posted 2026-11-19

Table of Contents

Like many other folks in the information security industry, I've worked at a managed service provider, handling information technology needs for a couple dozen customers at a time. At that organization, we used a remote monitoring and management tool as a critical part of how we got work done. Using our RMM, we'd configure alerting thresholds ("disk almost full," "case temp too high," etc), patching policies, and automated fixes for common tickets. When a customer would call in for help, we'd use our RMM to start a remote screen sharing session, or a built-in remote terminal tool to non-intrusively access the customer's computer and fix the issue. Usage of these tools is pretty much universal in the MSP space; by limiting truck rolls and enabling automation, it's what makes the business model possible in many cases!

RMM tools need "agents" running on customer computers to do their work: no agent means no way to reach in and manage the machine. (There are exceptions; "agentless" RMM tools exist, but they need your computers to stay on networks you can reach into, which is rarely an acceptable tradeoff in the era of remote and hybrid work.) Naturally, these agents run with administrative privileges on the remote machine (what's the point of a remote command line if you can't administer the system with it?). This is all fine so far.

Where do RMM tools get their instructions from? If you're an MSP, hopefully you or your technicians! But your instructions must always be sent through a management server of some sort — your agents aren't connecting back to your techs' computers directly, and this makes certain kinds of automations possible.

And that's the fatal flaw of every (to my knowledge) RMM tool currently in existence. Agents trust that management control plane with their life. If an attacker somehow compromises the management control plane, it's game over; they have admin access on every endpoint you manage. This happened most famously with Kaseya in 2021, but vulnerabilities are being disclosed in RMM control planes more and more frequently, especially as automated tools get better at finding exploitable bugs.

In 2024, ScreenConnect responded to an authentication bypass vulnerability exploited en masse. In 2025, unpatched SimpleHelp servers were compromised to distribute ransomware. Earlier this month, N-able took their central management platform offline for dozens of bugs like this.

MFA doesn't fix this, IP allowlisting doesn't fix this, audit logs don't fix this. Your control plane needs to be Internet-facing in order for a SaaS-delivered RMM tool to work, and these sorts of security mechanisms can only be enforced by the control plane itself. Well-built on-prem RMM tools can mitigate some risk by only exposing an "agent" interface to the Internet, but just by the nature of the toolset, there is always some kind of network path from the Internet to your control plane.

You might think this model is necessary. How can a remote management tool operate without trusting a server? The answer, while complex to actually implement, turns out to be straightforward: we can root agents' trust in the identities of the technicians actually issuing commands through the RMM. The management control plane isn't coming up with things for agents to do all on its own. Even when issuing commands to run scheduled tasks or patching, the control plane is always acting on behalf of a technician. If agents could trust technicians, instead of (or in addition to) the control plane, we could eliminate an entire class of vulnerability, and make it an order of magnitude harder to exploit our RMM tools.

How do we do that? And are there simpler intermediate states that give us a security benefit while being less complex to implement? (Spoiler: yes!)

Modeling what an RMM actually does

Let's first abstractly model what an RMM tool does. In the general case, a Technician issues a Command. To do so, she defines the parameters of her command (let's call them CommandName and CommandArgs), chooses agents to target (call them TargetAgents), and sends this information to a ControlServer. The ControlServer puts the command into a Queue for each agent, likely setting an ExpirationTime for the command (in case an agent is unavailable, etc). When an agent is available to receive commands, it polls the server for its queue and executes the commands. It sends a Report once it's done, hopefully signed so malicious actors can't impersonate the Agent.

If our technician is simply installing a patch non-interactively, for example, it's easy to see how this model works. Let's explore this model in a couple of common RMM use cases that don't obviously meet this pattern:

I'm sure there are cases I've missed, but between establishing interactive sessions and these "set [x] configuration"-type commands, you're hopefully able to see how almost everything our model RMM does can be rooted in a command like this.

Reducing trust in the control plane, piece by piece

Imagine you are an operator of SaaS-delivered RMM software. You've implemented a control plane service that all your agents trust with their lives, as described above. Now, you want to implement two-person safety for administering your control plane service: no one person (or team) operating your SaaS product should be able to issue commands to a customer endpoint on their own. How do we do this? Of course, we have all the usual tricks available (use Shamir's Secret Sharing to split an admin password-type thing), but can we enforce this rule in the design of our RMM software?

Let's introduce a new service called the Notary. It runs on separate hardware from your ControlServer, and a separate team manages it. The Notary has an associated NotaryKey which it holds the private half of. The public half of the NotaryKey is distributed to all your agents somehow (bootstrapped via Web PKI, maybe it's hard-coded in your update packages, etc).

Whenever our technician wants to issue a command, she:

When an agent gets the command from its queue, it independently verifies the signature against the NotaryKey, and refuses to execute the command if it isn't valid. Now, control of just the Notary or just the ControlServer isn't enough to take over an Agent — you need to compromise both services!

This does introduce one tricky problem. What about running scripts previously saved in the control plane? To solve that, first, pass hashes of the scripts in the Command sent to agents. Then, require notarizations when updating scripts in the control plane, too, and save the details of the notarization (including a hash of the script content). When a tech tries to notarize a command to run a script, pass the previous NotarizedCommand to the Notary, and have the Notary refuse if the previous notarization is invalid. This way, trust is preserved even when the control plane can't be trusted to return the same script each time we ask for it. You can use a similar technique to notarize patching policies, etc.

Note that a malicious NotaryKey distributed during bootstrapping could be detected (automatically or manually). You can display a key fingerprint during installation that you ask your techs to verify against. For automated deployment, you can bake the bootstrap key into the installer using a tool that operates independently of the control plane, generate a hash of the installer, and verify it before installing.

You'd probably want to rotate the NotaryKey periodically. A complete implementation of this model would likely allow multiple keys to sign a single command, with agents verifying against any single notarization, plus commands to add/drop notary keys (though see notes below about server trustworthiness in key management).

Putting trust back in our hands

Now imagine you want to make it difficult for any combination of employees operating your RMM service to take over a client endpoint. You open-source your Notary implementation and write up docs, so others can build their own Notary. You allow MSPs running your software to distribute their own NotaryKeys to their endpoints, and adjust your management app to support custom Notary servers.

Now, MSPs can control their own root of trust. Of course, they're now responsible for keeping their Notary server online and patched. But because the Notary can be nearly stateless (it doesn't need to keep its own Queues or anything, it just needs to maintain policy, which changes infrequently enough to look like configuration), they can run multiple instances of it for high availability. And because your Agents never directly talk to the Notary — they only verify its signatures using the public NotaryKey — the Notary can stay behind the MSP's firewall.

Is it a perfect solution? No:

  1. A compromised tech session can still access agents, because it can both request signatures from the Notary and queue commands with the ControlServer.
  2. If customers use the same IdP to authenticate to the notary and control server, and network access to the notary is also gated by the same IdP, a compromise of the IdP is still a compromise of the whole system.
  3. If we have automatic updates for our agents, a compromised vendor signing key still means compromised agents. (But there are mitigations that can help: we can enforce a mandatory waiting period after updates are made available as described above, or we can manually approve updates before they are released.)
  4. Similarly, automatically installing patches means trusting the vendor distributing those patches; this is the same software supply chain problem we've always dealt with, though.
  5. If the control plane is a web app, it can show techs one command, but direct their web browser to sign and submit a different command. There are lots of mitigations for this: have the Notary display the command and request explicit approval on a separate domain before permitting notarization, ship a native desktop app instead of using a web-hosted control plane…

But have we raised the security bar a lot? Absolutely. An attacker who compromises the control plane service — the only thing required to be Internet-facing in the stack, now — can only stop commands from being delivered, replay commands marked Replayable, and read or drop Reports. If they want to do more, they have to exploit point 5 above, or find a pre-notarized templated command they can abuse (template expansion is done at agents, with shell/etc escaping based on a defined type). They can't just steal active sessions or write new commands to the control server database. Instead, they have to actively modify the control plane to achieve their goals, which carries a much higher risk of early discovery.

The Notary can also implement policy like the ControlServer can. For example, you could have an allowlist of CommandNames per technician, or restrict certain TargetAgents to certain techs, at the notary level. This would mean even control of a low-privileged Technician account combined with high-privilege ControlServer access is not enough to compromise an entire MSP's client estate.

Even with an MSP-hosted control plane, the Notary model can still be useful. A Notary can be much simpler (and thus harder to exploit), and again, doesn't have to be exposed to the Internet at all.

Distrusting the notary

We can actually go further than this, but it has a big complexity cost. Imagine you're an MSP that needs even stricter security guarantees…

Give each technician their own TechKey. Put it on a smartcard or similar device (there's an implementation here that works even with cheap FIDO keys; see e.g. the OpenSSH implementation of sk-type keys) so the private half is never accessible to any Internet-connected device.

Whenever a tech issues a command, she signs it with her own TechKey too, to get a SignedCommand. (Hardware designed explicitly for this purpose could even display the command on-device and require a touch to sign, largely mitigating points 1 and 5 above!) Before the Notary notarizes anything, it verifies the TechKey (the set of valid keys doesn't change often, and so could be treated as notary configuration, to keep the statelessness of the notary process). Now we have a SignedNotarizedCommand, and Agents can verify both the TechKey signature and the NotaryKey signature before executing the command.

Automations can use pre-signed SignedCommands submitted to the notary with their own identity. Notary policy can ensure that they're only using the SignedCommands that they should be.

The first time an agent checks in, it gets a notarized command with a list of currently trusted TechKeys. (Or, you can bake in an initially trusted TechKey.) When you hire a tech, or a tech leaves your MSP, you send commands to trust or distrust TechKeys. Agents offline for a while could get snapshots of valid keys (though getting these signed by a valid TechKey itself could get a little complex).

Note that a compromised ControlServer could also drop "distrust key" commands. We can already mitigate this easily by having the Notary keep track of which keys are valid, too, and refusing to notarize if we see a command with an invalid TechKey. If Reports are signed, you can request a list of trusted keys every so often to detect this case, too.

But even so, we've created a system where without a valid TechKey (which can be stored on a secure element, never exposed to the Internet), you can't run malicious commands on customer computers. That's awesome!!

Closing thoughts

Yes, it's nontrivial to implement this sort of architecture. Yes, it has tradeoffs: a self-hosted notary means you have to design your own high-availability and break-glass strategy (though I can imagine a "managed notary" service offering or notary key escrow offered by another vendor, which you'd choose if the availability-security tradeoff made sense); tech-specific keys managed in this way are a pain to handle if lost. No, it's not a be-all/end-all, but it is a significant step up.

My description of this architecture is, of course, incomplete. For example, offboarding techs revokes their TechKeys; a naive implementation of TechKeys would break all their previously scheduled commands. In some cases (disgruntled staff) this could actually be useful! But in the normal case, timestamping commands and notarizations, and including an "effective at" timestamp in the distrust message, fixes this. Commands signed before the effective date, with a recent notarization (from an automation system), can still work. I'm probably missing dozens of cases like this…

But the security properties this type of architecture can give us are much stronger than anything available on the RMM market today. In a time where attackers can find exploitable bugs more easily than ever, and devastate our remote management tools, is it worth it…?

The ideas behind this architecture aren't new at all. Tailscale implements "Tailnet Lock" so you can distrust their control plane. "Notarization" is kind of like OpenSSH certificates in another context, if you squint. Multi-party authorization (notary + control plane, or tech + notary) has been understood since the dawn of time.

So, if you use an RMM toolset, especially something SaaS-delivered, ask your vendor what mitigations they have in place to ensure a control plane vulnerability doesn't become a ransomware vector. Ask them how you can ultimately control agents' root of trust, even when using their cloud-hosted software. If the answers are "none" and "you can't," ask them why! As an industry, we are actively choosing to develop and deploy tools built on a fundamentally flawed model, and we deserve better.