A guide to MSP patch management best practices
As software patch management challenges mount, industry experts offer advice to MSPs on prioritizing system risk levels, selecting proper tools and testing patches internally.
Long a part of managed services offerings, software patch management today is fraught with challenges. Applications remain a top cause of external breaches, and researchers are finding more security flaws. As a result, patch management is no longer a perfunctory service but a priority for MSPs.
Nearly 99% of audited codebases contain some amount of open source, and open source software (OSS) vulnerabilities continue to be pervasive, Forrester's "State of Application Security in 2021" reported. Yet firms are unable to react quickly enough to remediate the issues. More than half of respondents to the Forrester survey said that it took a week or longer to remediate known OSS vulnerabilities in their code.
Unpatched vulnerabilities continue to be the most prominent attack vector, according to Srinivas Mukkamala, senior vice president of security products at IT software provider Ivanti, citing a report by his company on ransomware attack patterns. "[O]rganizations," he wrote, "need to prioritize and patch weaponized vulnerabilities based on what's being targeted, whether those vulnerabilities are new or old."
What is MSP patch management?
MSP patch management is the process of making incremental changes to the code within different software applications or programs when a vulnerability or bug is flagged so users aren't at risk. Since managed service providers should be flagging risks on behalf of their customers, it's important for them to know what software updates and patches are available. MSPs should have a process in place to download and apply these patches as well as check to ensure they were installed correctly because some patches might not work with certain operating systems or applications.
MSP patch management has become trickier in recent years, partly due to growing complexities within cloud and on-premises environments and the increased pace of vendor software updates. But the main culprit is the need to update internal software like remote monitoring and management (RMM) software platforms in light of several security incidents, where hackers exploited unpatched systems to deliver malware to service provider customers.
"The whole point is to make sure you're plugging holes in software because most malware creators will take advantage of those vulnerabilities,'' said Gill Langston, senior product manager at cloud directory platform provider JumpCloud and former head security nerd at RMM software maker SolarWinds MSP.
Since users run the risk of clicking on something they shouldn't, patches help ensure an attacker can't exploit a vulnerability. "There could be a fix a customer is waiting for … to improve their ability to work, also," Langston added. "At the end of the day, what's most critical is the time to remediate the vulnerability that the patches are released to fix."
But that's where a lot of MSPs run into challenges, Langston said, because service provider customers balk at doing too many system reboots or having too much downtime. And users often must be available to click OK to reboot their machine.
MSP patch management systems contain a deferment mechanism so administrators can reboot machines after work hours, but some people take their laptops home and more people are working remotely since the onset of the COVID-19 pandemic. When that happens, Langston said, "you have to catch stragglers the next time they start their systems, which is probably during the day when they have to get to work."
Most exploited vulnerabilities are more than six months old, indicating a lot of systems aren't being patched in a timely manner because, Langston reasoned, they are either being overlooked "or there's a hole in an IT admin's process." So MSPs need to deploy patches as quickly and efficiently as possible, even though people don't want to be interrupted while working. "It's been a struggle for [all the] years I've been in business,'' Langston said.
Windows 10 feature packs, which come out about every six months on top of regular patches, are also a headache, said AJ Singh, co-founder and vice president of product management at RMM software provider NinjaOne. Then there are driver updates that must be installed on peripherals, which can create conflicts and cause crashes because of incompatibility due to underlying hardware issues, he added.
MSP patch management tips
MSPs should come up with an efficient patch management process to ensure patches are applied quickly and efficiently and keep organizations secure. Failing to patch software in a timely manner will leave systems vulnerable to attackers, and a business could ultimately experience outages and downtime. Experts offers MSPs several tips, including the following:
- Inventory all devices and assets that comprise the IT infrastructure.
- Prioritize systems to be patched according to their risk level.
- Select a tool that automates schedules and patching. "Any RMM tool will help [MSPs] automate schedules and make patching hands-free," Singh said. "It's up to them how they tweak and hone their model based on their target market."
- Test patches internally on a small subset of devices to verify they're working correctly before deployment. "If after a week, everything is OK, [MSPs] should then feel they can patch their customers' systems," said Oshri Moyal, CTO at RMM software provider Atera.
- Use internal data to evaluate how a computer performs after a particular patch has been applied.
- Monitor patch updates on internal MSP systems.
MSPs can be so focused on supporting their customers that sometimes they forget to look inward at their own tools, Langston noted. "You should be your best customer when it comes to keeping things updated'' since MSPs have become targets, he explained. "So, patch anything you're using, and this prevents you from becoming an entry point."
Langston recommended utilizing sources like the Cybersecurity and Infrastructure Security Agency (CISA), which publishes notifications about vulnerabilities, new patches and what steps to take when a patch can't be deployed right away.
How to choose an MSP patch management product
Automated patch management deployment systems have become an increasingly attractive option because manual patching is no longer realistic. Automated systems also free MSPs from performing mundane manual tasks to focus on more mission-critical areas.
The system should be easy to set up and use. If unclear about what features to select, start with a system offering an intuitive dashboard that can provide detailed information on all endpoints, devices and software in real time. Another good feature provides the ability to document all patch deployments. An automated system should also provide endpoint security and management as well as real-time visibility into patch compliance.
Configurable systems allow MSPs to customize patch deployment and upgrades when needed. A system that offers scanning and scalability lets an admin protect even complex network architectures, and MSPs no longer need to rely on spreadsheets and emails for tracking and assigning patch work. Some systems provide mass deployment of patches and let MSPs prioritize patches by criticality and preapprove or pre-deny patches, Singh said.
What's ahead for patch management?
Patch management is now based on risk and its proliferation. Automated systems are good but no longer enough. Hyperautomation -- the ability to be predictive and proactive in real time -- is the next phase, according to Ivanti's Mukkamala.
MSPs must be able to identify, understand and respond to patterns to keep pace with the level of sophisticated threats. Operational efficiency through hyperautomation can be achieved with AI and machine learning, allowing humans to take action based on the analysis provided, Mukkamala explained. Hyperautomation in patch management, he predicted, will be widespread over the next five years.
Likewise, Langston doesn't see patch management diminishing in importance in the coming years, calling it "one of the pillars of delivering managed services to your customers," along with remote monitoring and applying antivirus software.