The SANS Institute says that the highest priority security risk is client software that remains unpatched. That’s right—it’s the highest priority security issue there is. Number one. It trumps (or should trump) all other issues.
So then, why is it that 90% of the time we here at Lithik Systems perform IT security audits, we discover hundreds, thousands, or even tens of thousands of patches missing from Windows, Office, Adobe, Java and other programs? It has been amply publicized that the vast majority of successful security breaches, that is, breaches that result in actual theft of proprietary information, occur through two avenues: visiting infected web sites and opening infected e-mail messages. Yet we routinely run into organizations, even organizations with CIOs and IT security staff, that don’t keep their systems up to date.
Excuses
The typical reaction of our customers, when faced with the evidence of their bad patching habits, is to apologize. “We’d love to be up to date, but it’s just such a huge job,” they say. “It’s our policy to patch, but we’ve been so busy recently that we just haven’t had the time.” Patching systems does take time, a lot of time, because even with automation, patches don’t always stick, and each failed patch means someone has to research the problem and, through trial and error, try to get it fixed. We’ve seen Windows installations that wouldn’t accept certain patches for months or even years until the machine was finally re-imaged, which often means a half of some tech’s day reformatting the broken machine, reinstalling Windows, reinstalling all the desktop and business apps, and bringing the configuration up to corporate standards. It’s a nightmare of a job.
WSUS (Windows Server Update Services), a free tool from Microsoft for deploying Windows and Office patches, is very popular and does a good job of minimizing Internet bandwidth use because, rather than having each desktop computer download each patch individually from Windows Update, WSUS downloads one copy of each patch from Microsoft, then distributes them across the internal LAN. It also comes with some nice deployment and status tools, allowing you to decide which patches should be applied and which should be held back (hint: They should all be applied. Always). But just like a 12-year-old boy, it doesn’t always tell the truth. Why is a mystery, but just be aware of the fact that WSUS lies, then use a patch scanner to give you the real skinny on which patches are actually missing.
Sometimes patches break critical applications. A recent client found that running Adobe Reader version X caused a business application used by a dozen or so of his users to fail, while version 9.1 worked just fine. One or two experiences like that and an IT person becomes gun-shy, never knowing which patch might break something, raining down wrath from irate users that just want to be able to do their jobs. So rather than risk being chewed out, he starts taking the “safe” way out. Safe for him, that is. Dangerously risky for the organization.
Build Your Own Windows Patch Management System
If all you want to do is roll out Windows and Microsoft Office patches, the options are pretty good. Microsoft Baseline Security Analyzer (MBSA) is an excellent patch scanning tool that quickly and accurately reports on missing patches and other security issues in Windows and Microsoft Office. WSUS is also useful, being a good soldier on the front lines, applying patches automatically while keeping your Internet connection from becoming overloaded. But to deal with the inevitable patches that don’t “take”, I like to use something that acts less like a lawnmower and more like a scalpel. WuInstall is a free tool that will install all missing Windows or Office patches on a specific machin or only the patches you specify. Tie this together with a login script or deploy it remotely using a program like PsExec that lets you push a command to a Windows machine across the network and you have a system that, either automatically or manually, cleans up the bits left behind by WSUS.
And servers? Patch them with WuInstall at night, then reboot them with PsExec. Put it in a script and run it with Windows Scheduled Tasks and now your server patches are 100% automated without disrupting business.
Not a bad system. All free software, near-complete automation. Too bad it won’t do Adobe patches.
Third Party Apps
So what can we do about third party desktop apps such as Adobe Reader, Adobe Flash, Adobe Air, Oracle (formerly Sun) Java, or other common desktop programs? WSUS and MBSA don’t deal with them at all. It’s no longer safe to just ignore them, because Adobe won the Number One Hacker Target title from Microsoft in the summer of 2009. Leaving your Adobe Reader installations at versions 7, 8 and 9 could end up costing your company millions of dollars—$6.6 million on average for the typical security breach, to be specific. You could buy patch management software such as Shavlik (the original), Lumension (formerly PatchLink), Secunia or others, but that costs money, and you may not currently have the budget for it. Or you could engage a third party service such as ours to do the whole job, but we charge for our services, too. Isn’t there a cheaper way?
Flying Blind
Here’s one way to patch your third party apps without spending a dime, but it’s not for the squeamish or faint of heart. It involves forcing out third party application updates but without the benefit of a software inventory of exactly what’s installed on each machine. WPKG is an open source system (that is, it’s free) that can be set up to automatically keep each desktop in your organization synchronized with a central repository of patch updates for a wide variety of third party desktop applications. The WPKG software discovers the currently installed version of each program to see if an update is warranted, but I’m not comfortable without a separate monitoring system that tells me if the patch application program is working properly. There are, of course, open source systems out there that will maintain a software inventory for you (OCS Inventory NG comes to mind). But this whole approach is starting to look pretty complicated, especially when you consider that it’s all in addition to the WSUS/MBSA/WuInstall/PsExec system for Windows updates.
The Imaging Solution
There is one clean, clean, clean way to update your machines, but it also has costs as well as some pitfalls. Remote imaging tools such as Symantec Ghost, Acronis, Altiris Deployment Solution, Microsoft RIS (Remote Installation Services) and others allow an IT administrator to make a single “gold copy” of a desktop computer, then push that image out to every desktop in the organization. It’s not as simple as it sounds, though, as similarity of hardware platforms is critically important, as is the amount of bandwidth available between your image server and the target desktop. But when it works, it’s sweet—run Windows Update once to a pristine machine, apply the latest Adobe and Java patches, then use the imaging tool to create a new gold copy on your fileserver. Then push the magic button and all your desktops are fresh and squeaky clean, up to date with Microsoft and all third party apps.
This approach also has a very important hidden benefit—what we at Lithik Systems call “malware shedding.” If a desktop becomes infected with a noisy virus such as those that pop up advertisements for fake antivirus software or certain unmentionable products, you know it right away. Or if a lazy hacker infects a lazy IT administrator’s desktop with an old virus allowed by a missing old patch, the antivirus system is likely to complain quickly and loudly. But what if the hacker is smart and subtle? What if the attack is of the zero-day variety, meaning that no antivirus system out there has ever heard of it, and no patch for the exploited bug has ever been released? Who can possible defend against such an attack? And what if the attack is of the “rootkit” variety that buries itself so deep in the operating system that no antivirus system has a prayer of ever finding it, even if it eventually becomes well-known?
Well, that’s where “malware shedding” comes in. If you re-image your desktops every month, say, on the evening of Patch Tuesday, every rootkit and every bit of malware will be wiped off your network, no matter how tenacious the rootkit is, and no matter how clever the hacker was. They’ll all be gone, because every single Windows desktop has been wiped, reformatted and reinstalled. This feature is extremely valuable.
Imagine also what happens when, some other time during the month, you get one of those calls from a user saying, “My computer keeps popping up ads for designer watches,” or “My computer is acting, well, strange.” “Have no fear,” you say, “take a 20-minute break and I’ll make your computer sparkly clean.” Tell the imaging tool to remotely re-image the user’s computer and the infection vanishes. No diagnosis is necessary; just use the nuclear option on every problem as your first step and you’ll be right 90% of the time.
Other costs of using imaging include the redesign and conversion effort required to move every user’s desktop and My Documents folders over to a nearby fileserver. You should already be doing this anyway so that user files are backed up with your server’s other files, but it’s an absolute prerequisite of the imaging approach. A wipe-and-reinstall will wipe out not only malware, but also any files saved or changes made to the local disk.
“But hey,” you ask, “what about monitoring? What about your software inventory? Isn’t that important?” Well, sure it is, but you only need to check one machine. Go to Control Panel -> Add/Remove Programs and check the versions by hand. On just that one machine. Or go to Secunia.com and run their free Online Software Inspector. It will scan your computer, telling you about any applications that are out of date, whether Microsoft or otherwise. It’s just like a professional patch management solution, except that it’s free and you only have to run it on one machine. Actually, it is a professional patch management solution. A really nice one, too.
A Bit of Common Sense
Now that you know how hackers find and capture their prey, you can come to some conclusions that make life a little less painful. Here are a few “therefores” that we should be able to count on:
- Not all machines are created equal. Given that attackers use web browsing sessions and e-mail sessions to infect computers, a server should never become infected because it should never run a web browsing session and it should never run an e-mail client. Keep that in mind and don’t do those things on your servers. This is important because it is much harder to recover from a server infection than from a desktop computer infection. Much, much harder. It probably means rolling back to a backup, which probably means days of lost work, which may mean lost business.
- Some software purchases should be easier to justify than others. Paying $40 a desktop for permanent imaging system licenses could save you many times that much money in saved staff time, both because of accelerated IT diagnosis and rectification, and because of reduced downtime for the desktop PC user.
- You can’t tell if you’re secure unless you can measure it. Always use a reporting/scanning tool (MBSA, Secunia, etc.) to check your work.
Don’t be like all the others—be the outlier that actually keeps your systems patched. Do this and you’ll also be the outlier that never gets hacked. Well, probably never, anyway.
Comments on this entry are closed.