So then, why is it that 90% of the time we here at Lithik Systems perform IT security audits, we discover hundreds, thousands, or even tens of thousands of patches missing from Windows, Office, Adobe, Java and other programs? It has been amply publicized that the vast majority of successful security breaches, that is, breaches that result in actual theft of proprietary information, occur through two avenues: visiting infected web sites and opening infected e-mail messages. Yet we routinely run into organizations, even organizations with CIOs and IT security staff, that don’t keep their systems up to date.

Excuses

The typical reaction of our customers, when faced with the evidence of their bad patching habits, is to apologize. “We’d love to be up to date, but it’s just such a huge job,” they say. “It’s our policy to patch, but we’ve been so busy recently that we just haven’t had the time.” Patching systems does take time, a lot of time, because even with automation, patches don’t always stick, and each failed patch means someone has to research the problem and, through trial and error, try to get it fixed. We’ve seen Windows installations that wouldn’t accept certain patches for months or even years until the machine was finally re-imaged, which often means a half of some tech’s day reformatting the broken machine, reinstalling Windows, reinstalling all the desktop and business apps, and bringing the configuration up to corporate standards. It’s a nightmare of a job.

WSUS (Windows Server Update Services), a free tool from Microsoft for deploying Windows and Office patches, is very popular and does a good job of minimizing Internet bandwidth use because, rather than having each desktop computer download each patch individually from Windows Update, WSUS downloads one copy of each patch from Microsoft, then distributes them across the internal LAN. It also comes with some nice deployment and status tools, allowing you to decide which patches should be applied and which should be held back (hint: They should all be applied. Always). But just like a 12-year-old boy, it doesn’t always tell the truth. Why is a mystery, but just be aware of the fact that WSUS lies, then use a patch scanner to give you the real skinny on which patches are actually missing.

Sometimes patches break critical applications. A recent client found that running Adobe Reader version X caused a business application used by a dozen or so of his users to fail, while version 9.1 worked just fine. One or two experiences like that and an IT person becomes gun-shy, never knowing which patch might break something, raining down wrath from irate users that just want to be able to do their jobs. So rather than risk being chewed out, he starts taking the “safe” way out. Safe for him, that is. Dangerously risky for the organization.

Build Your Own Windows Patch Management System

If all you want to do is roll out Windows and Microsoft Office patches, the options are pretty good. Microsoft Baseline Security Analyzer (MBSA) is an excellent patch scanning tool that quickly and accurately reports on missing patches and other security issues in Windows and Microsoft Office. WSUS is also useful, being a good soldier on the front lines, applying patches automatically while keeping your Internet connection from becoming overloaded. But to deal with the inevitable patches that don’t “take”, I like to use something that acts less like a lawnmower and more like a scalpel. WuInstall is a free tool that will install all missing Windows or Office patches on a specific machin or only the patches you specify. Tie this together with a login script or deploy it remotely using a program like PsExec that lets you push a command to a Windows machine across the network and you have a system that, either automatically or manually, cleans up the bits left behind by WSUS.

And servers? Patch them with WuInstall at night, then reboot them with PsExec. Put it in a script and run it with Windows Scheduled Tasks and now your server patches are 100% automated without disrupting business.

Not a bad system. All free software, near-complete automation. Too bad it won’t do Adobe patches.

Third Party Apps

So what can we do about third party desktop apps such as Adobe Reader, Adobe Flash, Adobe Air, Oracle (formerly Sun) Java, or other common desktop programs? WSUS and MBSA don’t deal with them at all. It’s no longer safe to just ignore them, because Adobe won the Number One Hacker Target title from Microsoft in the summer of 2009. Leaving your Adobe Reader installations at versions 7, 8 and 9 could end up costing your company millions of dollars—$6.6 million on average for the typical security breach, to be specific. You could buy patch management software such as Shavlik (the original), Lumension (formerly PatchLink), Secunia or others, but that costs money, and you may not currently have the budget for it. Or you could engage a third party service such as ours to do the whole job, but we charge for our services, too. Isn’t there a cheaper way?

Flying Blind

Here’s one way to patch your third party apps without spending a dime, but it’s not for the squeamish or faint of heart. It involves forcing out third party application updates but without the benefit of a software inventory of exactly what’s installed on each machine. WPKG is an open source system (that is, it’s free) that can be set up to automatically keep each desktop in your organization synchronized with a central repository of patch updates for a wide variety of third party desktop applications. The WPKG software discovers the currently installed version of each program to see if an update is warranted, but I’m not comfortable without a separate monitoring system that tells me if the patch application program is working properly. There are, of course, open source systems out there that will maintain a software inventory for you (OCS Inventory NG comes to mind). But this whole approach is starting to look pretty complicated, especially when you consider that it’s all in addition to the WSUS/MBSA/WuInstall/PsExec system for Windows updates.

The Imaging Solution

There is one clean, clean, clean way to update your machines, but it also has costs as well as some pitfalls. Remote imaging tools such as Symantec Ghost, Acronis, Altiris Deployment Solution, Microsoft RIS (Remote Installation Services) and others allow an IT administrator to make a single “gold copy” of a desktop computer, then push that image out to every desktop in the organization. It’s not as simple as it sounds, though, as similarity of hardware platforms is critically important, as is the amount of bandwidth available between your image server and the target desktop. But when it works, it’s sweet—run Windows Update once to a pristine machine, apply the latest Adobe and Java patches, then use the imaging tool to create a new gold copy on your fileserver. Then push the magic button and all your desktops are fresh and squeaky clean, up to date with Microsoft and all third party apps.

This approach also has a very important hidden benefit—what we at Lithik Systems call “malware shedding.” If a desktop becomes infected with a noisy virus such as those that pop up advertisements for fake antivirus software or certain unmentionable products, you know it right away. Or if a lazy hacker infects a lazy IT administrator’s desktop with an old virus allowed by a missing old patch, the antivirus system is likely to complain quickly and loudly. But what if the hacker is smart and subtle? What if the attack is of the zero-day variety, meaning that no antivirus system out there has ever heard of it, and no patch for the exploited bug has ever been released? Who can possible defend against such an attack? And what if the attack is of the “rootkit” variety that buries itself so deep in the operating system that no antivirus system has a prayer of ever finding it, even if it eventually becomes well-known?

Well, that’s where “malware shedding” comes in. If you re-image your desktops every month, say, on the evening of Patch Tuesday, every rootkit and every bit of malware will be wiped off your network, no matter how tenacious the rootkit is, and no matter how clever the hacker was. They’ll all be gone, because every single Windows desktop has been wiped, reformatted and reinstalled. This feature is extremely valuable.

Imagine also what happens when, some other time during the month, you get one of those calls from a user saying, “My computer keeps popping up ads for designer watches,” or “My computer is acting, well, strange.” “Have no fear,” you say, “take a 20-minute break and I’ll make your computer sparkly clean.” Tell the imaging tool to remotely re-image the user’s computer and the infection vanishes. No diagnosis is necessary; just use the nuclear option on every problem as your first step and you’ll be right 90% of the time.

Other costs of using imaging include the redesign and conversion effort required to move every user’s desktop and My Documents folders over to a nearby fileserver. You should already be doing this anyway so that user files are backed up with your server’s other files, but it’s an absolute prerequisite of the imaging approach. A wipe-and-reinstall will wipe out not only malware, but also any files saved or changes made to the local disk.

“But hey,” you ask, “what about monitoring?  What about your software inventory?  Isn’t that important?” Well, sure it is, but you only need to check one machine.  Go to Control Panel -> Add/Remove Programs and check the versions by hand. On just that one machine.  Or go to Secunia.com and run their free Online Software Inspector. It will scan your computer, telling you about any applications that are out of date, whether Microsoft or otherwise. It’s just like a professional patch management solution, except that it’s free and you only have to run it on one machine. Actually, it is a professional patch management solution. A really nice one, too.

A Bit of Common Sense

Now that you know how hackers find and capture their prey, you can come to some conclusions that make life a little less painful. Here are a few “therefores” that we should be able to count on:

• Not all machines are created equal. Given that attackers use web browsing sessions and e-mail sessions to infect computers, a server should never become infected because it should never run a web browsing session and it should never run an e-mail client. Keep that in mind and don’t do those things on your servers. This is important because it is much harder to recover from a server infection than from a desktop computer infection.  Much, much harder. It probably means rolling back to a backup, which probably means days of lost work, which may mean lost business.
• Some software purchases should be easier to justify than others. Paying $40 a desktop for permanent imaging system licenses could save you many times that much money in saved staff time, both because of accelerated IT diagnosis and rectification, and because of reduced downtime for the desktop PC user.
• You can’t tell if you’re secure unless you can measure it. Always use a reporting/scanning tool (MBSA, Secunia, etc.) to check your work.

Don’t be like all the others—be the outlier that actually keeps your systems patched. Do this and you’ll also be the outlier that never gets hacked. Well, probably never, anyway.

If the idea of configuring Windows Group Policy makes you break out in a sweat, be afraid no longer. We’ll go step-by-step through the process so that even a relative neophyte can get good results.

GPMC

GPMC (Group Policy Management Console) is an excellent tool for configuring and managing Group Policy. If you don’t already have it installed, you can get it from here. Fire it up by clicking Start, selecting Run…, then typing gpmc.msc and clicking OK.

Navigate through the domain tree in the left pane to Forest (for your domain), Domains, then open your domain, then open Group Policy Objects. You should find at least two GPOs (Group Policy Objects) there, among them being Default Domain Controllers Policy and Default Domain Policy. Click on one of these policies to open it in the right pane. Select the Settings tab to display the rules that make up the policy. Click on show all in the upper right corner to expand the rule hierarchy. Look through these policies to see if you already have rules governing Automatic Updates. If so, you can modify them in place, but we will proceed assuming a clean slate.

Create GPOs

Right-click on Group Policy Objects in the left pane and select New. Assign a name like Automatic Updates-Desktops and click OK. Now right-click your new policy in the left pane and select Edit. This will start up the Group Policy Object Editor.

In the left pane, drill down to Computer Configuration -> Administrative Templates -> Windows Components -> Windows Update. All Automatic Updates settings will appear in the right pane. As you click on each one, a description of the item will appear adjacent to the list. We won’t set every value, but we will set most of them. Open a setting by double-clicking its entry to open a configuration window. Each configuration window also has an Explain tab to help you understand the meaning of each aspect of the setting.

Let’s walk through the first one. Double click Configure Automatic Updates. Select the Enabled radio button. Set the first pull-down to 4 – Auto download and schedule the install. Set the install day to 0 – Every day. Set the scheduled install time to whatever you prefer. Finally, click OK to save the settings and close the box.

Here’s how I typically set it up:

• Configure Automatic Updates
  • Enabled
  • Configure automatic updating: 4 – Auto download and schedule the install
  • Scheduled install day: 0 – Every day
  • Scheduled install time: 11:00
• Specify intranet Microsoft update service location
  • Disabled
• Reschedule Automatic Updates scheduled installations
  • Enabled
  • Wait after system startup (minutes): 5
• No auto-restart with logged on users for scheduled automatic updates installations
  • Disabled
• Automatic Updates detection frequency
  • Enabled
  • Check for updates at the following interval (hours): 4
• Allow Automatic Updates immediate installation
  • Enabled
• Delay Restart for scheduled installations
  • Enabled
  • Wait the following period before proceeding with a scheduled restart (minutes): 5
• Re-prompt for restart with scheduled installations
  • Enabled
  • Wait the following period before prompting again with a scheduled restart (minutes): 480
• Allow non-administrators to receive update notifications
  • Enabled
• Enabling Windows Update Power Management to automatically wake up the system to install scheduled updates
  • Enabled

A few notes on the above values: You’ve got to let Automatic Updates reboot the computer; time and time again I’ve seen installations go literally months without applying patches because a user doesn’t want to be bothered with maintaining his machine. That’s OK; we’ll do it for him. Don’t check for updates more often than 3 or 4 hours; Microsoft says it can cause problems, and I, for one, don’t need any more problems with Windows than I already have. The 480 minute re-prompt number works out to 8 hours; I figure if someone actually doing their job sees a notice, they don’t need to be nagged again. Let it update once they’ve gone home. The power management setting is mostly wishful thinking on my part. Maybe someday I’ll go around and set up Wake-on-LAN in the BIOS on all those machines.

That was a lot of work, but there’s still plenty to do. Create a nearly identical GPO called Automatic Updates-Servers. The only change I make is to set the Configure Automatic Updates value to 3 – Auto download and notify for install. This prevents the possibility of a server rebooting while in use, disrupting work flow and generally wreaking havoc, but it also prevents the server from staying up to date unless you put forth an effort to keep it so. Hint: A particularly easy way to complete 90% of the work on your second GPO is to duplicate, then rename, the first one.

You can now close the Group Policy Object Editor.

Tie GPOs to Machines

So far, nothing we have done has affected any machine, but that’s about to change. What we’d like to see is for the server policy to apply to all of our servers and for the desktop policy to apply to everything else. The first step in making that happen is to define which machines are servers.

Open up Active Directory Users and Computers. If you can’t find it under Start -> Administrative Tools, then go to Start -> Run, enter dsa.msc and click OK. Navigate in the left pane down through your domain to the Computers folder. Right-click on the Computers folder and select New -> Group. Name the group Servers, leave the scope and type as a global security group, then click OK. It should show up in the list in the right pane.

Double-click your new group and select the Members tab. Add each server, one-by-one using the Add button and making sure the Select this object type field contains the word Computers. You can change it by clicking the Object Types button. Click OK when complete, then close the Active Directory Users and Computers window.

Next, we tie the Automatic Updates-Servers GPO to our new Servers security group. Back in GPMC, select the servers GPO and click on the Scope tab. In the bottom window, click the Add button and type Servers into the window, then click OK. Your security group should now be listed in the Security Filtering table. Highlight the other entry in the table (Authenticated Users) and click Remove, then OK. Now servers and only servers will be able to use the Automatic Updates-Servers GPO.

Last but certainly not least, link your new GPOs to the domain. In the left pane of GPMC, locate your domain name; it should be directly under the folder called Domains. Right-click your domain and choose Link an Existing GPO. Highlight both of your new GPOs with shift-click then click OK.

You are now in business. Your policies are complete and in place.

Check Your Work

If you log on to a domain member machine and look at the Automatic Updates page of the control panel, all the settings should be grayed out, but should match the values you set in your policies. If they’re not grayed out, you can wait, or you can give your policies a push.

Group policy flows gradually to domain member machines. They update at reboot time, but sometimes it takes more than one reboot for new policies to take effect. If you are impatient, or if a machine seems to get stuck, you can hurry along the process by running the command gpupdate /force from within a command window on the stuck machine. That, and maybe a reboot, usually get things moving along.

Another way to check is by examining the Windows Registry. Nearly all Group Policy operates by forcing certain registry settings on member computers, which then modify the computer’s behavior. Enter the following in a command window on any computer:

reg query hklm\Software\Policies\Microsoft\Windows\WindowsUpdate\AU /v AUOptions

and it should display the value 0×3 on servers and 0×4 on all other machines. You can run all your queries from one machine if you prepend each query target with its IP address or hostname, as follows:

reg query \\hostname\hklm\Software\Policies\Microsoft\Windows\WindowsUpdate\AU /v AUOptions

If the value is not found, try the gpupdate trick or a reboot. If it is found but does not match what you expected, make sure your Servers security group contains the right machine names; also make sure it is properly linked to the server GPO.

Maintaining Patched Systems

Your GPOs will do a reasonably thorough job of keeping your systems up to date, but you still have work to do. Put Patch Tuesday on your calendar—Microsoft publishes new security updates on the second Tuesday of every month. Your desktops should automatically update themselves within a day or two, but you’ll have to patch the servers yourself. On each one, either go to Start -> All programs -> Windows Update, or find the yellow shield in the systray (the row of tiny icons in the lower right corner of your screen) and apply the patches that should be downloaded and ready to install.

I also recommend checking the patch state of each machine with MBSA, the Microsoft Baseline Security Analyzer (of which the latest version, currently 2.1.1, is available here). MBSA can scan a single machine, all machines on your domain, or a LAN segment. It displays quite a bit of information, but the most critical is the list of missing patches. If a machine is chronically behind, it may be more than just group policy slowness. Many malware infections disable automatic updates, so a reformat and reinstall may be in order.

Congratulations!

Your machines are patched. You have taken a critical step that every security policy framework and every set of security regulations requires. And you have eliminated much of your vulnerability to the most popular current malware attacks. And you are no longer a Group Policy newbie!