Operating system upgrades have always been something of a crapshoot. We as administrators make every effort to follow all of the recommended best practices, but it is difficult to know with absolute, 100 percent certainty how a new OS will behave in a production environment.
In the past, I have always recommended a multi-step approach to OS upgrades. The first step is to build an isolated lab environment, and then configure the lab in a way that mimics your production environment, but on a smaller scale. That way, you can work through the OS deployment in the lab prior to trying it in a production environment.
In the case of desktop OSes, I recommend following lab testing with a pilot deployment in which the new OS gets deployed to select power users prior to performing a large-scale distribution. This gives you the chance to work through any issues that were not caught during the lab testing. Of course, I also recommend performing a clean installation (as opposed to an upgrade) whenever possible, because doing so greatly reduces the chance that you will have problems.
This method of working through new OS deployments has worked well for me for years, but I will be the first to admit that it isn't perfect. After all, lab testing will only go so far in helping you to predict how the new OS will behave in a production environment. My experience has been that no matter how good the lab environment may be, there always seem to be some subtle nuances that are unique to the production environment.
Thanks to virtualization technology, however, the differences between a production environment and a lab environment can become a non-issue. I recently spoke with a friend who used his backup software's virtual lab feature for Windows Server 2012 deployment testing.
My friend is using Veeam Backup and Recovery to back up all of the servers in his fully virtualized datacenter. This particular backup application not only keeps the virtual machines backed up in near real time, it also allows for on-demand creation of virtual lab environments. In other words, it is possible to provision an isolated lab environment very quickly that is an exact replica of your production environment as it exists at that point in time.
There are several advantages to using this approach. First, administrators do not have to waste time manually creating a lab environment. Second, creating this type of virtual lab is much faster than building a virtual lab from scratch. Snapshot technology is used to instantly provision the lab environment. There is no need to work through a backup restoration. Perhaps more importantly, because the virtual lab runs directly from your backups, there is no significant amount of additional storage required for the virtual lab.
The virtual lab consists of a series of virtual machines running in a sandboxed environment. These virtual machines look and feel exactly like the virtual machines on the production network. The virtual machines that make up the virtual lab can communicate with one another, but they are completely isolated from the production environment. Likewise, snapshots are used in a way that makes it safe to make alterations to the lab virtual machines without having to worry about accidentally modifying your backup in the process.
A virtual lab environment is absolutely ideal for OS deployment testing. Because the virtual lab is an exact replica of your production environment, you can determine just how the deployment process will behave once upgrade for real. You can even use the labs to work through the deployment multiple times in an effort to rehearse or fine-tune the deployment process.
Veeam isn't the only company that offers a virtual lab product, but my friend who uses Veeam Backup and Recovery is the first person I have talked to who used this type of virtual lab to prepare for an OS upgrade. His overall assessment was that the virtual lab environment allowed him to take the risk out of the deployment process.