Troubleshooting Volume Shadow Copy Problems

Many different backup set types rely on Microsoft's VSS provider to create shadow copies of data for backup purposes. Having a shadow copy of data is often essential as the original data cannot be worked on directly in a production environment. Working off of a shadow copy avoids file inaccessibility, inconsistent file states, and service interruptions.

Explanation

How VSS Works

When troubleshooting VSS issues, it may help to understand more about the structure of the Volume Shadow Copy service. First, let's define some terms. These definitions are taken from Microsoft documentation.

• Volume Shadow Copy Service - A service that coordinates various components to create consistent shadow copies of one or more volumes.
• Requestor - An application that requests a volume shadow copy be taken. A backup application is an example.
• Writer - A component of an application that stores persistent information on one or more volumes that participate in shadow copy synchronization. Typically, this is a database application like SQL Server or Exchange Server, or a system service like Active Directory.
• Provider - A component that creates and maintains the shadow copies. Examples are the system provider included with the operating system and the hardware providers included with storage arrays.

For our purposes, the requestor is the Backup Agent, the provider is Microsoft's built-in System Provider, and the writers will be those included in Windows by default as well as any included with installed applications.

When the Backup Agent needs a shadow copy made for a backup job, the series of events is as follows:

1. The Backup Agent acts as the requestor and asks the Volume Shadow Copy Service to create a shadow copy. The writers are enumerated and writer metadata is gathered.
2. The writers prepare their respective applications for shadow copy creation, usually by flushing write buffers to disk, completing open transactions, etc. When that is done, the writers quiesce their data and temporarily freeze write I/Os during the shadow copy creation process.
3. The Volume Shadow Copy Service tells the provider to create the shadow copy.
4. The Volume Shadow Copy Service thaws the files system and the writers allow queued write I/Os to be completed; whether or not those held writes were successfully suspended is verified by the Volume Shadow Copy Service.
5. If the writes were not held, the shadow copy may be inconsistent and is deleted. If the writes were held successfully, the shadow copy is successful and the requestor (the Backup Agent) is given the shadow copy location.

Requirements for Creating Shadow Copies

In order to guarantee that shadow copies can be made successfully, the computer we are creating the shadow copies on must meet both configuration requirements as well as be structurally sound. Those configurations are as follows:

• VSS Storage Space - The VSS provider must have sufficient space to create the shadow copy of a volume. Many times, VSS snapshot creation will fail because there is no storage space allocated at all for a given volume. Directions for allocating shadow copy storage space are here.
• Volume Free Space - Make sure the volumes on which the shadow copy storage area is located has enough free space for the shadow copies being made. Generally speaking, usually 10%-15% free space is the minimum needed.

In terms of structural integrity, the following requirements must be met to guarantee shadow copy creation is successful.

• Stable VSS Writers - If the VSS writers on a machine are not in a stable state, they will not be able to successfully perform the steps outlined in the shadow copy creation process.
• Error-Free Hard Drive - Should the hard drive you are attempting to shadow copy have any file system or disk issues, this will prevent the shadow copy from being made successfully.

Resolution

If the backup software is unable to create a VSS snapshot/shadow copy, it will throw an error saying so along with another error giving the likely cause of the VSS failure. The cause presented by the software for the VSS failure is usually correct, however, the software can only make an educated guess as to what the VSS problem is. It is advised that you attempt to track down the VSS failure to either a "VSS" or "volsnap" error in the Application or System logs in Event Viewer, respectively. These events will more accurately reflect the nature of the VSS failure and allow you to troubleshoot the problem more effectively.

That being said, here are some common resolution steps for VSS errors you may see.

The VSS Writer Is In a Bad State

If a VSS writer is in a Failed or Timed Out state, it will not be able to complete step (2) in the shadow copy creation process successfully. VSS writers may malfunction for any number of reasons, however, generally, you can fix them by either restarting the service corresponding to the writer or rebooting the machine. To find out which VSS writers are in non-stable states, use the following command:

vssadmin list writers

This should list the writer names, states, and last error. If any writer's state is something besides "[1] Stable" then you may need to refresh that writer. To do this, first try restarting the service corresponding to the writer.

If restarting the writer's service does not put the writer in a stable state or the service cannot be restarted, you may need to reboot the machine to fix the VSS writers. If a reboot does not fix them, consider re-registering the VSS writers or reinstalling the application that writer corresponds to.

Insufficient Storage Space

If VSS storage space is really the issue, you will need to follow the steps outlined here. Make sure you allocate shadow copy storage for each volume being backed up; for Image backups, that includes the System Reserved volume.

Occasionally, however, this error will get thrown if the shadow copy creation fails at step (3) of the process even if space is not an issue. Double-check the Event Viewer logs to confirm the actual issue.

A Call to a VSS Operation Exceeded the Time Allotted

Usually, when a VSS operation times out, it does so during steps (2), (3), and (4). This may be due to high disk I/O on the drive being snapshot or it may stem from file system fragmentation. On the other hand, if simultaneous snapshot operations are being performed on the volume we are trying to snapshot, this can also contribute to VSS timing out.

To determine if Windows is creating shadow copies automatically, do the following:

1. Open Windows Explorer and right-click on any of the hard drives listed. From the menu, select Configure Shadow Copies...
   
2. If for any volume, the Next Run Time field is a date and time instead of Disabled then shadow copies are being made automatically.
   
3. To disable the automatic creation of shadow copies, click the Disable button; if a warning window pops up, click Yes to confirm. This does not prevent shadow copies from being made, only the automatic process of doing so.

In addition to multiple shadow copy requests causing time-outs, it is possible the Volume Shadow Copy Service is not responding in a timely fashion. If this may be the cause for the time-outs, it is recommended you try restarting the Microsoft Software Shadow Copy Provider and Volume Shadow Copy services.

Finally, since the reasons for VSS time-outs are complex, you may want to try a defrag of the volumes being snapshot, rebooting the machine, and/or deleting existing shadow copies.