Have you ever had a problem with snapshots in VMWare? Sometimes it’s enough just to know someone else feels your pain. Most of the time of course, it’s not. So, if you’re in that situation here are some things you should know about snapshots before you’re required to troubleshoot them.

The first thing I’ll mention is that if you look at a datastore through VI client you’ll see virtual disks represented like this:

server.vmdk

Via PuTTY they’ll appear as two separate files:

server.vmdk
server-flat.vmdk

server.vmdk is a text file that describes the disk, whereas server-flat.vmdk is the actual file containing the disk’s data.

If you have a server called ‘server’ stored on a datastore with the friendly name of ‘san_mirror_1’, you can view the text file via PuTTY as follows (once you’ve logged in)

cd /vmfs/volumes/san_mirror_1/server
vi server.vmdk

Here you’ll note some interesting things.

CID=ab1234cd <- This is a randomly generated string, but it’s quite important as you’ll see in just a second.

ParentCID = ffffffff

The ParentCID is the next disk up in the chain. A value of ffffffff denotes that this disk is the ‘original’ disk.

You’ll also find information here on Adapter type, geometry, etc.

So, using our earlier example, we have a server called ‘server’ with one VMDK of 30GB. We then create a snapshot called test.

If you do an ls on /vmfs/volumes/san_mirror_1/server we’ll now see the following files (there’ll be more obviously, but these are the ones we’re interested in):

server.vmdk
server-flat.vmdk
server-000001.vmdk
server-000001-delta.vmdk

If you run vi server-000001.vmdk you’ll notice something about the CID values, namely this:

CID = <another random string>

ParentCID = ab1234cd (i.e. the value of the CID attribute for the base disk)

This relates the snapshot’s disk to the flat file. If the snapshot’s ParentCID and the base disk’s CID values don’t match, you won’t be able to apply the snapshot until you manually edit the files so that this CID relationship exists.

If you added another snapshot (so, you had server-000002.vmdk) this disk would have a ParentCID equal to the CID of server-000001.vmdk

One more attribute in the snapshot’s VMDK file is of interest:

parentFileNameHint=”server.vmdk”

If the snapshot relates to a disk on another LUN this value will contain a full path. It’s possible you’ll need to amend this if disk or snapshot files have been moved around.

Another file worth knowing about is server-Snapshot#.vmsn (Where # denotes the number of times a server has been snapshotted). This appears to be a text file if you use vi to look at it, but actually you won’t want to alter this file.

Finally then, files wise, we have server.vmsd

This file contains a wealth of information and is basically the Virtual Machine’s way of tracking where it is with snapshots.

It contains a complete list of snapshots for the VM, the number of disks, friendly name for the snapshot, etc.

Assume we have the same server, but it has two Virtual Disks, both of which have been snapshotted once. You’ll see lines in the vmsd like this:

snapshot0.disk0.filename = “server.vmdk”
snapshot0.disk0.node = “scsi0:0”
snapshot0.disk1.filename = “server_1.vmdk”
snapshot0.disk1.node = “scsi0:1”

Ok, now imagine you’ve deleted that second virtual disk and you then try and delete the snapshot. You’ll get an error message telling you that the virtual machine configuration has changed since the snapshot was taken.

In this instance I’ve had some success with deleting the ‘disk1’ lines in the vmsd and then reapplying the snapshot.

It’s interesting to note that if you view the server.vmx file with vi you’ll notice that the disks listed point to the snapshot files, e.g.

scsi0:0.present = “true”
scsi0:0.filename = “server-000001.vmdk”
scsi0:0.deviceType = “scsi-hardDisk”

Even if the CID chaining, parentFileNameHint, etc are all correct you might still have problems with deleting snapshots. By the way, ‘delete’ is a pretty confusing term, because to delete a snapshot actually means to merge the changes in a way that they are committed to the base disk (rather than to simply revert to the base disk).

If you’ve read this far, you’ve almost certainly got a snapshot that won’t apply (or you just like to read maybe). Either way, here are some tips for fixing this kind of issue:

As crazy as it might sound, and probably also like the last thing you’d want to do. If the snapshot isn’t showing up in VI client’s snapshot manager, and yet you know the server does have a snapshot running. then snapshot the server again. VI should then be able to show you the new snapshot, and previous snapshots. ‘Delete All’ will then remove all the snapshots.

Seen as how I’ve mentioned ‘Delete All’, it’s important that you know how this works. Because, to be honest, it’ll look like it’s not doing anything. This is especially important when you consider that if you try a ‘delete all’ from VI client it will time out after 15 minutes (unless you changed the time out value), but will keep going.

When you select ‘delete all’, the last snapshot in the chain is rolled into the previous snapshot and so on, until all the data is back into the base disk. Then all the snapshot files are deleted. This takes up a lot of space, and if you have a long chain of snapshots, or they’re very big, this will make it look like nothing is happening, when actually it is (even if it takes several hours for very large VM’s)

For this reason I’d recommend you delete snapshots one at a time if you’ve got a long chain of them. Also, turn off the VM if you’re able, it’ll speed things up. You also have the option to run a ‘delete all’ from the command line:

vmkfstools /vmfs/volumes/san_mirror_1/server/server.vmx removesnapshots

There is one, last ditch effort you can make to recover a VM that’s just a mess of snapshots (but you can still power it on). Specifically, you can clone the disk.

To do this, run the following command, giving the path to the last snapshot in the chain.

vmkfstools –i /vmfs/volumes/san_mirror_1/server/server-000002.vmdk /vmfs/volumes/special_spare_lun/new_server.vmdk

When it’s finished cloning (this may take a while, and you can only do it when the VM is powered off) you should be able to create a new VM (or edit an existing one) using this disk.

I’ve been in a situation that required edits to the vmsd, vmx, and vmdk files, as well as several hours of patient waiting for the ‘delete all’ to finish on a critical, live server. My advice is to draw a diagram before you start pressing buttons, check the CID chains, check the parentFileNameHint, check the VMX file, check the VMSD file, make a plan, then execute it. Good luck.

- Thesysadmin