As a request from one of our VMWare admins, I’ve started testing the VMWare image backups. In the past, we’ve installed the regular backup/archive client on each VMWare guest OS. This has let us standardize our install regardless of if it’s a virtual or physical server. This doesn’t allow you to take a full snapshot of a VM, instead you have to rely on your baremetal recover procedures just as if it was a physical server.
Unfortunately, an application admin messed up a server, almost beyond repair. If the VMWare admins had been made aware of the work, they could have generated a snapshot just before the work started, and recovering would have been quick and simple.
Starting in the 6.2.2 version of the Windows client TSM supports full image backups using the VStorage API. I’ve heard people complain there isn’t a lot of documentation on this, so I thought I would write about my testing.
VMWare image backups have been in TSM for a while, but previous versions have relied on VMWare Consolidated Backup (VCB), which VMWare is withdrawing support for. The VCB functionality is still in the client, so you can upgrade to the latest version if you are already using image backups with VCB.
Like some other similar products, you will need a windows server to do the backups. This does not do a proxy backup as the file level backups using the VStorage API, but saves the files in TSM under the node doing the backups. I used the VCenter server because it was handy, though I don’t see why you couldn’t use another Windows box. My VCenter server isn’t super powerful, but it has plenty of power to spare for just running VCenter, and should be adequate to use for backups of our VMWare cluster. Here are the specs:
Intel Xeon E5530 2.4 GHz
4 GB RAM
Windows Server 2008 R2 Standard
2 – 140GB SCSI drives (mirrored)
2 – 1GB NICs (no etherchannel)
Setup could not be much more simple. In the client options, there is a “VM Backup” tab with several options. First, you must select if you want to do a VMWare File Level backup (does a proxy backup for Windows guests), VMWare Full VM backup (either using VCB or VStorage), or a Hyper-V Full VM Backup (for Microsoft Hyper-V users).
Next, you can specify doing a “Domain Full VM” or a “Domain File Level VM” backup. Domain Full VM backups are image backups of the whole VM. Domain File Level backups seem to be proxy node backups of Windows boxes. I don’t know why there are two options to specify this in the GUI, as they seem to be mutually exclusive. In the same section, you can list VMs to backup, use pre-defined groups like “ALL-VM” or “ALL-WINDOWS”, or specify folders of VMs to backup if you’ve broken down your VMs into folders.
Next where you specify your VMWare VCenter or ESX server login information. You’ll probably want to specify a role account for this. In testing I just used my regular VCenter login.
And, finally, you can tell the client to use a specific management class for your image backups. I made a new management class called VM_IMAGES. Expiration and retention seem to work as expected. You must also specify if you’re using the VStorage API or VCB.
I turned off client deduplication and compression for my testing. I’ll test it’s effects later. I found that the test VM images grew in size when compressed, and enabling compression and deduplication cut the transfer speed in half. There is a note in the documentation that says that compression can only be used with the VStore Full VM backup if it’s being saved to a deduplication enabled storage pool.
There are a couple of other things to note, the VCB backup feature uses a staging area on a datastore to copy the files to before backing them up. The new VStorage API backup doesn’t. This may make it somewhat faster, and in my testing the disk usage did not increase during backups. And, even cooler, the VStorage API uses the changed block tracking feature of ESX 4 and later to only backup the occupied space. So, if you have a VM with a 100GB volume and only 4GB of data, it’s not going to backup the whole 100GB.
Lets do a backup! In the GUI, you can select Actions -> Backup VM which opens a GUI selection box. It’s a lot like the file backup GUI, just select the VM:
When you hit the backup button, the same status window used for file backups opens and everything progresses as usual. You can also use the commandline:
tsm> backup vm DCFM -vmfulltype=vstor -vmmc=VM_IMAGES Full BACKUP VM of virtual machines 'DCFM'. Backup VM command started. Total number of virtual machines to process: 1 Backup of Virtual Machine 'DCFM' started Starting Full VM backup of Virtual Machine 'DCFM' Backing up Full VM configuration information for 'DCFM' 4,086 VM Configuration [Sent] Processing snapshot disk [OpsXIV2 Datastore 1] DCFM/DCFM.vmdk (Hard Disk 1), Capacity: 17,179,869,184, Bytes to Send: 15,998,124,032 (nbdssl) Volume --> 15,998,124,032 Hard Disk 1 Backup processing of 'DCFM' finished without failure. Total number of objects inspected: 1 Total number of objects backed up: 1 ... Total number of bytes inspected: 14.89 GB Total number of bytes transferred: 14.89 GB Data transfer time: 1,075.59 sec Network data transfer rate: 14,525.08 KB/sec Aggregate data transfer rate: 14,487.67 KB/sec Objects compressed by: 0% Total data reduction ratio: 0.00% Subfile objects reduced by: 0% Elapsed processing time: 00:17:58 Successful Full VM backup of Virtual Machine 'DCFM' Unmount virtual machine disk on backup proxy for VM 'DCFM (nbdssl)' Deleted directory C:\Users\pvaughan\AppData\Local\Temp\2\vmware-pvaughan\421a7a97-64a4-79d1-b854-c9b30ea6dca7-vm-65\san Deleted directory C:\Users\pvaughan\AppData\Local\Temp\2\vmware-pvaughan\421a7a97-64a4-79d1-b854-c9b30ea6dca7-vm-65\nbdssl Backup VM command complete Total number of virtual machines backed up successfully: 1 virtual machine DCFM backed up to nodename MMC-OPSVC1 Total number of virtual machines failed: 0 Total number of virtual machines processed: 1
When the backup starts the TSM client creates a new snapshot of the VM. When the backup is finished (or interrupted) the snapshot is dropped. The disk images of that snapshot is what you end up with. The up-shot of this is that the disks are probably in pretty good shape (fsck ran clean for me) and you can get a good backup of a running VM. Here’s a screenshot taken during the backup:
Backups are cool and all, but what about restores? The restore process is pretty simple too. Just select Actions -> Restore VM from the GUI. A familiar window pops open, and you select the VM backup to restore:
When you select Restore another window opens. This allows you to restore back to the original VM image (boring), or specify a new VM. This MAY be useful for cloning a VM between VM Clusters if you can’t use the VM Converter tool. If you want to make a new VM, just specify the new name, the Datacenter, ESX host, and Datastore:
The client will reach out and create a new VM exactly like the original, and then restore the files for it. I successfully cloned my test VM while it was running the first try. And, the restored VM booted without incident.
I’m doing some test on FullVM and we had to interrupt it as it was too much slow (max we had 3MB/sec) . I’m talking about a Windows 2003R2 Server with FC adapters and enterprise storage IBM DS8100 and HP XP24000 and TSM server on AIX with LTO4-FC drives.
I would like if you can explain how MC will act on FullVM backups. I mean, TSM will ever recognize automatically a deleted VM and make it expire?
I’ve not seen slow restore times, but we are storing most of our data on disk, so I’m not hitting any tape drives. LTO4 drives should see much better throughput than that anyway. After ruling out the network, and verifying that you get better throughput when restoring other large files from tape, I would engage IBM support. There was a recent webinar on TSM client instrumentation. You’ll have to use the commandline client to collect the data, but it may help pinpoint the issue.
As for expiration, TSM should treat the removal of a VM just like removing a filesystem. If you’re using a schedule with ALL-VM, and remove a VM, I would expect TSM to mark the virtual filesystem as “Inactive” and remove it according to the expiration policy. I’m out of the office until the middle of next week, but I’ll test it when I get back.
TSM 6.2 documentation describes a scenario called In-guest backup/restore.
My understanding is that the host *MUST* be a Windows box (although not clear in the document). Is this true?
Also, does the Host server (VCenter in your case) need access to the SAN or disks that the ESX access?
You can backup an image of the whole VM with any guest OS. I’ve testing windows and Linux without issues. The down-side is that these are image backups. You are backing up the whole VM every time, so don’t keep too many versions around. If you want to backup the individual files with the VStorage API, then it MUST be a Windows guest VM. I started another post about doing file level backups, I guess I should finish it. 🙂
No, if you are using the Vstorage API, then your host server does NOT need access to the same disks as the ESX servers. The documentation from IBM still says that you do in a couple of places, but that’s not the case.
you might also want to take a look at the new TSM for VE, it’ll allow you to perform file lever restores from those image backups, eliminating the need for file level backup. there’s a nice IBM wiki about it (and the basic VMware backup) here:
We haven’t looked at it yet. Unfortunately, with a recent price increase we are probably not spendng more on TSM. In fact, Avamar will probably be getting a looking at in the near future.