VM snapshots are one of the best virtualization features ever. But…have you ever had a VMware vSphere or Hyper-V snapshot grow out of control and fill the datastore the VM resides on? You know what happens… even the best VM admins out there seem to get burned once when they create a snapshot for some routine maintenance and just forget about it afterwards. The VM crashes and goes into a suspended state until free space is added to the datastore.
Sometimes this task is harder than it sounds. If your SAN is out of space, or the SAN management tools are out of your control, you could be stuck.
But… follow a simple trick to give yourself that last little bit of wiggle room in the event that a snapshot fills a datastore.
Add a large text file to the root of the datastore that you can delete if you need headroom! I know it sounds too simple… but it’s simple and effective.
These directions are updated for VMware 5.5 environments, and is validated through vSphere 6.5, but the same concept applies to any hypervisor out there today.
How do you do this?
Enable the SSH server, and remote into one of the ESXi hosts. This example is from a vSphere 5.5 ESXi server from my home lab.
Change directories into the root of the datastore that you want to create the file in.
Use the Linux command ‘dd’ to write a 10GB file (or whatever size of your choosing). Basically, we’re creating a file full of zeroes with a 1MB block size and ten thousand blocks. Simple, eh?
dd if=/dev/zero of=snapshotfailsafe.txt count=10000 obs=1M ibs=1M
It will take a few minutes to create the file. You can see the file creation process with another terminal, and can see the activity in the hosts’ disk performance view in the vSphere Client.
Once done, you’ll see it in the terminal and in the datastore view screen.
If you have a VMware or Hyper-V snapshot that has filled the drive, just hit the datastore browser and delete the failsafe file! You just gained time to better manage the snapshot or move data around to accommodate the growth.
I know this seems unnecessary, but every VMware admin seems to get burned with a rogue snapshot once. Occasionally, it’s more than once.
How can I prevent actually needing this?
That part is pretty simple, too. Just set up a vCenter alert to warn you if you have a snapshot that is growing out of control! The directions for how to do this are located in this VMware KB article. (Just remember to set up the email server so you actually receive the emails!)