Jan 282014

OK, so maybe they do not lie to you, but I got your attention. In virtualized environments, the performance statistics that you see from the management interface may not give you the entire picture, and can lead to bad assumptions or misleading recommendations.

Remember, the virtualization layer does not understand the nuances of the applications that you are running inside the virtual machine, so it might be giving you performance statistics that are not representative of that application. 

For example, SQL Server is very memory hungry, as it uses memory to cache data reads from disk. If you give a SQL Server virtual machine 32GB of vRAM, and allocate 26GB to the SQL Server buffer pool, chances are that all 26GB is going to get consumed as your workload reads data. However, the physical host server where your VM is running might not understand this usage pattern, and might misinterpret things. 

Take this scenario. I recently had to load a 20GB dataset into a SQL Server for performance statistic analysis. The data is read from disk and inserted into a new SQL Server table. At the end of the load, and because nothing else was running on that server at the time of the load, almost all of the inserted data was also now in the buffer pool. But, look at what the VMware performance counters for memory for this virtual machine reported.

vmware stats active memory during bulk data load

You’ll notice that the active memory counter grows as the data is loaded. Once the data is loaded, however, the data resides in the buffer pool, but VMware is reporting that the memory is no longer active as the timer on the block deltas grows. During this time, I am actively performing the analysis of the data, so the data is still being read from, but just does not change. This activity is not reflected in the counter at all.

It’s not just memory either. LogicMonitor has a great writeup of the differences between Windows Perfmon sampling inside a VM versus the actual CPU usage, as reported by the physical server. 

I have heard virtualization administrators tell DBAs that because these counters show that most of the memory is stale and not active, it’s OK for them to reduce the amount of memory allocated to that virtual machine.  This counter is pretty misleading, and I see quite a bit of misinterpretation by virtualization administrators on this topic. Now, they should not be expected to know the specifics on the workloads executing in their environments, but understanding that this counter does not show you the entire picture is going to save everyone a lot of time during these types of discussions.

As a result, this disparity is just one of the reasons why I constantly push the process of actively collecting ongoing performance statistics from both the physical host and from inside the guest, as well as from within the application that you have running inside the VM. Correlate everything together to see the full picture of what is going on, because any single metric probably does not give you the complete picture.

UPDATE:  (t) from VMware has previously written a great blog post on vmware.com discussing this exact topic, and he goes into greater detail on the counter. You should go check it out here!

Jun 172012

Ever run out of drive letters on a server, or have some other business requirement, and have the need to go with mount points? Do you wonder how you’ll monitor those new drives with Perfmon? Here’s how!

By the way… If you’re not monitoring any of your disk activity on your production servers, shame on you. I like to take five minute performance metric samples and rotate log files by day on all of the production servers that I work with. If you ever need to reference back to see how a server was performing, you have the ability to do so. Too much information is never a bad thing, right?

First, determine where you’ll be mounting the drives. I’m going to mount two drives in the E:\mount folder. One is for SQL Server database data files, and the other is for log files.

Second, I’m adding two thin provisioned 1GB virtual drives (drives 3 and 4 below) to an existing VM in my home lab. The server runs Windows Server 2008R2, but the same concepts apply to Windows Server 2003 as well.

Perform a drive rescan in Disk Management, and you’ll see the new drives appear as offline and uninitialized.

Set one of the drives to online, then initialize it. Next, right click on the unallocated space and create a new simple volume. Where you would normally assign a drive letter, select one of the paths you previously created on the disk.

If you are doing this for a SQL Server, you are formatting each partition to a 64KB NTFS allocation size, right?

Perform the same actions for the other drive. Once completed, your disk list should resemble the following.

When you click through to the folder in Windows Explorer, you see the following items, telling you (poorly) that these are mount points.

OK. We’re all connected and ready to test. First, find a way to generate some disk activity. I like to use IOmeter for simple things like this.

Errrr. Maybe not. IOmeter does not distinguish mount points from the base drive. Oh well. We’ll just copy a large file back and forth.

Stage something to move around. In my case, it was just an ISO file.

Open Perfmon and click into the counters list.

Check it out! Perfmon has the two mount points listed out appropriately. Add a couple of appropriate counters, and start your file movement testing. Problem solved!

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.