Jul 052017

If you are actively managing VMware environments with workloads that have high performance needs (such as all of the virtualized SQL Servers that we work on), this new book called VMware vSphere 6.5 Host Resources Deep Dive, written by Frank Denneman and Niels Hagoort, is a must read book! Designed for VMware-oriented system architects, this book walks the reader through each layer of host resource allocation and management in a way that no other book has ever accomplished.

This book redefined the phrase “deep dive”, and I’m taking a lot of notes for future authoring and presentations. The depth of content is unparalleled in tech authoring.

Topic sections such as CPU, memory, storage, and networking, the four main areas of resource management on any virtualization host, are presented. Anyone can talk about these settings in general, but the advanced VCDX-level topics are covered in incredible depth. We work with a lot of advanced virtualized SQL Server VMs, and significant discussions from this book for performance tuning these SQL Servers include:

  • Advanced vNUMA balancing and optimization
  • CPU core counts versus clock speed
  • vNUMA memory speeds and non-local memory access
  • Clearing up misconceptions about vSphere Balanced Power Management
  • Queues and resource allocations

Go get this book – NOW! It’s a must read, and read it twice. No, three times. Then give the book to colleagues who can benefit. I’ve got two copies on my desk right now, and know who these will be sent to after I’m done re-reading them!

Dec 022013

A wonderful gentleman who attended my SQL PASS Summit session this year recently emailed me and asked me a great question about optimal setup for SQL Server virtual machine vCPU and vNUMA configurations. I thought about the question and realized that while the session went into a light level of detail, I can go much deeper here on my blog.

While the examples and screenshots in this post are all from my VMware vSphere lab environment, the same concepts directly apply to all other hypervisors on the market.

First, what is NUMA?

NUMA, or non-uniform memory access, is an architectural approach in computer hardware design that places banks of memory in close proximity to each physical CPU socket. It adds efficiency to the system as CPU counts and memory capacity increase, which in turn improves performance. The proximity of the memory bank is extended into the operating system so that applications can be coded for workload efficiency. SQL Server is NUMA aware, and therefore takes advantage of this awareness. More information on NUMA can be found over at Wikipedia.

The trick with NUMA and virtual machines is to determine how to construct a virtual machine to take advantage of NUMA when performance truly counts. Some virtual machines are somewhat idle and not resource intensive at all, and these are generally less impactful with the performance tuning. However, other virtual machines are absolute resource hogs and consume quite a bit of resources to satisfy their workload requirements. These workloads will certainly benefit from NUMA.

With VMware vSphere, the ability to extend these NUMA details into a virtual machine was first brought to us in vSphere 5.0. Prior to this point, you could just select the number of vCPUs that a VM was assigned, and the hypervisor was left to balance the CPU and memory requests.

Host Physical Machine

The modern physical servers in a virtualization server farm all have individual NUMA configurations if they have more than one CPU socket. From my home lab, the vCenter management interface tells me the CPU configuration of the physical server.


You can see that one of my home lab servers, an HP DL 380 G6, has eight Intel Xeon X5550 CPUs. If you drill into the setting a little further, you can see that I have two CPU sockets, each with a quad core chip in it, with Hyperthreading enabled.


So, the NUMA information is one socket with four physical CPU cores per bank of memory. A quick Google search will show you that the motherboard from this server has two banks of memory located on either side of the CPUs. The server contains 72GB of memory, so each NUMA node also contains half of this amount, at 36GB of memory.

Excellent. We now have our NUMA parameters that we can design around.

Virtual Machine Sizing

If the virtual machine is small but the workload is ideally sized within one of these NUMA nodes, keep the size the virtual machine small so it fits! The hypervisor will keep the virtual machine CPU and memory tasks confined to one physical NUMA node and the performance will benefit.

If the virtual machine will not fit inside one NUMA node, balance it.

For example, my SQL PASS Summit demonstration virtual machines were configured for six vCPUs because I had “right-sized” them according to the SQL Server workload I was running on them.

For the non-tuned virtual machine, the default CPU configuration at VM creation time is to create six virtual CPU sockets, each with one virtual CPU core. You can see this by editing the VM configuration.


This configuration means that the CPU and memory requests are free to jump around the NUMA nodes, which potentially leads to performance degradation because of the inherent inefficiency in these transitions.

On the flip side, the tuned virtual machine configuration specified two virtual CPU sockets, each with three virtual CPU cores. This configuration helps the workload fit within the two nodes with room to spare for other processes.


I also explicitly did not enable CPU hot add. If you enable this, under no circumstances will NUMA be extended into the virtual machine, even if the other configuration components are set properly.

Finally, by default VMware vSphere will only extend NUMA into the virtual machine when the vCPU count is greater than eight. If you add the advanced VM configuration parameter numa.vcpu.min and set the value to the number of vCPUs of the VM (in this case four), you can see this within the operating system of the virtual machine.


More details on this setting can be found in the whitepaper Performance Best Practices for vSphere 5.5 on page 44.

Now you know how to set vNUMA settings within VMware vSphere. All other hypervisors have their methods to do the same thing. Understand your environment and you can more efficiently tune your mission-critical workload virtual machines for performance!

 Posted by at 8:58 am  Tagged with:

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.