Sep 122016
 

Bottlenecks in your testing tools, infrastructure, or methodology might just hurt your load test results, and the results can skew your metrics.

For example…

A few weeks ago we were running an iperf load test to see what improvements (if any) could be made to the configuration of a Windows Server networking stack. Setting up iperf for maximum throughput testing is easy.

iperf1

iperf2

The base test proved pretty fast, too!

iperf3

This testing was performed on our new 10GbE lab environment without jumbo frames enabled in the guest. We’ve got a Cisco Nexus 5k underneath this environment, so it is nice and fast.

We wanted to see the performance improvements with keeping this stream of data on the backplane of one physical server in our VM lab. We moved both of these VMs to the same host. Performing the same test should improve performance. I’ve seen upwards of a 16x performance improvement on a 1GbE network, so we should see at least a decent improvement on 10GbE, right?

The next test is after we moved them to the same host.

iperf4

It’s virtually (no pun intended) identical.

 

OK. So, let’s try to make Windows a bit more efficient.

autotunedisable

The results show a bit of improvement.

autotunedisablediperf

But… why did it stop there? 

OK…

Looking at the system state showed us an instant bottleneck. One vCPU was maxed out, while the other was stagnant. This imbalance is one key reason why you should rarely trust a single CPU average metric coming from your monitoring tools.

ProcessorUsage

The load test utility had maxed out the compute resource that it had available, due to internal limitations within iperf itself.  It’s a shame that this utility is not multi-threaded, because I think we could have a much greater result of improvement on this system.

Monitor the utilities that you’re using to do load testing, because limitations like this might skew your results!

Oct 132015
 

networkcable

Have you ever wondered if that black box in your datacenter called the network is performing as well as it should be? Do you have random periods of time where normally fast data movement processes seem to take forever, but the diagnostics on your servers all seem fine? What about validating the network performance back and forth to your disaster recovery site? Let’s go through how to check the network to make sure that all is well, or generate some objective repeatable tests that you can bring to your networking group and ask questions with.

I frequently mention in my presentations a free utility that I use called iperf that you can use to validate your network performance. Several questions came up about where to get it and how to use it. Read on for a walk-through of how to use iperf!

Continue reading »

Mar 122014
 

Today I am starting a new series of short posts designed to help SQL Server administrators with virtualized SQL Servers twist the mind a bit and think about virtualization in a different light. I call it “Smart Moves with SQL Server VMs“.

Virtualization technologies for non-VM-admins usually result in just P2V’ing a system and then forgetting about it. Once virtual, the world can change (for the better, I promise) in ways not usually considered, and you – the data manager – can dramatically benefit from embracing the new technology.

The first post in this series discusses a trick with using virtual machine proximity when dealing with large volumes of data movement, such as nightly ETL processes, application server data handling, or database backups.

First, we’ll be using a free utility called iperf to demonstrate the raw performance differences here. You can read more about how to use it at this blog post.

So the scenario is as follows. You have two servers – a SQL Server VM that runs a nightly ETL process, and an application server VM where you are pulling large quantities of data from. The traffic occurs over the physical network.

A quick iperf test can show the total possible network bandwidth between the two servers.

iperf04

This test shows that we’ve got just about the maximum possible bandwidth between the two servers on this 1Gb network. At this point, the data that you will transfer is most likely limited by this bandwidth bottleneck. 

Now, what if you were to place both of these VMs on the same physical machine? If the networking is set up properly, the situation can change. You can rerun the same iperf test and see the following results.

iperf01

We get an over 15x performance boost in this scenario.

Your network traffic now is passing through the backplane of the physical machine and not even touching the physical network. All of it is transparent to the SQL Server, operating system, or application code. There are no code changes, no funny tricks with Windows Server networking…. just a very dramatic performance boost.

Think about this performance difference for a moment. With the networking stack out of the picture, your large data movement bottleneck could now be the storage speed reading from disk, or it could be the CPU scalability of the ETL process itself. Both of these bottlenecks have much higher thresholds than the networking stack’s performance.

Your large volume data transfer process is virtually sure to experience a performance improvement, and therefore cut down the run time of the process and resource consumption on the networking stack. 

The best part is that both VMware vSphere and Microsoft Hyper-V have PowerShell interfaces into the hypervisors, and scripting out a command to co-locate these two VMs on the same host is a relatively trivial task. All you need are simple permissions from the virtualization admins to achieve this. It can be programmed to execute as a prerequisite step in the ETL processes or jobs that perform these tasks.

Go forth and improve performance! More tips and tricks like this are coming in the following weeks!

Nov 112013
 

networkcable

Have you ever wondered if that black box in your datacenter called the network is performing as well as it should be? Do you have random periods of time where normally fast processes seem to take forever, but the diagnostics on your servers all seem fine? What about validating the network performance back and forth to your disaster recovery site? Let’s go through how to check the network to make sure that all is well, or generate some objective repeatable tests that you can bring to your networking group and ask questions with.

In my SQL PASS Summit presentation from last month, I mentioned a free utility that I use called iperf that you can use to validate your network performance. Several questions came up about where to get it and how to use it. Read on for a walk-through of how to use iperf!

Continue reading »

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close