# Slow Logical Disk write times on SBS 2003 on VMWARE



## ed56 (Feb 19, 2009)

We recently did a migration from a standard installation SBS 2003 from an IBM box to a HP ProLiant ML370 G5, virtualizing the SBS 2003 on VMware ESX Server 3i.

Details as follows:

VM Host
VMware ESX Server 3i, 3.5.0, 123629 
HP ProLiant ML370 G5
Single Xeon E5430 (4 CPU x 2.666 GHz)
4091.9 MB Physical Memory
693.75 GB in datastore 1 on Smart Array P400
Broadcom NetXtreme II BCM5708 100Base-T dual NICs, one to ISP, other to private network. 

VM Guest
Guest OS:SBS 2003 SP2
CPU: 1 vCPU
Memory: 2560MB
Mem Overhead: 140.79 MB
CPU usage: 169 MHz
Host memory usage: 1.66 GB
Guest memory usage: 512.00 MB
Windows Paging file size for C: Initial: 1536 MB, Max: 3072 MB, Currently allocated: 1536 MB, Recommended: 3838 MB.

The server has been VERY slow, but improved "some" with installing the LSI Logic in the VM configuration, replacing the VM's BusLogic SCSI controller and drivers in the guest (SBS 2003).

This could be some kind of IO problem, a RAID configuration problem, but I have eliminated the possibility of any AV file checking problem. Frankly I am stumped as how to proceed.

The disk read and write speeds are shown below:










Any suggestions would be appreciated!


----------



## Tekmazter (May 22, 2008)

how many vm's are you running? Remember, VMware tries to load balance I/O across provisioned LUN's. 

have you run esxtop for queue's etc..?


----------



## ed56 (Feb 19, 2009)

We havebeen running only 1 VM, the 2003 SBS until today. I added a DM appliance just a little while ago.

Not really familiar with esxtop, could you ellaborate?

BTW, the 2003 SBS has very light usage, probably only 5 - 10 shares on at a time.


----------



## Tekmazter (May 22, 2008)

There was a good discussion over at arstechnica about this earlier this month 

http://episteme.arstechnica.com/eve/forums/a/tpc/f/96509133/m/266001696931

There is lots of talk around RAM requirements under VMware to improve disk I/O and of course overall spindle count too. The former may apply in your case although I would really take a look at ESXtop to see what you can find out. 

esxtop is run at the console of ESX and similar to the Linux top command. A good read would be the following: http://blogs.vmware.com/performance/2007/01/index.html


----------



## ed56 (Feb 19, 2009)

Thanks.

I'll check it out.

BTW, the DM virtual appliance is slow as well, an Apache server running on Linux, so it should not be. The only thing the SBS has to do with it is ISA. :sigh:


----------



## Tekmazter (May 22, 2008)

ed56 said:


> Thanks.
> 
> I'll check it out.
> 
> BTW, the DM virtual appliance is slow as well, an Apache server running on Linux, so it should not be. The only thing the SBS has to do with it is ISA. :sigh:


Curious ... what RAID type are you using, how many drives do you have in the RAID set, and what type of drives are you using?

If you have fairly fast drives and your RAID set is one built for both performance and redundancy, then I would really start looking at your memory. It's problably cost effective to upgrade anyway. Your server certainly has the horsepower to throw a lot more at it if you scale it up in terms of just memory resources.


----------



## ed56 (Feb 19, 2009)

I did not set up the RAID, but I am told it is a typical set up. I THINK RAID 4 (we have 5 250 GB drives) 

The drives are SATA

What is curious, is that I moved the DMS from an internal IP via a port accessed via ISA to its own public IP and the DMS is now running very fast. The appliance the DMS is on is a Linux/Apache configuration. The only difference is the public IP that accesses it now rather than a defined port on ISA - - - now it is plugged in to the public NIC rather than the office NIC.

Could this be a NIC or an ISA issue? Or perhaps just too bogged down to pass packets. 

regarding memory, both Exchang and SQL are grabbing all of the extra memory (as usual) and are supposed to give it up as needed. They are using a ton, so it seems there is plenty of memory.

Still does not explain my test.


----------



## Tekmazter (May 22, 2008)

Wow, it's not often I've found someone running RAID4. You should note that under RAID4 there is quite a bit of loss in performance when it comes to random writes due to the single parity drive.

As for the performance gains you noticed in moving the appliance, I'd like some more information on how you have your ESX setup. 

Also, ISA in any mix is going to add a layer of inspection to traffic which otherwise would flow w/o resistance. Keep this in mind. 

Ha...I'm actually deploying a rather hefty ISA setup of my own in the next 30 days. I'm going with Celestix from Celestix.com. It's a two-node array with a separate management console. Fully HA --very nice!


----------



## ed56 (Feb 19, 2009)

I am double chcking on the RAID 4.

Here are various screenshots of the configuration:


----------



## Tekmazter (May 22, 2008)

Looking at your memory ... you want to remember that what you allocate in RAM for a guest OS is only part of the story. You need to remember that the there are numerous other things (outside of ESX too) which take advantage of RAM.

http://www.kingston.com/branded/pdf_files/final_esx3_memory.pdf

Even a physical ESX box with maybe 3 VM's and only 4-8GB of RAM will thrive if you add more.

At this point I would begin the process of looking for your bottleneck in the ISA area. What kind of switching are you using on the LAN which is configured for the ports your ESX touches?


----------



## bilbus (Aug 29, 2006)

I am sure he means raid 5, as the Smart array controler only supports raid 0,1,5,6,10

How much ram are you giving the VMs, if its to low .. that will cause issues.

Try using the ATTO or HDtach benchmark tools, post the results


----------



## ed56 (Feb 19, 2009)

Here is the result from HDTach:










We are giving the SBS Server 2560MB.


----------



## ed56 (Feb 19, 2009)

Tekmazter said:


> Wow, it's not often I've found someone running RAID4. You should note that under RAID4 there is quite a bit of loss in performance when it comes to random writes due to the single parity drive.



One of our team members thinks it is Raid 5. He is trying to attempt to see if VMware or someone has a utility to change it, either on the VMware or SBS level.


----------



## bilbus (Aug 29, 2006)

wow, that is not what it should be looking like thats wierd. The burst should be faster then the read.

Try using atto, and see what the read / write times are.

really strange that the read times are so bad at the start and shoot up.

2.5gb is plenty of ram.


----------



## ed56 (Feb 19, 2009)

bilbus said:


> wow, that is not what it should be looking like thats wierd. The burst should be faster then the read.
> 
> Try using atto, and see what the read / write times are.
> QUOTE]
> ...


----------



## ed56 (Feb 19, 2009)

Tekmazter said:


> Looking at your memory ... you want to remember that what you allocate in RAM for a guest OS is only part of the story. You need to remember that the there are numerous other things (outside of ESX too) which take advantage of RAM.
> 
> http://www.kingston.com/branded/pdf_files/final_esx3_memory.pdf
> 
> ...


The ISA area is pretty stripped down, OWS, and OWA are set up. 

One of our team looked at the kinsington guide and commented:

"Kingston/VMware pdf article discusses Balloon drivers and Swap files. I wonder if this was optimized during the physical to virtual conversion or if there are opportunities to improve the setup here?"

I do note, periodically via Terminal Services, the display freezes, but keyboard commands are accepted (noted when things happen after the screen unfreezes). Makes me think that you are right about memory being an issue.


----------



## bilbus (Aug 29, 2006)

http://www.techpowerup.com/downloads/1137/ATTO_Disk_Benchmark_v2.34.html


----------



## ed56 (Feb 19, 2009)

bilbus said:


> http://www.techpowerup.com/downloads/1137/ATTO_Disk_Benchmark_v2.34.html


Here is the results of the ATTO test:










Hereis the Disk activity on ESX:










I am no expert, but seems as though ESX is reporting better resilts than the SBS

BTW, when I started ATTO, the server was nearly unresponsive (at least through terminal services) and did not return to normal until the ATTO program started showing results (the GUI of the ATTO was blank until then).

CPU usage was nearly 100% according to Task Manager (got that started, slowly, during the nearly unresponsive period)


----------



## ed56 (Feb 19, 2009)

BTW,
Having the IDE controller installed in addition to the SCSI controller should not be a problem should it?


----------



## ed56 (Feb 19, 2009)

ed56 said:


> BTW,
> Having the IDE controller installed in addition to the SCSI controller should not be a problem should it?
> 
> Actually, thinking about it, that is probably for the CD drive.


----------



## bilbus (Aug 29, 2006)

no thats fine, i will post a pic of my esx server later .. but i can tell you your results are not correct. They should start off low (mine went from 20mb to 180 or so)

Here is a pic of one of my fileservers, non esx ( i will need to login to VC later).


----------



## ed56 (Feb 19, 2009)

Thanks,

Yes a big difference.

SBS 2003 supposedly does not cache beacuse it is also the server that handles active directory, but I would doubt it would hurt performance that much.

In summary to this long thread:


SBS 2003 was migrated directly to VMware ESXi
Benchmark tests show slow disk access problem
Slow access to a virtual linux server that was located on an interal IP and accessed via a custom port through ISA (resolved with public IP taking SBS 2003 out of the picture)
Display on terminal service connections stall at times, but seem to still accept key strokes
Client machines find accessing exchange and other data stores very slow at times


----------



## bilbus (Aug 29, 2006)

i know of plenty of people who have run sbs in vmware with no problems.

Do any of the other VMs have this problem? if you dont have any other windows vms, go install one and see.


----------



## ed56 (Feb 19, 2009)

Good idea, probably just installan XP pro on one.

At least that could narrow it down.


----------



## carlos371 (Jul 13, 2009)

Hi ed56, Have you solved the issue? 
I have the same problem with SBS 2003 on ESXi 4 and I don't have any clue of a solution.

I hope you can give me same useful information about it.
Regards


----------



## bilbus (Aug 29, 2006)

what are the specs of your server?

SBS is just 2003 + exchange and sql.


----------

