Thursday 22 November 2012

Tips for Windows Server 2008 and IIS7 Tuning

Recently I moved from windows servers 2003 32 bit and IIS6 to windows servers 2008 64 bit and IIS7. My experience was rather painful, windows 2008 seems like a wild animal after windows 2003 experience. 2008 has lots networking and scalability issues and it took me quiet a while and many tweaks and hacks to achieve reliability and good performace plus scalability and solve many issues.

If you own a windows 2008 server under some significant load from the web issues described in this post maybe relevant for you. Maybe it may save you some time in desperate searching around the net for solutions...

Before we continue to windows 2008 and IIS7 tips - some relevant info about IIS7.

IIS7 breaking changes

IIS7 introduced new integrated pipeline model where asp.net pipeline is integrated into IIS which has extensibility and performance benefits. This comes together with some breaking changes to configuration and asp.net.

Follow this link to learn more:

asp.net 2 breaking changes on-iis 7

OK, now some tips...

Disable static compression

IIS7 has static compression turned on by default and dynamic compression turned off by default. My first tip is to disable static compression (website - compression in IIS7 manager).

It can be the problem when you have static XML files on you server for example. XML files become unreadable by XML parsers with IIS7 static compression. Browsers and other 3rd parties will not be able to read them anymore because XML is malformed (possible bug?).

Change max concurrent requests per CPU setting

By default IIS7 has a limit of handling 12 concurrent requests per CPU and will queue requests above this limit. If you have some significant web load and many AJAX style requests to your server - this setting maybe very restrictive and it is hard to find out the root of the problem when you server performance is suddenly degraded.

See this post to get the picture how it can cause performance issues on your server.

Some relevant info about asp.net thread usage on IIS7 here:

asp.net thread usage on IIS7 and IIS6

Thomas Marquardt advice is to change this default limit. Recommended settings:

"All of this may be a little confusing, but for nearly everyone, my recommendation is that for ASP.NET 2.0 you should use the same settings as the defaults in ASP.NET v4.0; that is, set maxConcurrentRequestsPerCPU = "5000" and maxConcurrentThreadsPerCPU="0".

This is done by adding DWORD MaxConcurrentRequestsPerCPU to the registry under

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0

called MaxConcurrentRequestsPerCPU (DWORD). This key doesn't exist by default. Or/and in aspnet.config section which overrides registry setting (also doesn't exist by default) aspnet.config is here on windows 64bit:

%windir%\Microsoft.NET\Framework64\v2.0.50727\aspnet.config

You need to add the following section under "configuration" section (here I used default values) - don't forget change maxConcurrentRequestsPerCPU to 5000.

< system.web>
    < applicationPool 
        maxConcurrentRequestsPerCPU="12" 
        maxConcurrentThreadsPerCPU="0" 
        requestQueueLimit="5000" />
< /system.web>

Disable offload network enhancements if you have network issues

Different unexplained network issue on your windows 2008 server could be related to TCP Chimney and related "networking enhancements" in windows 2008 and certain hardware vendors when hardware doesn't play well with these enhancements.

If you have networking problems similar to these below - try to disable these features.

Information about the TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access features in Windows Server 2008

Related threads:

  1. IIS 7 http status 400 errors
  2. impacts to sql server workloads
  3. the effect of tcp chimney offload on viewing network traffic

Set Connection "close" request header explicitly posting to AJAX web services

When moving from IIS6 to IIS7 integrated mode you may experience issues similar to below (POST request time-outs and request aborted errors) and all related to AJAX POST requests to IIS7 while AJAX GET requests don't have any problem.

Related threads (all unresolved)

  1. a few "Request timed out" exceptions every day
  2. "Request timed out" errors moving from IIS6 to IIS7
  3. Request Timed Out on Windows Server 2008/IIS7
  4. IIS7 Integrated Mode Request timed out

I experienced the same issue. Some HTTP POST requests got aborted in IIS7 pipeline and never reach server code. It can happen one per 1000 requests. However it may be still a significant number and if you have many users it can degrade performance for other requests.

In HTTPERR logs you can notice Timer_EntityBody errors related. POST requests usually arrive split in 2 segments: headers and following request body while GET requests arrive in one segment. On network monitor you can notice that server received POST request headers but request body never arrives to the server and after some timeout IIS aborts such request.

The solution which can help minimize the number of these errors is to set Connection "close" request header on the caller explicitly. Somehow IIS7 doesn't treat ajax POST requests very well, while IIS6 is more robust and able to handle missing Connection "close" header. Go figure.

Example for XMLHTTP javascript client:

oXmlHttp.setRequestHeader("Connection", "close");

Enjoy :)

source: http://blogs.x2line.com/al/archive/2010/01/04/3718.aspx