Sluggish site, multiple authentication prompts - caused by stack overflow in IIS application pool worker process

On one of my servers, users were reporting:

• Site was sluggish
• Sometimes several authentication prompts

I check the site, and here is what I found:

In the ULS log, some messages logged by mssdmn.exe including:

CSTS3Accessor::Init: InitRequest failed for URL http://my-server/clientxyz/Pages/Home.aspx Return error to caller, hr=80041204  [sts3acc.cxx:546]  d:\office\source\search\native\gather\protocols\sts3\sts3acc.cxx

In the System log, regular Warning events logged by WAS, ID 5011 with the message:

A process serving application pool 'SharePoint - [my-server]80' suffered a fatal communication error with the Windows Process Activation Service. The process id was '12028'. The data field contains the error number.

In the Application log, regular Information events logged by Windows Error Reporting, ID 1001 with the message:

Fault bucket , type 0

Event Name: CLR20r3

Response: Not available

Cab Id: 0

Problem signature:

P1: w3wp.exe

P2: 7.5.7601.17514

P3: 4ce7afa2

P4: Microsoft.SharePoint

P5: 14.0.0.0

P6: 4bad8a7a

P7: 5cc9

P8: 0

P9: System.StackOverflowException

P10: 

Attached files:

These files may be available here:

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_69fc9436e8b5896ded626b3ecf45dce746b517_3ed230ba

Analysis symbol: 

Rechecking for solution: 0

Report Id: 245f9c35-aaa0-11e0-96d0-f4ef7acc1a53

Report Status: 4

These all happened every 15 min - coincidentally the frequency of my incremental crawls

What can I conclude from this evidence? The W3WP is crashing. This would explain the users' various reports of sluggish behaviour and repeated authentication prompts. Now to figure out why the application pool process is crashing.

Some searching and I find this TechNet article Event ID 5011 - IIS Application Pool Availability under Troubleshoot Windows Server 2008 R2. So, I go ahead and download and install Debug Diagnostics x64. So far so good, but for some reason it runs only in Analyze Only mode. The documentation and help screens don't match the application. So, I thought I'd try the 32-bit one. This one matched the documentation, but is unable to attach to my w3wp.exe processes. Searching some more, I find that Debug Diagnostics is not supported on Windows 2008 R2 (major WTF moment!) and to follow How To: Collect a Crash dump of an IIS worker process on IIS 7.0 (and above) instead. So, I am now looking for WERCON. Well, of course this doesn't exist either, but I can read the .wer file with Notepad and this didn't give me much more information than the Application Log.

Frustrated, I give up on the diagnostics and try to do a little more experimenting. Is my FAST search crawler overloading the application pool? I create a Crawler Impact Rule to limit the crawler to one document request at a time. Still, this did not help.

Even more frustrated, I decide to see how much of an impact the crawl has. I go back to the ULS log and look at the URLs and try them out in a browser. Jackpot! The very first address I tried crashed the worker process. I try it a few more times and each time I got the same crash. 

Curious, I did more digging. The site was created at roughly the same time the WAS warning messages started showing up in the System log.

So, it turns out that one of my sites was causing the W3WP to crash and whenever the search crawler runs, it tries to crawl that site causing the crash. To workaround the problem, I added a new Crawl Rule to exclude the culprit site. This appears to have worked.

"Windows server features or role services required by this product are not enabled" when installing Office Web Apps

I was trying to install Office Web Apps on my SharePoint server farm and came across this error when I launched the installer: "Windows server features or role services required by this product are not enabled"

I checked all the prerequisites listed under the Hardware and Software Requirements (SharePoint Server 2010) and they are all there.

After a bit of head scratching, I decided to launch Server Manager and see what roles and features were installed. I clicked on Add Feature and it complained that another user was still configuring a role. It turned out that when I was enabling the Desktop Experience Feature using a different account, however that required a reboot and was waiting for that account to login again in order to continue the installation of Ink and Handwriting Services. Once I logged in, that feature was successfully enabled.

Then the prerequisite check succeeded.

Listing sites, their creators, template and date created

Just a simple Powershell script using the splat operator (which is quite new to me).


(Get-SPSite myurl).AllWebs | Select-Object url, @{label="Creator"; Expression={$_.Author}}, @{label="Template"; Expression={$_.WebTemplate}}, @{label="Date Created"; Expression={$_.Created}} | Format-Table -AutoSize


This returns a table of URL, Creator, Template and Date Created of each site in the site collection specified by myurl. Note that your screen needs to be wide enough to see the whole table otherwise some columns may not display.

Host name validation failed when configuring FAST Search Server 2010 for SharePoint

This was extremely frustrating while trying to figure it out. Now that I have it is pretty obvious. I was configuring a FAST Search Server 2010 for SharePoint. In the Configuration Wizard at the Server settings screen, it asks for a Server name. The documentation says that this is the FQDN of the server. All my servers have their own name but also have an alias pointing to them. Of course, I always refer to these servers via their alias. So when it asked me for the server name, I naturally entered the alias. The wizard rejects this with a dialog box "Host name validation failed - Please provide a valid server name". I checked firewall settings, event logs and found nothing. It finally dawned on me to actually use the FQDN (easiest way to confirm this is to right click My Computer and select Properties) and it worked.

The strange thing is that if the wizard is able to determine an incorrect FQDN, then why couldn't it just set that value?

Cannot open database User Profile Service Application_SyncDB ... The login failed

Some of my servers were mysteriously no longer synchronizing the user profiles after some time. Upon further investigation this appeared to happen after a server reboot. Here is an error found in the Application Log with a source of Forefront Identity Manager:


.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Cannot open database "User Profile Service Application_SyncDB_35eb65afa42c4f8e9ece6a33e9348849" requested by the login. The login failed.
Login failed for user '[my user]'.
   at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
   at Microsoft.ResourceManagement.Data.DatabaseConnection.Open(SqlConnection connection)
   at Microsoft.ResourceManagement.Data.DatabaseConnection.Open(DataStore store)
   at Microsoft.ResourceManagement.Data.TransactionAndConnectionScope..ctor(Boolean createTransaction, IsolationLevel isolationLevel, DataStore dataStore)
   at Microsoft.ResourceManagement.Data.TransactionAndConnectionScope..ctor(Boolean createTransaction)
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.Initialize()
   at Microsoft.ResourceManagement.WebServices.ResourceManagementServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)

A few seconds later in the log

Starting up database 'User Profile Service Application_SyncDB_35eb65afa42c4f8e9ece6a33e9348849'.

So, it looks like the Forefront Identity Manager Service is starting up before SQL has started the necessary databases. This appears to cause it to crash. Now that I figured out the problem, I do a search to figure out how to delay the start of the FIM Service and find this article: Troubleshoot User Profile Synchronization Service start issues (SharePoint Server 2010). Scrolling down to the bottom, there is a section titled "User Profile Synchronization service cannot start after a server restart" This suggests using the Services Manager to change the startup of the FIM services to "Delayed Start". (Note that this contradicts the Important note at the top of the article stating "Starting the FIM services manually or changing the FIM service configuration is not supported.").

Managed Service Accounts not supported by SQL Server

I have been setting up a new Active Directory Domain Services running in Windows Server 2008 R2 functional level. According to the documentation What's New in Services Accounts in Windows Server 2008 and Windows 7: "The managed service account is designed to provide crucial applications such as SQL Server and IIS with the isolation of their own domain accounts, while eliminating the need for an administrator to manually administer the service principal name (SPN) and credentials for these accounts." So I create an account to run the SQL services and install SQL server. However, when I try to pick the service account, my SQL user will not show up in the picker at all. Searching around some more, I come across the Managed Service Accounts Frequently Asked Questions (FAQ). It states: "The use of managed service accounts with Microsoft SQL Server is not supported." Argh!

Using Alternate Access Mappings to "change" the host header of a SharePoint web application

It looks like there is no easy way to change the host header of a SharePoint web application. This is unfortunate as I want to create many virtual instances of SharePoint and have precanned data for demo purposes. To get around this, I figured I can create the initial copy and then use Alternate Access Mappings (AAM) to provide another way to access my SharePoint sites. It looks like all I have to do is the following:

1. Go to Central Administration->System Settings->Configure Alternate Access Mappings
2. Select Edit Public URLs
3. Change the default setting to the new URL using the new host header
4. Go to IIS Manager
5. Locate the IIS web site that has the binding for the old host header
6. Add a new binding with the new host header
7. Go to Central Adminstration->Application Management->Manage Service Applications
8. Manage User Profile Service Application
9. Click Setup My Sites
10. Change the My Site Host location using the new host header

That seems to do do the trick. I hope I am not missing anything.

Here are some references:

MSDN Video: Microsoft SharePoint 2010: Alternate Access Mappings (Level 200)

What every SharePoint administrator needs to know about Alternate Access Mappings (Part 1) (Part 2) (Part 3)