Active Directory Primary Domain Controller went belly up

I was alerted to a login problem on one of our servers by a user. I found a Event 537 with status code 0xC000005E logged in the event viewer. Here is the corresponding KB article: http://support.microsoft.com/kb/908355

It turns out our primary active directory domain controller had died. I don't really have any experience with setting up AD servers so needed a little help with this. Luckily this was quite useful: Installing an Additional Domain Controller by Using the Graphical User Interface (GUI).

So, I started doing this, but then ran into a problem when running the Active Directory Domain Services Installation Wizard (dcpromo.exe): You will not be able to install a writable replica domain controller at this time because the RID master [my primary DC] is offline.
After clicking on No, the following popup was displayed: The transfer of the operations master role cannot be performed because: The requested FSM operation failed. The current FSMO holder could not be contacted.
Well, duh! As I was saying, the primary DC is down. Doing some more searching I found this: Seizing FSMO Roles.

So, I went back to my old backup DC, seized all FSMO roles, then made my new box the new backup DC as I originally started out doing. Now everything is okay again. I just have to update all my DNS search settings since the new box has a different IP address.

Listing public and private IP addresses of Amazon instances using Powershell

I'm a bit of a Powershell newbie so it took me a few minutes to get this figured out so I thought I'd write it down. Here is how you would list the ID, public IP address and private IP address for each Amazon instance. The script basically gets a list of instances, filters out the instance information and shows just the 3 columns.
.\ec2-describe-instances | %{ if ($_.Split()[0] -match "INSTANCE") { Write-Host $_.Split()[2,16,17] } }

Random error with User Profile Service Application remedied by reboot

Some of my servers are running into an intermittent problem. Every so often, the User Profile Service Application is not accessible. A reboot seems to fix the issue, but I don't know why and haven't found anything conclusive out there. Here's what I am seeing in the ULS logs:


User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814
User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814
User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814
User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814
User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814






Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.    at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.CheckAdministrationAccess(UserProfileApplicationAdminRights rights, Boolean requireAllRights)
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.CheckAdministrationAccess(UserProfileApplicationAdminRights rights)
     at Microsoft.Office.Server.UserProfiles.ProfileManagerBase.CanManagePeople(UserProfileApplicationProxy userProfileApplicationProxy)
     at Microsoft.Office.Server.UserProfiles.ProfileManagerBase.get_IsProfileAdmin()
     at Microsoft.Office.Server.UserProfiles.UserProfileManager..ctor(SPServiceContext serviceContext, Boolean IgnoreUserPrivacy, Boolean backwardCompatible)
     at Microsoft.Office.Server.UserProfiles.UserProfileManager..ctor(SPServiceContext serviceContext, Boolean IgnoreUserPrivacy)
     at NewsGator.Social.Library.Users.CachingUserProfileManager.<>c__DisplayClass10.<GetCachedInstance>b__f()
     at NewsGator.Social.Library.Caching.NGCache.RetrieveAndCache[T](String cacheKey, TimeSpan ts, Func`1 dataLoader)
     at NewsGator.Social.Library.Users.CachingUserProfileManager.GetCachedInstance(Boolean ignorePrivacy)
     at NewsGator.Social.Library.UserManager..ctor()
     at NewsGator.Social.Web.ClientRuntime.RegisterPageScripts(Page page, ScriptManager scriptManager)
     at NewsGator.Social.Web.CommunitiesOverviewWebPart.CreateChildControls()
     at System.Web.UI.Control.EnsureChildControls()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) a99603f0-3dc1-4312-af7a-5e2240082814






Exception occured while connecting to WCF endpoint: System.ServiceModel.CommunicationException: Could not connect to http://[hostname removed]:32843/06721706d40049bf9fe12b94be8f1912/ProfilePropertyService.svc. TCP error code 10048: Only one usage of each socket address (protocol/network address/port) is normally permitted [ip address removed]:32843.
  ---> System.Net.WebException: Unable to connect to the remote server
 ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted [ip address removed]:32843
     at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
     at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
     --- End of inner exception stack trace ---
     at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
     at System.Net.HttpWebRequest.GetRequestStream()
     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
     --- End of inner exception stack trace ---
    Server stack trace: 
     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
     at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
     at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
     at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
     at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]: 
     at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
     at System.Runtime.Remoting.Proxies.RealProxyPrivateInvoke(MessageData& msgData, Int32 type)
     at Microsoft.Office.Server.UserProfiles.IProfilePropertyService.GetProfileProperties()
     at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.<>c__DisplayClass1.<GetProfileProperties>b__0(IProfilePropertyService channel)
     at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock)
     at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock) 

Access denied errors from crawl

My crawl log was showing a couple of top level errors with this message:

Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled

It turns out that I should have disabled the loopback check as mentioned in SharePoint disable loopback check (DisableLoopbackCheck in registry). This is just a matter of setting a DWORD value for DisableLoopbackCheck in the registry under HKLM\SYSTEM\CurrentControlSet\Contro\Lsa

Curious, I did some more research on what DisableLoopbackCheck actually does and came across KB896861 which mentions that the preferred approach would be to create a MultiString Value BackConnectionHostNames in the registry under HKLM\SYSTEM\CurrentControlSet\Contro\Lsa\MSV1_0 and add a list of your host names.

I also found this reference about DisableLoopbackCheck & SharePoint: What every admin and developer should know. This explains why the latter is better for security reasons.

Unfortunately, when I tried the latter, the http start address crawled properly but the sps3 one would not. Since I was short on time and this was just a demo server I decided to cheat and go with the former. Maybe someday I will reinvestigate this.

The local farm is not accessible. Cmdlets with FeatureDependencyId are not registered

I ran into this error today when starting up the SharePoint 2010 Management Shell (ie: Powershell): The local farm is not accessible. Cmdlets with FeatureDependencyId are not registered.

A quick search shows that many people have run into this before. It has to do with the user with which you are running Powershell does not have the required permissions for the SharePoint_Config database

You can correct this either by giving the user enough permissions through SQL Server or run Add-SPShellAdmin. Both options require access to an account that already has the rights.

SharePoint KB983497, KB2281364, FIMSynchronizationService, User Profile Service Application synchronization issues

So, I needed to install hotfix KB983497 on my SharePoint 2010 server today because I needed a particular bug fixed. This was pretty straightforward. However, it breaks various things with user profile. For example, if you go to Manage User Profiles, you get a "File Not Found". You will also see something like this in the ULS log:

Exception: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.ResourceManagement, Version=4.0.2450.9, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. File name: 'Microsoft.ResourceManagement, Version=4.0.2450.9, Culture=neutral, PublicKeyToken=31bf3856ad364e35' at Microsoft.Office.Server.UserProfiles.UserProfileConfigManager.InitializeIlmClient(String ILMMachineName, Int32 FIMWebClientTimeOut) at Microsoft.Office.Server.UserProfiles.UserProfileConfigManager..ctor(UserProfileApplicationProxy userProfileApplicationProxy, Guid partitionID) at Microsoft.SharePoint.Portal.WebControls.UserProfileServiceStatisticsWebPartBase.Loa...

The good thing is that the KB clearly states that you also need to install hotfix KB2281364 to fix this problem. This was also quite straightforward.

So, I go into the User Profile Service Application to create a synchronization connection, but I get this:

originalMaConfiguration.Create or UpdateResource failed at step Create Connection --- Microsoft.ResourceManagement.WebServices.Faults.ServiceFaultException: Unable to process Create message at Microsoft.ResourceManagement.WebServices.Client.ResourceTemplate.CreateResource() at Microsoft.ResourceManagement.WebServices.ResourceManager.CreateResource() at Microsoft.Office.Server.UserProfiles.DirectoryServiceConnection.UpdateInternal()

and I noticed that the Forefront Identity Management Service no longer starts. There are a bunch events similar to this one:

Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManagerException: Forefront Identity Management Service does not support workflows of type 'Microsoft.ResourceManagement.Workflow.Activities.SequentialWorkflow, Microsoft.ResourceManagement, Version=4.0.2450.9, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

So, apparently, this one does not affect the profile synchronization feature, so you can ignore it. More about this at KB2432041

But also, it complains about a bunch of things including the following in the Application Log:

The Forefront Identity Manager Service cannot connect to the SQL Database Server.

The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.

Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, verify the connection information has been configured properly. This configuration is stored in the Windows Registry.

To resolve this, I followed the suggestion at FIMSynchronizationService and Sharepoint 2010 August update which goes like this:

1. Go to Central Administration->System Settings->Manage Services on Server
2. Stop the User Profile Service
3. Stop the User Profile Synchronization Service
4. Start the User Profile Service
5. Start the User Profile Synchronization Service and wait for it to finish starting

This will cause the 2 FIM Services to be started.

However, now I get the following:

UserProfileServiceUserStatisticsWebPart:LoadControl failed, Exception: System.IO.FileLoadException: The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) at Microsoft.Office.Server.UserProfiles.UserProfileConfigManager.InitializeIlmClient(String ILMMachineName, Int32 FIMWebClientTimeOut) at Microsoft.Office.Server.UserProfiles.UserProfileConfigManager..ctor(UserProfileApplicationProxy userProfileApplicationProxy, Guid partitionID) at Microsoft.SharePoint.Portal.WebControls.UserProfileServiceStatisticsWebPartBase.LoadControl(Object sender, EventArgs e)

To fix this, I found a post on UserProfileServiceUserStatisticsWebPart:LoadControl failed. To resolve, just do the following:

1. Verify that the Forefront Identity Management Synchronization Service and the Forefront Identity Manager Service are running. This should have been started by the steps above, but you can verify it anyway by going to Start->Administrative Tools->Services
2. Run iisreset, since Central Administration and User Profile Synchronization Service are provisioned on the same server. (I'm not sure why, though)
3. Now go into User Profile Service Application and continue as normal.

At some point, I should probably read how to Configure profile synchronization (SharePoint Server 2010) and maybe Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization and Configuring User Profiles in SharePoint Server 2010, but no time now - must get some sleep.