UserProfileApplicationNotAvailableException System.TimeoutException

I have a service the frequently accesses the User Profile Service Application. Intermittently, I would get the following:

Error generating SyndicationItem for ActivityEvent 0B275DAC41FD141CB2382AAE5987540D  Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: System.TimeoutException    
 at Microsoft.Office.Server.UserProfiles.ProfileDBCacheServiceClient.GetUserData(UserSearchCriteria searchCriteria)    
 at Microsoft.Office.Server.UserProfiles.UserProfileCache.GetUserData(UserProfileManager objManager, Nullable`1 recordId, Guid gAcct, String strAcct, Byte[] bSid, String strEmail, Boolean doNotResolveToMasterAccount)    
 at Microsoft.Office.Server.UserProfiles.UserProfile.RetrieveUser(String strAcct, Guid gAcct, Byte[] bSid, Nullable`1 recordId, Boolean doNotResolveToMasterAccount, Boolean loadFullProfile)    
 at Microsoft.Office.Server.UserProfiles.UserProfile..ctor(UserProfileManager objManager, Int64 recordId)    
 at Microsoft.Office.Server.UserProfiles.UserProfileManager.GetUserProfile(Int64 recordId)

I tried the following to confirm:

$app = Get-SPWebApplication
$site = Get-SPSite $app.Url
$context = Get-SPServiceContext $site
$upm = New-Object Microsoft.Office.Server.UserProfiles.UserProfileManager($context)
$upm.UserExists('someuser')

... and it just hangs.

Since this is a demo machine, there could be many factors:

1. Not enough resources

2. Application pool crash

3. Database not responding

4. General UPA flakiness

5. All of the above (and possibly pointing to #1 as the root cause).

I tried stop/start on the UPS and UPSS, then an iisreset. This didn't fix the problem. However, a reboot did.

Further looking at the event logs, I do see the following events from MsiInstaller that might be suspicious:

Detection of product '{90140000-104C-0000-1000-0000000FF1CE}', feature 'PeopleILM', component '{1C12B6E6-898C-4D58-9774-AAAFBDFE273C}' failed.  The resource 'C:\Program Files\Microsoft Office Servers\14.0\Service\Microsoft.ResourceManagement.Service.exe' does not exist.
Detection of product '{90140000-104C-0000-1000-0000000FF1CE}', feature 'PeopleILM' failed during request for component '{9AE4D8E0-D3F6-47A8-8FAE-38496FE32FF5}'
Failed to connect to server. Error: 0x80070005
Detection of product '{90140000-104C-0000-1000-0000000FF1CE}', feature 'PeopleILM', component '{1C12B6E6-898C-4D58-9774-AAAFBDFE273C}' failed.  The resource 'C:\Program Files\Microsoft Office Servers\14.0\Service\Microsoft.ResourceManagement.Service.exe' does not exist.
Detection of product '{90140000-104C-0000-1000-0000000FF1CE}', feature 'PeopleILM' failed during request for component '{1681AE41-ADA8-4B70-BC11-98A5A4EDD046}'
Failed to connect to server. Error: 0x80070005

... followed by one from FIMSynchronizationService:

The management agent "MOSS-3692cc7e-f3bf-4090-ae09-a552d3c61b7e" completed run profile "MOSS_DELTAIMPORT_4adf3894-708b-4ae2-ab9f-60438a49cae1" with a delta import or delta synchronization step type. The rules configuration has changed since the last full import or full synchronization.
 
User Action
To ensure the updated rules are applied to all objects, a run with step type of full import and full synchronization should be completed.

It was patch Tuesday a couple of days ago, perhaps that had a factor since I have auto updates enabled (which curiously did not install).

KB2560890 update without SharePoint SP1

It appears that the Security Update for Microsoft SharePoint Server 2010 (KB2560890) breaks the User Profile Service Application if you don't have SP1 installed for SharePoint 2010. The error I am seeing is:
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.ResourceManagement, Version=4.0.2450.34, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified
According to UserProfile not working after security update KB2560890, it seems like others are having the same issue.

The file list for KB2560890 refers to version 4.0.2450.34 of various Microsoft.resourcemanagement.* files, but does not specifically list Microsoft.ResourceManagement.dll.

In my GAC, I see the following:
Microsoft.ResourceManagement 4.0.2450.5

Microsoft.ResourceManagement.Service 4.0.2450.11

The machine had rebooted from the Windows Update. So, on a whim I decided to try shutting down and restarting the User Profile Service and the User Profile Synchronization Service (remember to iisreset if Central Administration is on the same machine).

I then checked back in my GAC and now I see the following:

Microsoft.ResourceManagement 4.0.2450.5
Microsoft.ResourceManagement 4.0.2450.34

Microsoft.ResourceManagement 4.0.2450.11
Microsoft.ResourceManagement.Service 4.0.2450.34

Microsoft.ResourceManagement.Service 4.0.2450.11

I can also manage my User Profile Service Application!

There's some sort of voodoo here, but as far as I can tell it works. Perhaps there was a continuation task after the Windows Update reboot that required yet another reboot. Who knows?!

Argh! Spoke too soon. If I go to Setup My Sites, I now get the following:
Cannot create an object of type 'Microsoft.SharePoint.Portal.WebControls.LocStringId' from its string representation 'SiteAdminPersonalSite_SocialSecurityTrimmerEnabledSectionTitle_Text' for the 'TitleLocId' property.

Time to give up and just do SP1 sometime in the future.

Cannot open database User Profile Service Application_SyncDB ... The login failed

Some of my servers were mysteriously no longer synchronizing the user profiles after some time. Upon further investigation this appeared to happen after a server reboot. Here is an error found in the Application Log with a source of Forefront Identity Manager:


.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Cannot open database "User Profile Service Application_SyncDB_35eb65afa42c4f8e9ece6a33e9348849" requested by the login. The login failed.
Login failed for user '[my user]'.
   at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
   at Microsoft.ResourceManagement.Data.DatabaseConnection.Open(SqlConnection connection)
   at Microsoft.ResourceManagement.Data.DatabaseConnection.Open(DataStore store)
   at Microsoft.ResourceManagement.Data.TransactionAndConnectionScope..ctor(Boolean createTransaction, IsolationLevel isolationLevel, DataStore dataStore)
   at Microsoft.ResourceManagement.Data.TransactionAndConnectionScope..ctor(Boolean createTransaction)
   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RegisterService(String hostName)
   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.Initialize()
   at Microsoft.ResourceManagement.WebServices.ResourceManagementServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)
   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)

A few seconds later in the log

Starting up database 'User Profile Service Application_SyncDB_35eb65afa42c4f8e9ece6a33e9348849'.

So, it looks like the Forefront Identity Manager Service is starting up before SQL has started the necessary databases. This appears to cause it to crash. Now that I figured out the problem, I do a search to figure out how to delay the start of the FIM Service and find this article: Troubleshoot User Profile Synchronization Service start issues (SharePoint Server 2010). Scrolling down to the bottom, there is a section titled "User Profile Synchronization service cannot start after a server restart" This suggests using the Services Manager to change the startup of the FIM services to "Delayed Start". (Note that this contradicts the Important note at the top of the article stating "Starting the FIM services manually or changing the FIM service configuration is not supported.").

Moving the User Profile Service Application Synchronization Database

So, the problems with my Sync DB as I mentioned in a previous post finally caught up to me. Running low on disk space, we decided to add another block of disk space to the server and move the Sync DB over there.

Adding the volume on Amazon EC2 was very straightforward. It was just a matter creating a new volume, attaching it to the server, enabling and then formatting the drive. This is covered quite nicely in Attaching an EBS Volume to EC2.

Now with the drive ready, I needed to move the DB files over. The instructions for Moving User Databases are pretty straightforward. However, I ran into some hiccups - I had open connections to the database (from when I was troubleshooting the database size :-) ) and the command to take the database offline simply waits until those connections are closed and you're none the wiser. Also, the new files / locations need to have the right permissions otherwise you can't bring the database back online. So, here are my updated steps.

1. Close all connections to your database. You can check for open connections by executing an sp_who2
2. Take the database offline by running the following:
   ALTER DATABASE [Sync DB] SET OFFLINE
3. Move the files to the new location
4. Update the new location in the system catalog as follows (make sure you use your own location!)
   ALTER DATABASE [Sync DB] MODIFY FILE ( NAME = 'Sync DB', FILENAME = 'E:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\Sync DB.mdf' );
   ALTER DATABASE [Sync DB] MODIFY FILE ( NAME = 'Sync DB_log', FILENAME = 'E:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\Sync DB_log.LDF' );
5. Make sure that the database user has the correct permissions to the database files. I just copied the original settings which is giving the SQLServerMSSQLUser group full control access to the DATA folder (and the new database files). 
6. Bring the database back online by running the following:
   ALTER DATABASE [Sync DB] SET ONLINE
7. Verify the file change by running the following:
   SELECT  name, physical_name FROM sys.master_files WHERE sys.master_files.database_id = DB_ID(N'Sync DB')

I still haven't gotten to the root cause of the database file being so large. However, I just needed to buy some time for now.

User Profile Service Application Synchronization DB size

I have a SharePoint instance that is running out of disk space. Doing some poking around, I found that the User Profile Service Application Sync_DB has grown to almost 7GB. So, I ran the following SQL query to get more insight into what is going on:
exec sp_MSforeachtable @command1="EXEC sp_spaceused '?'"
Looking through the results I see that the culprits are as follows:

name	rows	reserved	data	index_size	unused
ObjectsInternal	3156618	2089728 KB	800296 KB	1288776 KB	656 KB
InstanceData	177997	4698144 KB	4697048 KB	296 KB	800 KB

I decided to take a peek inside these tables and it looks like there are thousands of objects created every day. So what are these and do they get cleared out any time?

I found a similar question asked: User Profile Service Application SyncDB Database Size. There is an response that is marked as an answer, but it merely leads us to recommended provisioning sizes of these databases to be medium to large (100GB to 1TB). There is no explanation why. This is quite bogus as I only have 75 profiles.

Argh!

I have posted a followup here: Moving the User Profile Service Application Synchronization Database

User Profile Service Application stuck in Synchronization state

I had a User Profile Synchronization job stuck in the Synchronizing state. Clicking Stop did nothing. I also tried stopping the User Profile Synchronization Service, but got the following error:

An update conflict has occurred, and you must re-try this action. The object UserProfileApplication Name=User Profile Service Application was updated by [farm admin id], in the OWSTIMER (7292) process, on machine [machine name].  View the tracing log for more information about the conflict.

I didn't see anything more helpful in the ULS logs, but I did see the same errors in the Application Log (Event Viewer)

Apparently, this is due to a well known problem with the cache in SharePoint 2010 (although it would be nice to know the cause). In any case, here's how you would fix it (as described in An update conflict has occurred, and you must re-try this action, but adapted to SharePoint 2010).

1. Stop the timer service by doing the following:
1. Click Start->Administrative Tools->Services
2. Right click SharePoint 2010 Timer, select Stop
2. Delete the contents of the folder: C:\ProgramData\Microsoft\SharePoint\Config\<some GUID>
3. Start the timer service by doing the following
1. Click Start->Administrative Tools->Services
2. Right click SharePoint 2010 Timer, select Start

Once the timer is restarted, the cache folder will rebuild itself.

I was then able to go back into Manage my User Profile Service Application and stop the running sync job. 

Random error with User Profile Service Application remedied by reboot

Some of my servers are running into an intermittent problem. Every so often, the User Profile Service Application is not accessible. A reboot seems to fix the issue, but I don't know why and haven't found anything conclusive out there. Here's what I am seeing in the ULS logs:


User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814
User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814
User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814
User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814
User Profile Application Proxy failed to retrieve partitions from User Profile Application: Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_PartitionIDs()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.IsAvailable(SPServiceContext serviceContext) a99603f0-3dc1-4312-af7a-5e2240082814






Microsoft.Office.Server.UserProfiles.UserProfileApplicationNotAvailableException: No User Profile Application available to service the request. Contact your farm administrator.    at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.get_ApplicationProperties()
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.CheckAdministrationAccess(UserProfileApplicationAdminRights rights, Boolean requireAllRights)
     at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.CheckAdministrationAccess(UserProfileApplicationAdminRights rights)
     at Microsoft.Office.Server.UserProfiles.ProfileManagerBase.CanManagePeople(UserProfileApplicationProxy userProfileApplicationProxy)
     at Microsoft.Office.Server.UserProfiles.ProfileManagerBase.get_IsProfileAdmin()
     at Microsoft.Office.Server.UserProfiles.UserProfileManager..ctor(SPServiceContext serviceContext, Boolean IgnoreUserPrivacy, Boolean backwardCompatible)
     at Microsoft.Office.Server.UserProfiles.UserProfileManager..ctor(SPServiceContext serviceContext, Boolean IgnoreUserPrivacy)
     at NewsGator.Social.Library.Users.CachingUserProfileManager.<>c__DisplayClass10.<GetCachedInstance>b__f()
     at NewsGator.Social.Library.Caching.NGCache.RetrieveAndCache[T](String cacheKey, TimeSpan ts, Func`1 dataLoader)
     at NewsGator.Social.Library.Users.CachingUserProfileManager.GetCachedInstance(Boolean ignorePrivacy)
     at NewsGator.Social.Library.UserManager..ctor()
     at NewsGator.Social.Web.ClientRuntime.RegisterPageScripts(Page page, ScriptManager scriptManager)
     at NewsGator.Social.Web.CommunitiesOverviewWebPart.CreateChildControls()
     at System.Web.UI.Control.EnsureChildControls()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Control.PreRenderRecursiveInternal()
     at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) a99603f0-3dc1-4312-af7a-5e2240082814






Exception occured while connecting to WCF endpoint: System.ServiceModel.CommunicationException: Could not connect to http://[hostname removed]:32843/06721706d40049bf9fe12b94be8f1912/ProfilePropertyService.svc. TCP error code 10048: Only one usage of each socket address (protocol/network address/port) is normally permitted [ip address removed]:32843.
  ---> System.Net.WebException: Unable to connect to the remote server
 ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted [ip address removed]:32843
     at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
     at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
     --- End of inner exception stack trace ---
     at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
     at System.Net.HttpWebRequest.GetRequestStream()
     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
     --- End of inner exception stack trace ---
    Server stack trace: 
     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
     at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
     at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
     at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)
     at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]: 
     at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
     at System.Runtime.Remoting.Proxies.RealProxyPrivateInvoke(MessageData& msgData, Int32 type)
     at Microsoft.Office.Server.UserProfiles.IProfilePropertyService.GetProfileProperties()
     at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.<>c__DisplayClass1.<GetProfileProperties>b__0(IProfilePropertyService channel)
     at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock)
     at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock)