SharePoint Timer Service not starting up

I recently discovered that on one of my farms my SharePoint Timer Service would continuously start up and crash.
There are numerous entries like the following 2 in the System log:

The SharePoint 2010 Timer service terminated unexpectedly.  It has done this 2965 time(s).  The following corrective action will be taken in 30000 milliseconds: Restart the service.

and

The timer service could not initialize its configuration, please check the configuration database.  Will retry later.

The following is found in ULS:

SPTimerStore.InitializeTimer: SPConfigurationDatabase.RefreshCache returned SPConstants.InvalidRowVersion

and

The timer service could not initialize its configuration, please check the configuration database.  Will retry later.

I verified that my service's user context had the correct username and password.

I also cleared the timer cache (and, yes, I emptied the GUID folder and instead of deleting it).

There were a couple of DCOM 10016 errors which seemed irrelevant to this issue, but I fixed those anyway (as per http://sajiviswam.wordpress.com/2011/04/15/the-machine-default-permission-settings-do-not-grant-local-activation-permission-for-the-com-server-application-with-clsid-000c101c-0000-0000-c000-000000000046-sharepoint-2010/)

I'm still getting the same problem.

I started looking at the Fusion log as described in http://soerennielsen.wordpress.com/2009/01/14/fixing-the-timer-service-when-everything-breaks-down/ but think I'm in too deep.

What's curious is that the Timer Service stopped functioning around the time the web application was extended in order to support HTTPS and there are some new files in C:\windows\assembly\GAC_MSIL. However, I am unable to make any connection. As far as I can tell, my assemblies in the GAC are not corrupted since all other functions appear to be working.

Determining who has an open file handle on DLLs in the GAC

I recently needed to update a solution in my SharePoint farm however:

1. The update required administration and connection permissions to the User Profile Service Application (UPA)
2. Some assemblies in our solution was causing the the UPA to be inaccessible

So, my code is broken and needed update, my update is broken because it can't access UPA, my UPA is broken because of my code ... argh!

Then, I had this idea, figure out which DLLs were breaking the UPA and update those directly in the GAC, hoping that it will at least free up the UPA so the update. I figured my DLLs were reasonably compatible over the different versions. So this is what I did:

1. Grab my updated WSPs
2. Use 7-zip to open the WSPs and extract the DLLs I needed
3. Open up the GAC (C:\Windows\Assembly)
4. Copy the DLLs I need into the GAC
5. Verify, by version number, that my DLLs were indeed copied into the GAC (I have seen cases where the copy seems to work, but nothing actually happens)
6. iisreset
7. restart owstimer
8. Run my installer to update my code as normal.

This worked very well for me, but when a colleague tried it, he found that he was unable to get step 4 working. He kept getting access denied issues. It turns out that he had UAC turned on and some processes (not just iisadmin) had an open file handle on the very DLLs he needed to update. Getting around the UAC issue is just a matter of running things as administrator. To solve the file handle issue, we used Handle from Sysinternals. 

Now it was simply a matter of opening up a command prompt and doing something like this:

Handle.exe c:\Windows\assembly\gac_msil\<your DLL name>

Handle will also recurse into subdirectories for you! Now it's just a matter of stopping those processes and continuing on.

User Profile Service Application stuck in Synchronization state

I had a User Profile Synchronization job stuck in the Synchronizing state. Clicking Stop did nothing. I also tried stopping the User Profile Synchronization Service, but got the following error:

An update conflict has occurred, and you must re-try this action. The object UserProfileApplication Name=User Profile Service Application was updated by [farm admin id], in the OWSTIMER (7292) process, on machine [machine name].  View the tracing log for more information about the conflict.

I didn't see anything more helpful in the ULS logs, but I did see the same errors in the Application Log (Event Viewer)

Apparently, this is due to a well known problem with the cache in SharePoint 2010 (although it would be nice to know the cause). In any case, here's how you would fix it (as described in An update conflict has occurred, and you must re-try this action, but adapted to SharePoint 2010).

1. Stop the timer service by doing the following:
1. Click Start->Administrative Tools->Services
2. Right click SharePoint 2010 Timer, select Stop
2. Delete the contents of the folder: C:\ProgramData\Microsoft\SharePoint\Config\<some GUID>
3. Start the timer service by doing the following
1. Click Start->Administrative Tools->Services
2. Right click SharePoint 2010 Timer, select Start

Once the timer is restarted, the cache folder will rebuild itself.

I was then able to go back into Manage my User Profile Service Application and stop the running sync job.