Amazon Windows Activation Problems

Some of my instances recently started showing up with the black background with:
Error: 0xC004F074 The Software Licensing Service reported that the computer could not be activated. The Key Management Service (KMS) is unavailable.

I recall having done this a long time ago and searching around I found some old correspondence from back in 2011 with Amazon Support that involved activating with the external IP of their KMS servers. However this was what was giving me that error. Unfortunately, I did not make good notes of that so I was going to have to figure out again. Fortunately, Internet to the rescue, I found this: http://kwjblogs.blogspot.ca/2012/01/amazon-ec2-instance-windows-activation.html

... and recalled essentially doing this:

slmgr.vbs /skms 169.254.169.250

slmgr.vbs /ato

Done. Now I also have noted it.

Amazon AWS value cannot be null, parameter name awsAccessKeyId

Today, none of my AWS Powershell scripts would work any more. They were all returning
... : Value cannot be null.
Parameter name: awsAccessKeyId

Coincidentally, my Visual Studio would keep on crashing upon startup. I eventually dug into the crash logs and realized that it had to do with the Amazon plugin as well.

Initially I thought it had to do with expired credentials or even the Windows Patch Tuesday.

However, I ended up reinstalling AWS Powershell (and with a much newer version) and the problem went away.

Amazon EC2 Powershell

I keep forgetting where to find the latest and somehow never use the right search terms. So, here is the location of the Amazon EC2 API Tools which include the Powershell cmdlets
http://aws.amazon.com/developertools/351

And for completeness, the AWS toolkit is here: http://aws.amazon.com/sdkfornet/

And now for Windows is here: http://aws.amazon.com/powershell/

ec2-describe-instances filter definitions must have format 'name=value'

I wanted to get a list of my Amazon instances with a particular filter, so I tried something simple as suggested in the ec2-describe-instances documentation: 

ec2-describe-instances --filter "instance-type=m1.small"

However, this returns the following error:

Filter definitions must have format 'name=value', but found 'instance-type'

After a bit of frustration, I finally realized that you have to pass in the quotes in the argument. This means that you need to escape the quotes in Powershell like so:

ec2-describe-instances --filter "`"instance-type=m1.small`""




Resizing Amazon EC2 Windows drives

The disk sizes that come with the Amazon Windows AMIs are a little small (35GB for a Windows 2008 R2). Since I am running demo servers I have not really been paying too much attention to laying out the disks properly. However, the 35GB disk is simply not enough so I had to resize it. This turned out to be surprisingly easy:

1. Stop the instance
2. Create a snapshot of the volume
3. Create a new volume from that snapshot, specifying the new size
4. Detach old volume from the instance
5. Attach new volume to the instance
6. Restart the instance
7. Run diskpart and enter the following commands
• select disk <disk number>
• select volume <volume number>
• extend
8. Delete the old volume (unless you want to keep it as a backup)

Simple!

Amazon EC2 Powershell environment setup

I always keep forgetting these so I figured I put them here:

$env:JAVA_HOME = 'C:\Program Files\Java\jre6'
$env:EC2_HOME = 'C:\Program Files\Amazon\ec2-api-tools'
$env:EC2_PRIVATE_KEY = '<path to private key>'
$env:EC2_CERT = '<path to cert>'
$env:Path = $env:Path + ";C:\Program Files\Amazon\ec2-api-tools\bin"

Resizing an Amazon EC2 instance

We had overbuilt some of our Amazon instances because we were not sure of the performance requirements. This was costing us lots of money so we recently started downsizing some of these instances. This is how I was doing it:

1. Confirm that Ec2SetComputerName is disabled
2. Stop the instance
3. Launch a new instance in the same zone
4. Stop the new instance
5. Attach the volumes from the old instance to the new one
6. Start the new instance
7. Terminate the old instance
8. Delete the volume that came with the new instance
9. Reassociate the Elastic IP address

Wow, that was rather cumbersome, but that is the only way to do this via the EC2 console. Looking at the API, I found the following Powershell command that replaces steps 3 to 8:
ec2-modify-instance-attribute <instance-id> --instance-type <new-instance-type>

That's it! I can't believe I did it the hard way that many times.

Disabling automatic naming of machines in Amazon EC2

I run SharePoint 2010 in my Amazon EC2 instances. As a result, I do not want my machines automatically renamed by the EC2Config service when I stop / start them. This would totally screw up SharePoint as it does not behave well with machine name changes. Fortunately, Amazon EC2 has a way to do this via the EC2Config Service Settings executable or by directly editing C:\Program Files\Amazon\Ec2ConfigSetup\config.xml and changing the Ec2SetComputerName value to Disabled. The latter is quite handy as I had to stop and restart instances recently due to some deadlock issues.

Loading and modifying the registry of a dead Amazon EC2 instance

In a recent post, I had to troubleshoot an issue with an Amazon EC2 instance not accessible via RDP after Windows Update and reboot. Back then, I didn't realize that I could have edited the registry of the unresponsive instance. Here is how to do it (Thanks to Nick Greising at Amazon for providing me with the steps). You will first need a repair instance in the same zone.

1. Note down instance information such as instance ID, attached block devices (volumes), private IP address, associated elastic IP address
2. Stop the instance
3. Detach the root volume
4. Attach the volume to repair instance 
5. Login to the repair instance
6. Bring the disk online (eg: drive E)
7. Run regedit
8. Go to HKLM
9. Select File->Load Hive
10. Browse to E:\Windows\System32\config
11. Open the hive you want (eg: SYSTEM)
12. Pick a Key Name (eg: System_old)
13. Make whatever changes you need
14. Select the root of the hive you just loaded and modified (eg: HKLM\System_old)
15. Select File->Unload Hive
16. [Optional: Note if you are running SharePoint you may need to set Ec2SetComputerName to Disabled so the machine does not change names on restart]
17. Take the disk offline
18. You can now logoff or close the connection to the repair instance
19. Detach the volume from the repair instance
20. Attach volume to original instance
21. Start instance
22. You will also need reconfigure your security groups as the internal IP address would have changed and to reassociate the Elastic IP Address.

Now, I could just plop in the steps from Amazon EC2 instance not accessible via RDP after Windows Update and reboot into step 13 and I can repair those unresponsive instances. Note that when the hive is loaded, there won't be a CurrentControlSet. However, you can look at the value of HKLM\System_old\Select\Current to determine which ControlSet to use. See the knowledgebase article What are Control Sets? What is CurrentControlSet? for details.

Amazon EC2 instance not accessible via RDP after Windows Update and reboot

For the last couple of days, I have been running into a problem where my Amazon EC2 instance is no longer accessible via remote desktop after a Windows Update and a reboot.

My process for setting up these servers is pretty straightforward: Install SQL 2008, Install SharePoint 2010 Prerequisites, Install SharePoint 2010, run Windows Update. I have done this a couple of dozen times now without any problems. Just a couple of days ago, I was finding that after the Windows Update and ensuing reboot, the server comes up, status is active, but it is not accessible via RDP or HTTP (just times out).

My first thought was that one of the more recent Windows Updates is incompatible Amazon EC2. Comparing my last successful installation with the latest Windows Update packages, I found that the following might be the culprit:

• Cumulative Security Update for Internet Explorer 8 for Windows Server 2008 x64 Edition (KB2482017)
• Platform Update Supplement for Windows Server 2008 x64 Edition (KB2117917)
• Security Update for Windows Server 2008 x64 Edition (KB2393802)
• Security Update for Windows Server 2008 x64 Edition (KB2479628)
• Security Update for Windows Server 2008 x64 Edition (KB2483185)
• Security Update for Windows Server 2008 x64 Edition (KB2485376)
• Update for Windows Server 2008 x64 Edition (KB971029)
• Windows Malicious Software Remove Tool x64 - February 2011 (KB890830)

So, I'd figure I would try again and leave out these specific updates, but upon reboot, my new instance would also be unaccessible via RDP or HTTP.

Perplexed and after a lot of searching, swearing, hair pulling, I came across this post: Avoiding RDP connectivity issues when running SharePoint 2010 on Amazon EC2. In particular, it mentions the Microsoft Article KB2379016: A computer that is running Windows Vista or Windows Server 2008 stops responding at the "Applying User Settings" stage of the logon process which describes the problem as being a deadlock in the Service Control Manager database. To break the deadlock, it is just a matter of forcing HTTP.sys depend on CryptSvc. This can be accomplished as follows:

1. Run regedit
2. Locate and the registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP
3. Create a New, Multi-string Value: DependOnService
4. Set a single value CRYPTSVC

So far so good. Maybe I'll get some sleep tonight.

 

Automated or remote installation of components on an Amazon instance

I am trying to figure out a way to either have an Amazon instance install various software components automatically right after launch or to execute commands to do those installations remotely.

I have done some searching and so far have found the following references.

AMAZON EC2 - Launch command on remote Windows machine, given admin credentials

Unattended Amazon EC2 Install Script - Unfortunately, this is for Linux. I would have to figure out how to write a Windows equivalent, if at all possible

Perhaps the Invoke-Command Powershell cmdlet would help.



Unfortunately, I don't have time to investigate further so I will have to come back to this later.

Listing public and private IP addresses of Amazon instances using Powershell

I'm a bit of a Powershell newbie so it took me a few minutes to get this figured out so I thought I'd write it down. Here is how you would list the ID, public IP address and private IP address for each Amazon instance. The script basically gets a list of instances, filters out the instance information and shows just the 3 columns.
.\ec2-describe-instances | %{ if ($_.Split()[0] -match "INSTANCE") { Write-Host $_.Split()[2,16,17] } }