$webpart = New-Object Microsoft.SharePoint.WebPartPages.SilverlightWebPart
$wpm = $web.GetLimitedWebPartManager($page.Uri.OriginalString, "Shared")
$wpm.AddWebPart($webpart, 'Header', 0)
... and got this error:
Exception calling "AddWebPart" with "3" argument(s): "<nativehr>0x80070002</nativehr><nativestack></nativestack>"
At line:1 char:16
+ $wpm.AddWebPart <<<< ($webpart, 'Header', 0)
+ CategoryInfo : NotSpecified: (:) , MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
It turns out that my $page was the wrong one which was not checked out nor editable at the time.