So you want to host a Silverlight application (aka a .xap) on a server, making it available to be called from other servers. I just had to do this and figure I'll document the steps. It isn't rocket science, but here's the story.
First, you need to place a clientaccesspolicy.xml or a crossdomain.xml file at the root of your server. Read this MSDN documentation for more.
Then, you need to set the enableHTMLAccess attribute to true in the object tag:
<param name="enableHtmlAccess" value="true" />
Lastly, make sure your server is sending the right mime type for the .xap (application/x-silverlight-app). Tip o' the hat to Joe Field for that one.
And, with that, walla -- you are good to go. Here's an example of a music player being served from http://visitmix.com where I'm dynamically passing in the URL to the .mp3, which sits entirely on another server: