FDS-TEAM
Linux, Windows, Programming and more...
Pipelight: using Silverlight in Linux browsers
16 Aug 2013 21:23 CEST  - 240 comment(s)

Today we want to present you our latest project Pipelight, which allows to run your favorite Silverlight application directly inside your Linux browser. The project combines the effort by Erich E. Hoover with a new browser plugin that embeds Silverlight directly in any Linux browser supporting the Netscape Plugin API (Firefox, Chrome / Chromium, Midori, Opera, …). He worked on a set of Wine patches to get Playready DRM protected content working inside Wine and afterwards created an Ubuntu package called Netflix Desktop. This package allows one to use Silverlight inside a Windows version of Firefox, which works as a temporary solution but is not really user-friendly and moreover requires Wine to translate all API calls of the browser. To solve this problem we created Pipelight.

Pipelight consists out of two parts: A Linux library which is loaded into the browser and a Windows program started in Wine. The Windows program, called pluginloader.exe, simply simulates a browser and loads the Silverlight DLLs. When you open a page with a Silverlight application the library will send all commands from the browser through a pipe to the Windows process and act like a bridge between your browser and Silverlight. The used pipes do not have any big impact on the speed of the rendered video since all the video and audio data is not send through the pipe. Only the initialization parameters and (sometimes) the network traffic is send through them. As a user you will not notice anything from that "magic" and you can simply use Silverlight the same way as on Windows, like you can see on the following screenshot:

Silverlight 5 via Wine running in a native Google Chrome instance on Ubuntu. The screenshot was taken on this page and is showing the movie Big Buck Bunny (© copyright 2008, Blender Foundation) via Silverlight Smooth Streaming.

You can easily try it out on your own with our prepared and ready to use PPA. Just follow the instructions in the next section. If you are interested in story behind the project and how all the things work internally, you just need to scroll down a bit.

UPDATE 2014-01-19: Version 0.2.4.1 is now available, take a look at https://launchpad.net/pipelight/+announcement/12364 for the changes.
UPDATE 2013-12-14: Version 0.2.3 is now available, take a look at https://launchpad.net/pipelight/+announcement/12266 for the changes.
UPDATE 2013-11-14: Version 0.2.2 is now available.
UPDATE 2013-10-31: Version 0.2.1 is now available, take a look at https://launchpad.net/pipelight/+announcement/12115 for the changes.
UPDATE 2013-10-15: Version 0.2.0 is now available, take a look at https://launchpad.net/pipelight/+announcement/12054 for the changes.
UPDATE 2013-09-16: Version 0.1-4 is now available, take a look at https://launchpad.net/pipelight/+announcement/11996 for the changes.
UPDATE 2013-09-08: Version 0.1-3 is now available.
UPDATE 2013-08-28: Version 0.1-2 is now available, take a look at https://launchpad.net/pipelight/+announcement/11868 for the changes.

The instructions have been moved to a separate page.

# 2. Help, its not working / I found a bug!

This section has been moved to a different page.

# 3. Story behind the project

When we started our project there was no Silverlight 5 DRM support in Wine, only Silverlight 4 could be used by applying the patches of Erich E. Hoover. Unfortunately since the release of his patches, some of the streaming websites like Maxdome upgraded to Silverlight 5 so our first goal was to get version 5 to work. Whenever you probably have used Wine before, you might have noticed the whole bunch of "fixme" and "stub" messages scrolling down when executing native Windows programs. These messages are caused by the fact that Wine is still far away from being a complete replacement for Windows. Of course it would be possible to solve all these issues and get everything working on Wine, but this would need infinite amounts of resources and moreover is an increasingly difficult task, especially because not all Windows API commands are documented exactly. Besides these obvious messages which show that there is something missing, there are also cases where there isn't any visible output.

At the beginning we first focused on the most obvious output messages, and tried to write quick & dirty implementations for the the missing functions. In most cases it seemed to be a success at first, for example when new error messages appeared at a later point in the program. But even after fixing more of them, nothing had changed. We continued trying different methods to find the issue and at one point even were convinced, that the newer versions of Silverlight must have some kind of active protection against Wine and/or virtual machines. We still don't know if this is probably the case, but nevertheless Michael by accident discovered the crucial point where it failed in our previous attempts.

We always though the errors were related to the kind of DRM protection which was used, but as it turns out they weren't. Just to try out the effect Michael modified the value of one of the arguments passed to the Silverlight applet - enableGPUAcceleration to false - and although we didn't expect this would change anything, suddenly Silverlight 5 DRM worked almost perfectly. The only remaining problem - one of the streaming pages didn't have any sound - finally was solved by a smart guess of Erich E. Hoover, who suggested to try another patch file which just adds a single word in one of the million lines of code! A string comparison did not work as expected and Silverlight was unable to select the correct audio steam.

Our second goal was to find a way to circumvent simulating a whole browser environment and just to execute the Silverlight plugin. As Silverlight uses the NPAPI interface (except for IE) and nearly every modern browser (both Windows and Linux) supports this plugin API, we decided to write some kind of wrapper plugin. Since it's not directly possible to load a Win32 DLL in an Linux process, we created on the one side a Linux compatible NPAPI plugin (internally just a shared library) and on the other side a Windows program which simulates some kind of "browser" to load the Silverlight library. Here the term browser just means that we are providing the browser side part of the NPAPI to Silverlight.

It didn't take very long until we had the first version finished, but as usual, nothing worked at the beginning. After another additional day of debugging we finally figured out what was wrong: Silverlight assumes that the COM-Library is already initialized when loading the plugin! This is not part of the API, but if you take a look at the source code of Webkit (this file - line 65), you will see that there are 21 work arounds for plugins which do not behave according to the API (which seems to be the majority of the plugins). Nevertheless after adding one simple line of code we got Silverlight to start:

CoInitialize(NULL);


As a next step we continued implementing more and more features of the API till it worked with all our testing websites. The last and most difficult step was to embed the window directly into the native browser, which required additional patching of Wine. The final version is now able to transparently redirect and convert most API calls between the browser and plugin, such that it is possible to use Silverlight just like any other NPAPI browser plugin.

The main advantage over the previous solution to run Firefox in Wine is that this approach runs much more smooth and doesn't require as much resources. Moreover it might be easier to just watch movies in your favorite browser than switching each time you want to watch something.

# 4. Development

If you are programmer yourself you may also be interested in the technical details and how everything works internally. In this section we will describe the way Pipelight works from a programmers perspective. Please note that we cannot describe here everything completely in detail, but as its open source software, you can easily take a look at it yourself.

Like in any other open source project patches and bug fixes are welcome. Just open a new bug report here and add your patch file as attachment under extras. We will take a look at it and decide whether we apply it to the repository or if the patch needs further improvements.

If you have questions or just want to know more about the internals of Pipelight, you are free to join our IRC channel #pipelight on Freenode.

## 4.1. Quick overview

The project is split in two parts: a shared library which is loaded into your Linux browser and a pluginloader which is executed in Wine. The shared library offers the Netscape Plugin API (NPAPI) to the browsers and acts like all other plugins, except the fact that it does not provide the API functions directly. When the library is loaded by the browsers it starts the pluginloader in Wine and sends all API calls to the plugin through a pipe. The loader will listen for this calls and send them to the Silverlight plugin. All handles, interfaces and objects which are only available at one side are recreated as fake objects on the other side, so that we can capture all calls and redirect them through the pipe. The real handles are replaced by fake 64 bit IDs during the transmission, which allows us to load 32 bit plugins into 64 bit browsers and vice versa without having to pay attention at the size of the real handles. The only real difference in the API between Linux and Windows is the handling of drawing and input events, which requires additional code inside the pluginloader.

The communication itself is done using using two pipes and a very simple protocol. Whenever the browser wants the plugin to execute some specific function, it writes all required information into one end of the first pipe. The plugin decodes and dispatches the request and sends back the response in the second pipe. This method even works if the plugin itself wants to call back a browser function by just using stack recursion.

In early version of Pipelight (or when setting embed=false in the configuration file) the plugin is loaded in a separate window. We have developed a Wine patch adding Xembed-client support which allows us to embed the window directly into the native browser. The alternative method is still available as a fallback - in some cases it might be even useful, for example when you want to resize the applet window.

## 4.2. Which file corresponds to which methods?

The following summary should give you a quick introduction where in the sources you can find which specific component:

/src/linux/

• configloader.c is used to parse and load the configuration file
• basicplugin.c is responsible for starting Wine and dispatching events coming in from a pipe
• pluginfunctions.c contains all the NP_* and NPP_* plugin functions called by your Linux browser
• npclass.c acts as a proxy for non-existent NPClass objects on the Linux side (those created by the Silverlight plugin)

/src/windows/

• pluginloader.c contains the main function (loading Silverlight) and dispatching events coming from the Linux side
• npnfunctions.c provides the browser API used by Silverlight
• npclass.c is just a dummy file containing functions which should never be called (if Silverlight behaves according to the documentation)

/src/communication/

• communication.c contains most of the pipe communication functions used by the rest of the plugin

/src/handlemanager/

• handlemanager.c is responsible for converting handles to IDs and contains functions to send handles across the pipe

## 4.3. Patch status

The Wine patch status has been moved to a separate page.

# 5. Conclusion

We've had a lot of luck - finding the Silverlight 5 issue by accident - and the help of other great people - a big thanks to Erich E. Hoover for his cooperation and Joachim Carlsen for testing and finding bugs! This allowed us to get this whole project finished in just two weeks. Of course such a project will never be finished completely as there might be some rare cases where Silverlight doesn't behave according to the specification (we already found several such cases) or uses some less-usual API command which is not implemented yet, but we are planning to extend this project step by step until it is suitable to fit most of the use-cases.

Until now we already have confirmed the following sites (VOD and other applications) to work almost perfectly using Silverlight 5.1 and Pipelight (ordered by language):

The following pages only work with Silverlight 5.0 - its not a big deal to downgrade but you should note that older versions probably contain some security issues when using them on untrusted sites. For some distributions there are detailed instructions available how to do that: Ubuntu, …

Of course this list is far from being complete - it will be extended when we get some feedback of other users. If you give Pipelight a try please share your opinion and tell us what you think about it! You can either comment directly in this blog or ask a question / fill a bug at Launchpad. If you have a site which works with Pipelight and is not included in the above list, please drop us a comment. Even if you find a site which doesn't work just contact us and we will try to find out whats going wrong.

27 Feb 2014 18:35 CET

Hello all,

we decided to disable the comment system on all Pipelight related articles.

During the last time the comments were often used as a replacement for our bug tracker / question system, although our website doesn't offer all the required features to handle them appropriately there: There is no way to check if the same problem was already reported or to search the comments for specific keywords, there is often no hint if the problem was resolved by any of our suggestions, and moreover a lot of different requests are mixed into a single stream. We also don't have the time to add all the missing functions immediately.

To make things easier for both sides, we now only accept support requests on Launchpad at the following locations:

Sebastian

27 Feb 2014 18:10 CET

Hi Michal,

sorry for the late reply, but we were all very busy during the last days.

I'm sorry but I also cannot provide you with some strict hardware requirements. Since Linux/Wine adds an additional layer of abstraction I would also expect that it requires a better PC than on Windows.

One thing you should also take into account: On Windows Flash and Silverlight can use GPU accelerated video decoding, which is still completely missing in Wine. This means that based on the video codec and resolution the performance can be quite different. Even if your CPU is probably good enough for one video, it could mean that it doesn't work with an other video, when a different codec was used.

What works well is GPU accelerated video rendering, but only if the 32bit OpenGL and the rest of the graphic card drivers is installed correctly. Based on our experience so far Nvidia graphic cards with the proprietary drivers have the best performance (it doesn't even have to be a very powerful one, they all have very good Linux support). Concerning the two Intel Core2 PCs you already used for testing: I'm not 100% sure, but maybe the problem with the slower one also has to do with missing driver stuff.

So, to sum it up, I really think the N2820 processor doesn't have enough power to ensure it works properly (it could work, but on your own risk). I would suggest you instead to use the i3 Haswell + a dedicated Nvidia graphic card to make sure it works well with accelerated GPU rendering. This should also ensure that the solution still works well in a few years, if screen resolution or codecs improve further and have higher requirements.

Sebastian

22 Feb 2014 21:22 CET

Hello,
any idea about hardware requirements for fullscreen Flash and Silverlight video?
I am considering buying an Intel NUC and wonder whether it is enough with Bay Trail Celereon (N2820), or if I need an i3 Haswell?
I guess the requirements are higher than under Windows, or am I wrong?

Just for info: my laptop with Intel Core2 Duo @1.86GHz (L9400) is to weak for this, while my desktop with Intel Core2 Quad @2.33GHz (Q8200) can play the videos fine, using ca. 50% of all four cores.

Thanks.

Michal

12 Feb 2014 16:30 CET

Hi Stefan,

the only way to be sure that hardware rendering is enabled, is to take a look at the debug output of Pipelight. Even forcing gpu acceleration does not necessarily mean that is really used. Some users do not have the required 32 bit libraries of their driver installed and mesa will fallback to software rasterization and you will not even notice that something is going wrong. You should therefore close all your browser windows, start your browser from a terminal and look for opengl related messages when opening sky go / watchever.

Michael

04 Feb 2014 08:57 CET

Hi,
Hardware Acceleration is 100% activated. My CPU is a Pentium G3220 "Haswell" with 2x3Ghz, so I think "he" should be able to handle it.
Unfornutately I can't deactivate pulseaudio, as my onboard soundcard just has one subdevice, so I won't be able to try this tip.
My Problem is still not 100% percent solved, so I am glad for any further tipss. But It has gotten much smoother, here is a little summary of my steps:

1. I used this guide http://ubuntuforums.org/showthread.php?t=2144468 to get rid of the tearing and it works like a charm! I even think It runs smoother now, if this is even possible?!

2. Changed the Screen to 50Hz, seems like whatever skygo is streaming is more suited.

3. Tried native 720p resolution -> did not see any difference.

So far my steps. As I said before, still not that smooth, but Tearing und big stutters are eliminated

Note: I had to use silverlight 5.0 to make skygo work!

Writing comments is currently disabled, as we're updating the content management system.