What Do We Learn from the Broken Firefox WebDriver Support

What do we Learn From the Broken Firefox WebDriver Support

With the new release of Firefox 47, the WebDriver support was left in limbo. On one hand, the old WebDriver API was not accessible anymore, on the other hand the new API (Marionette) explicitly didn’t support it. Kid you not, they actually used the word explicitly when saying they don’t support version 47, despite Mozilla telling on the release notes to use it.

“Go use it, it’s broken” [probably someone at Mozilla]

Of course, this lead to the Mozilla team releasing a hotfix for Firefox, namely 47.0.1 which fixed, you guessed it, allowing the old WebDriver API to work. The only small problem is that Mozilla knowingly introduced a bug that nuked all the WebDriver Firefox tests. For 3 weeks. Between June 7 and June 28.

Unreal.

Solution

Now that we know this might happen, how do we mitigate this?

The answer is containers. Docker containers.

Germanium, out of the box, comes into two flavours. On one hand it’s the library itself that we know and love and that gets tested against a set of browsers.

Now, for Firefox and Chrome, also docker images are automatically built, that guarantees such changes in infrastructure aren’t destroying your tests stability. For Firefox, since version 47 wasn’t running anymore using WebDriver, the container used Firefox 46 and the old API (the Marionette support is at this stage abysmal).

Of course, this means we’re not on the bleedingest of edges of the browser version, but that is OK. In the Continuous Integration system, we want to be sure we don’t get all our tests failing just because WebDriver has a bug now, especially since the API itself it’s still coagulating, so this is bound to happen some more. The key is to lock the moving parts of the testing environment, and Germanium does that by default.

Also these images themselves are used to test Germanium itself. So you know that all the API calls that are documented there are running as expected.