Integration Testing Beyond The Surface

Integration Testing Beyond The Surface

When thinking about integration tests, a lot of people think only about the source code of the tests. Of course, seeing tests like that is not only wrong, but also might prove fatal for your testing infrastructure. Let’s see why.

Reliable testing has two major problems:

1. Writing the tests in a reliable and repeatable way,

2. Running the tests in a reliable and repeatable way.

If you’re like me, you’re thinking, I’m just reiterating obvious things, and you would be right.

The key in how we should look at this is through cost.

Estimating Cost

Since we have two kinds of problems in regards with testing (writing, and running), we will look at them independently.

Cost for Writing Tests

When we talk about writing the tests, estimating the cost might seem easy: How much time do I need until I can write the test trying out my feature?

The big problem in these scenario is the moment your test is written, the cost transfers into maintaining the test suite, and ensuring it responds correctly to application changes.

Cost for Running Tests

Running the tests, especially in the case of web integration tests, it’s a far more treacherous subject. It’s not only setting up the infrastructure, but also software updates for Selenium, Java, or installation of new binary drivers.

Since this is actually an operations issue (the Ops from DevOps), it’s usually even tougher to measure and to maintain, because it doesn’t reflect directly in time spent by the development team, and the cost is even higher.

Reducing Costs

Germanium is built to reduce both of these costs.

On one hand the API offered greatly simplifies writing reliable tests, via the usage of selectors and locators. In some cases, WebDriver support itself is broken, or misbehaving.

Certain versions of WebDriver/browser combinations are broken, therefore Germanium works these issues around, so you don’t waste time patching your tests, but rather your focus is spent on writing those tests.

On the other hand, when releasing a new version, each version must pass a rigorous testing across all the supported browsers, before the release.

2.0.1 Public Change Log

This is the published list of "changes" in Germanium for version 2.0.1:

Project	Description
Germanium Java Drivers	Update drivers to the latest version
Germanium API	Box API should report not finding elements.
Germanium Python Drivers	Update drivers to the latest version
Germanium API	Allow specifying URLs with query parameters in open_browser.
Germanium Java	go_to() is missing from the API
Germanium Python	The default iframe selector doesn’t report correctly the unknown frame name.

2.0.1 Internal Change Log

And this is the actual list of issues that were internally resolved in 2.0.1:

Project	Description
Germanium Testing Infrastructure	Feature #1: Create a redmine server
Germanium Testing Infrastructure	Feature #3: Create Redmine projects for all the Germanium projects.
Germanium Testing Infrastructure	Feature #5: Create virtual machines for IE.
Germanium Java Drivers	Feature #14: Update drivers to the latest version
Germanium Testing Infrastructure	Feature #25: Create a Jenkins server.
Germanium API	Feature #30: Box API should report not finding elements.
Germanium Python Drivers	Feature #31: Update drivers to the latest version
Germanium API	Feature #32: Allow specifying URLs with query parameters in open_browser.
Germanium Java	Bug #4: go_to() is missing from the API
Germanium Python	Bug #24: The default iframe selector doesn’t report correctly the unknown frame name.
Germanium Testing Infrastructure	Task #7: Create an IE8 machine
Germanium Testing Infrastructure	Task #8: Create an IE9 machine.
Germanium Testing Infrastructure	Task #9: Create an IE11 machine.
Germanium Testing Infrastructure	Task #11: Update Jenkins to support Ansible
Germanium Testing Infrastructure	Task #26: Install Jenkins locally

Conclusions

Quite a longer number of issues, isn’t it? And most of all are infrastructure related.

We do this, so you don’t have to.

We take stability very-very seriously, because we know at the end, we want truly the best testing API out there.

In the end, our tests and infrastructure depend on it, and if we, the Germanium team, go the extra mile to guarantee our stuff works, your priority will be the tests, and not maintaining the test library.

Image Credit

Iceberg picture by AWeith from Wikipedia.