What to Do When No Clue: Scientific Testing

What to Do When No Clue: Scientific Testing

Sometimes there’s no end in sight. You have the map to navigate around the code, but the sheer complexity of the thing makes finding that one bug evasive. It’s time for Scientific Testing.

As the name implies, we’ll create some theories on why the bug happens, then try to validate them. As we generate hypothesis, we try them out with a small experiment. We document our findings, and keep getting further. This allows us to be sure we already checked avenues of bugs, so we’re not running in circles when trying to figure out what’s going on.

We just need to create a new spreadsheet with 4 columns:

1. Hypothesis : Assumption on why that happens

2. Test : How you’re gonna test that assumption

3. Test results : What did the test return

4. Conclusions : What do the results mean

We separate the test results from the conclusions, since conclusions are subjective. They are the meaning we assign to the test results.

Let’s assume that we have a problem in a REST service that for some payloads it dies misteriously with a 400 error. A bad encoding can be the source of this problem. So does calling a wrong service. It can be headers, such as content-type, or even the request type GET/POST. It can also be a middleware problem, where some proxy layer fails.

We’d start with assuming that the headers failed, or that the request type are wrong, the resource URI is valid, etc. We try a test, write our findings down, then start another test.

A test experiment ends up something like this:

Hypothesis   : Wrong headers
Test         : Check headers in the browser requests
Test results : Content-Type is `application/json`
Conclusions  : Content-Type is valid.

Hypothesis   : Wrong path
Test         : Check request in browser requests
Test results : Resource is `/rest/v1/get/resource/1`
Conclusions  : Resource path is good.

Hypothesis   : Wrong request encoding
Test         : Pass in some UTF8 escaped characters in another request that passes.
Test results : Resource does work well for some other requests.
Conclusions  : Encoding seems to process.

Hypothesis   : Wrong mapping in API GW
Test         : Ask John about the mappings.
Test results : John expected `/rest/get/resource/1`, without `v1`
Conclusions  : API GW is misconfigured

What’s nice about this, is that even after the weekend, just going over the log of the previous tests alone is a good refresher on how to advance further. If also opens the door for introspection later, if our test was insufficient at some step, to try another path.

Happy bug hunting!

Article Photo from Pexels: https://www.pexels.com/photo/ask-blackboard-chalk-board-chalkboard-356079/