Bug hunters in the code jungle

There are different ways. Often it's developers who have studied computer science in the traditional way and want to try something new. The other way, as in my case, is a computer science-related education with a more or less direct route into testing. I first worked in support at a software company, but my responsibilities overlapped with testing from time to time.

I joined KIX in 2017 and today I coordinate this area. Testers don't necessarily need to know the software in detail, that's the job of the developers. But they need to see it through the eyes of a customer. And they often have to think outside the box.

We already have structures, processes and steps in place. Whether we are testing a new feature, debugging a bug, or running RC, release, or regression tests. The usual daily routine consists of testing newly developed features and fixed bugs and checking their expected behavior.

What makes it exciting and varied, however, is that we are confronted with new topics and tasks every day, which means that we are constantly expanding our technical and professional knowledge. We never get bored. And only when we have completed all the steps do we give the go-ahead.

First, we try to reproduce the problem on a current system using the reproduction steps provided. If that fails, we try it on a system with the same version as the customer's system. If this also fails, we take a closer look at the customer's system. The specific configurations are often an important factor in the cause of the error.

So it's a mixture of precise mapping of the customer's steps and analytical detective work. Sometimes I feel like a detective (laughs). But of course I prefer to catch the culprit or the bug in advance. There are plenty of examples where a buggy program has caused millions of dollars in damage.

Yes, of course. We use test automation, which I can only recommend to all colleagues. In other words, a tool that performs test steps that are repeated on a regular basis. It takes some of the work out of our hands, but our main job is to feed the tool with requirements.

This means that we define the test scenarios and steps according to the requirements, and it usually runs them once a day. If the tool finds an error, we are automatically notified. This allows us to work much more efficiently.

Does it put you under particular pressure that all users can see the source code and discover errors that you have overlooked?
On the contrary, the advantages of open source simply outweigh the disadvantages. Potential bugs can be seen by a much larger number of people, and often can be fixed right away. With thousands of users, a problem simply becomes apparent more quickly than it would in an internal testing and development environment.

Sometimes it can be a bit of a professional challenge, but completely bug-free software is simply utopian. Even the software for the space shuttle had bugs.

The natural limits of testing are usually economic. There is always a trade-off between the cost of defects and the cost of preventing them. Conversely, this means that there is always a so-called gray area of possible bugs that are still present in the software. Prior to a release, tests are performed that cover all important functions whose failure could have a serious impact on the customer's business.

At KIX, we work with five error classes from A to E. For example, an A-class error would be when users are unable to create tickets. In other words, a serious problem that we have to fix immediately. Class E, on the other hand, includes more trivial things. If an icon is misaligned or contains a spelling error, the probability that this will lead to a business-critical situation is very low.

The most important functions are then part of the regression test. The less important test cases are only executed when the time frame allows. Back to the space shuttle example: NASA had less than one defect per 420,000 lines of code. They were able to do this because they are not a commercial organization, but have a large government-funded budget. One written, tested, and documented line of code cost NASA about $1,000 in 1973. Adjusted for inflation, that would be about $7,000 per line today. In other words, if the project were done the same way today, it would cost nearly three billion dollars for the software alone. And not a single rocket would have been built.

Sure, it happens from time to time. When we get an error message, the first thing we do is check if it really is a bug. The phrase "That's not a bug, it's a feature" sometimes comes up (laughs).

Thanks to our feature list, we can quickly determine if this is intentional or not. And even if it is a new bug, it's not necessarily bad. Sometimes there are moments when we incorporate a supposed problem into our solution - a happy accident, so to speak. This happens from time to time in the industry. A well-known example is the nearly 50-year-old Space Invaders bug.

In short: the more spaceships the player shot down, the less processing power was needed. This meant that the remaining enemies moved faster towards the player. The developers did not plan for this, but left it in the game to increase the difficulty.