Regression testing with ‘behave’

Regression testing plays a crucial role in software development, especially when an application is based on microservices which are constantly developed by different teams. It assures that every single part which was created and tested in the past still works as it was designed.

Regression testing can get really time-consuming, especially when your system has got a lot of features and the testing process is not automated. We had some issues with it and there was only one solution – to develop an easy, reliable and automated way of regression testing which could be used by QAs and developers without spending days on manual tests.

In order to achieve it, I came up with two solutions. One was a separate application, so a custom solution which would need to be built from a scratch. The second idea was to use a Python library called ‘behave’ which is used for behavior testing in Behaviour Driven Development.

A quick description of the requested project

Interface which will make us able to automate regression testing for particular parts of a microservice application. The idea is to make it extendable – add new tests, amend input data, expected results and checks if required. The main goal is to improve our regression testing, automate it and make it accessible (at least to run tests and receive results) for QAs and developers, so critical things can be checked independently.

Requirements & priorities

• The application is able to execute regression tests – MUST HAVE
• The application has got predefined test data – MUST HAVE
• Checks are defined and a developer can modify them – MUST HAVE
• Test results are saved – MUST HAVE
• The application can be extended to test other systems – HIGH
• Test results can be browsed by a user – MEDIUM
• Users are able to add new test cases – MEDIUM
• Results are sent via email – LOW
• Use the interface to run tests – LOW

The first concept – a dedicated application

The first idea was to create a separate application for regression testing. It would consist of the main core-API which would be responsible for controlling the flow of testing (for example triggering it) and Mongo database to store the test sets and results. Core-API would also allow the user to modify test sets and browse results. Also, worker/workers would be required to generate test results and save them. The application would be required to support communication at least via HTTP.

Three separate parts would need to be created from a scratch:

• API
• workers
• database

Sounds great – a lot of things to be designed, but one of the main ideas was to make this regression testing accessible for QAs, developer. If it has to be accessible, then it must be easy.

The first two steps that I started to work on were:

• communication
• database structure

HTTP protocol was the answer for both ways of communication:

• ‘user’ <=> ‘test_app’
• ‘test_app’ <=> ‘application’

As for the database, I chose MongoDB, as it is widely used by many companies and it meets the requirements (document structure does not depend on a collection, support for embedded documents). The application would also require some workers which will be executing tests and saving the results.

The main concern here was the usability of this solution. API for the application would need to be secure and well documented in order to start using. What is more, the database used to store test cases and test results for all microservices would get messy after a very short time (each microservice and feature would require a different structure of a “test case” and a “test result”).

The second concept – behavioral testing based on ‘behave’ for Python

After taking a closer look at the first solution, it was quite obvious that it is not user-friendly – QA or a developer would need to spend some extra time in order to start working with it. The application would also need to be hosted somewhere with a separate database – that generates costs which we would like to avoid.

That was the moment when I started to read about Behaviour Driven Development and how it works. I realized that maybe we will not use the BDD approach but we could use behavioral testing to achieve our goal related to regression testing. After getting a little bit more familiar with ‘behave’ library I decided to use it.

In general, BDD (Behavior Driven Development) which emerged from Test Driven Development is a way in which you develop your software focusing on behavior which is expected from it. This approach works best when used from the beginning of the project however, it doesn’t mean that you cannot start using it after your application is ready and deployed. For sure you will need to change your approach to writing code but it pays off.

In my case, I noticed that even making the smallest step towards BDD which is using ‘behave’ for testing can be really beneficial for me and the whole team. It helped me to build regression testing from a scratch without:

• separate API
• separate database
• workers

Thanks to ‘behave’ I saved a lot of time that I would spend on building a thing that is not really required. In addition, execution of these tests is much easier compared to what we had in the past or what we would have with a custom solution. Readability is also increased – clear ‘behave’ output instead of browsing database results.

‘Behave’ allowed me to have a structured test suite which thanks to Gherkin syntax is easy to be updated by business people, QAs, and developers. Structure of a ‘behave’ is defined – you have got a separate place for feature files and another one for their implementation in Python – everything is well documented, so it will not take you a lot of time to get familiar with it.

For our team, using ‘behave’ worked from the beginning. I proposed using behavioral testing to improve our software quality and make the risk of failure as low as possible. Business people got involved in this idea by creating test scenarios in Gherkin syntax for our current features – as we already know what these features should do.

As for the new features, behavioral testing makes developing and testing easier. If we have got a clear list of test scenarios to cover from the business perspective, we can easily say if the feature is done or not at any moment of development. These ‘scenarios’ can be also used for regression testing. Collaboration is taken to the next level here – the team understands client’s needs and is more aware of what the code must do, also business, development, and QA are involved in the process.

Why ‘behave’ approach worked better for me:

• It can be used for regression testing
• It is much more user-friendly than any ‘custom’ solution
• The barrier to entry is low – everything is documented, you can easily start using it
• With @tags and environment.py you can control the test flow
  test scenarios are easier to maintain compared to the “custom” solution with a separate application
• This solution does not require any additional database
• It is easy to connect it with Jenkins or Docker – it makes it even more automated and accessible
• Using ‘behave’ can improve collaboration between business and development

Once these two approaches are compared, you can see that sometimes it is better to use already existing mechanisms instead of writing your own. By doing that, you can save a lot of time for further development of your project, also using well-known solutions decreases the chance of failure (documentation, community).

NOTE: Do not forget that behavioral testing cannot replace neither unit nor integration testing as ‘behave’ is focused on a higher “feature” level. Unit and integration tests are required in order to have proper test coverage and code design (if something is hard to test, then you should probably re-write it).