How TDD reduces costs: case study

The concept of TDD (Test-Driven Development) based on automated tests promises higher quality software. However, does the need to write and maintain unit tests according to the TDD approach not significantly raise project costs?

As we have proved together with ING Bank Śląski, the well-implemented TDD methodology not only significantly reduced the number of errors, but also reduced costs by 20%.

Verification of profits from TDD: comparison of 2 projects

Together with ING, we have built the ING Financing Invoices application. It is a tool for quick and convenient financing of sales invoices, which you can use without leaving your company.

The project was a great opportunity to verify the benefits of TDD as well as the costs generated. We could compare ING Financing Invoices to another project in which a similar team worked in a similar environment - hereinafter referred to as a "control project".

Today, we have already completed the production launch and the first months of maintenance and development of ING Financing Invoices and we can compare them to the results of the control project. Time for the assessment.

A start-up project

ING Financing Invoices was intended to act like a start-up: during the work on MVP, the scope of works was constantly changing. On the one hand, we wanted to add as many functions as possible, and on the other - quickly launch the application on the market. That is why we were ready to change the concept, and therefore changes in the code. It had to be fast and resistant to regression errors.

We assumed that we would be able to remove all errors from the application at any time within a maximum of one day, or 8 hours of teamwork. In other words, within 1 business day:

• the client was given the current version of the application for the first time for testing;
• the client tested it and reported amendments;
• we fixed mistakes.

The production version update was ready within 8 hours of submitting the previous version for testing.

The goal: quality

Our goal was to reduce the number of errors in the system and shorten the delivery cycle. Certainly we achieved this goal: the first serious error appeared only during the seventh sprint and surprised the client, who, in this project, had not encountered significant shortcomings. Of course, we removed the reported error in one day.

Subsequent versions of the application, given at the end of the sprint, did not contain significant errors. In many cases, they were installed on production without corrections, within a few hours of commissioning.

The low number of errors persisted regardless of the complexity of changes in the system, and regression errors did not occur, although the core of the application was thoroughly rebuilt several times. This is due to the very high 95% code coverage tests.

"ING Financing Invoices is a completely new online service through which companies and entrepreneurs can convert their sales invoices into cash".

Maciej Bukowiec
ING Bank Śląski, Product Owner

Starting practically from scratch, we entered the market with a completely new proposal in approx. 9 months. We worked in the Agile methodology and very dynamically defined the additional functionalities of the application. At the same time, we were able to maintain high quality that was confirmed by a very small number of errors, which allowed us to achieve constant progress without having to spend time on repairs - says Maciej Bukowiec.

The cost of performing tasks

The costs of both projects were estimated by the technical and business manager of the ING Financing Invoices project, the business manager of the control project and an architect who divided his time equally between both projects. We made the estimation a year after starting the project, after its production launch.

For detailed information, we estimated the cost of performing the same tasks in both projects. We adopted several standard modules and functionalities. The size of the tasks corresponded to those that were typically implemented as part of sprints. In every case, the costs of executing the same functionality in ING Financing Invoices were 80% - 125% of the costs of implementation in the control project. On average, let's assume that the team working in TDD in each sprint provided about 10% less work.

No UAT in TDD - lower costs

In the ING Financing Invoices project, conducted with the TDD method, there were no UAT phases. However, in the control project, which also had sprints that lasted 3 weeks, every 2-3 sprints carried out a UAT session which lasted an average of 10-12 working days and only after a new version of the application was ready for production launch. The UAT phase not only delayed the implementation of the application, but also generated additional costs. On average, each sprint was more expensive by 30%.

The lack of UAT was possible only thanks to the TDD project and the code quality obtained in this way. The next sprint could start on the next working day after the previous one. We were not been able to achieve this in any other project that would not be run by the TDD method.

Why TDD reduces costs?

The cost reduction was not only due to the lack of a UAT phase. There were other important factors:

• no need to debug the system: tests of individual classes (unit) avoid the situation in which the programmer must manually analyze the system in a low level;
• no manual verification: you do not have to regularly check that all the code you have written works - tests do it automatically. It also saves the other team members - errors that someone would have to describe and report;
• no manual search for regression errors: tests written previously, for earlier developed modules, automatically check whether new changes have not corrupted the existing system - you do not have to do it manually;
• instant problem detection and feedback: an automated test allows you to quickly detect an error (eg resulting from contradictions in the assumptions), the consequences of which could escalate and generate much higher costs in the future;
• fixing errors immediately: if the error was detected long after it was created, returning to the context would require more time from the developer - he would have to remember the assumptions of the module after switching to new tasks. Working in TDD, identifies and corrects the error immediately when it is updated with a given topic;
• better understanding of the system and better architecture: starting from writing tests, we focus on what the system should do, not HOW. Focusing on the operation of the system, and not on the method of implementation, opens a different perspective, which translates into various aspects of programming: architecture, detection of edge cases, searching for bugs in the concept...
• speed and security of introducing complex changes: TDD guarantees a space in which the programmer does not hesitate to introduce complex changes, he does not hesitate with them, and also devotes less time to it. This also allows for greater flexibility.

The above savings result directly from the application of the TDD methodology. It would not be possible if only elements of this concept were used selectively.

Summary: costs in TDD lower by 20%

Let's summarize.
ING Financing Invoices team performs about 10% less work per sprint, but it does not include additional UAT phases that occur in the control project and increase the cost of each sprint by about 30%. With the same size of units, the same TDD functionality was made by 20% cheaper.

By joining the ING Financing Invoices project, we allowed the possibility of increasing costs in general due to the need for very high coverage of the code with tests. We also expected that the lack of UATs would reduce costs, partly compensating TDD costs. However, the final result - a 20% reduction in costs - exceeded our expectations.

Let us not forget that the lack of errors and the UAT phase allowed to lower the costs on the part of ING. Fewer people were involved in testing and receiving the application. Significant reductions in detection, correction and re-verification cycles reduced the amount of managerial work. We have not included these savings in our calculation, but are another argument in favor of TDD values.

We have achieved a reduction of costs by providing a very high-quality system and thousands of tests that allow us to test every aspect of the application at any time. In addition, in the ING Financing Invoices project, we finally delivered not only a SPA application (Single Page Application), operating entirely in the user's browser, but also a server-side application with a publicly exposed interface in the form of RESTful API. Therefore, as part of the work, we provided not only the application but the API as well.

Methodology and roles in the project

How did you achieve this? The choice of methodology was important. For the start-up project, where volatility and flexible adaptation to changes in the concept were needed, the Scrum method was a natural choice. We decided on 3-week sprints.

In the TDD process, the proper composition of the team and the division of roles is of great importance: the programmer is also a tester of his code, and at the stage of writing tests he must analyze the use cases. Thanks to this, the focus is on the results of the action (and not on how the implementation will look).

In the ING Financing Invoices project, each team member became the manager of the functionality he created and was responsible for the final result. In addition to coding, he also managed the entire life cycle of his task on the Kanban board until it was picked up by the customer. This means that his tasks also included mobilizing, at the right moments, team members and those responsible for this functionality on the part of ING.

Most of the team members therefore combined the roles of the developer, tester and manager. 85% of people in the team had developer competences, and 100% competences of the developer or architect. Thanks to this:

• we have optimized the work of all members in the team: in each sprint we used 100% of each person's working time;
• we could work efficiently even in the absence of part of the team;
• we have improved communication between e-point and ING teams.

Individual members of the team increased their involvement (by explicitly transferring responsibility to the owners of specific tasks) and increased awareness of the purpose and value of the entire project, as well as the module performed by a given person.

In this way, the team has become a self-sufficient organism with high efficiency and organization. This, in turn, reduced the costs associated with the process or management.

Conclusions

The ING Financing Invoices project has shown that thanks to the TDD methodology, it is possible to deliver higher quality code, flexibly adapt to the changing business expectations in a start-up project and concurrently reduce its costs by 20%.

We have created an innovative solution: the product was presented at Finovate Fall in New York on September 24, 2018.