Welcome to our second part of the series "Holistic Software Testing". In the first part of this series we explain the importance of software testing and here in the second part we introduce the concept of a holistic testing approach: money-saving & productivity increasing.
Assessment of previous testing needs and forecast for three subsequent years
twim developed a holistic testing approach tailored to the customer's needs with various productivity-enhancing attributes. In preparation, however, a comprehensive data collection was carried out on the basis of project documentation and a survey of those responsible for testing.
In particular, the testing volume of the past years was determined on a project-by-project basis and its annual increase was calculated. It was also documented, for example, how many testers were needed to cope with the testing effort and what financial expenditure was allocated to software testing.
«A survey of past years' figures showed an average annual growth in demand for software testing of 22.5%.»
Martina Schelling, Senior Softwaretester
Finally, based on these lists, the further growth of testing demand was forecast for three years. The forecast corresponds to an average increase in demand of 22.5% per year, combined with a similar annual increase in costs. According to the above, the declared goal was to cope with the increasing testing effort for a time horizon of three years; and to do so while saving costs, without the need for additional testers and without a loss of quality (or even with an increase in quality) in software testing.
Cost savings and other positive side effects of increased productivity
The cost-generating factor in software testing is mainly the test personnel. According to the client's specifications, the number of testers (manpower) was unchangeable. Neither was the dismissal of test personnel to be considered, nor was it an option to hire additional test personnel over a period of three years. The only levers left to achieve cost savings were testing workload and efficiency/quality improvements.
«Target: Neither new hires nor layoffs.»
The twim's holistic approach to testing glosses over both workload and efficiency and quality improvements. Since the workload is continuously increasing, as explained above, workload reduction here can only mean reducing the projected workload increase, so that the workload ultimately grows to a lesser extent through the holistic measures than without the measures.
The increase in efficiency and quality compensates for the decreasing manpower in relation to the growing workload (shown in the graph). Workload reduction and efficiency and quality improvement are the two overarching levers that the holistic testing approach influences. The influence on workload and efficiency results in an overall increase in productivity.
Beyond workload reduction and increased efficiency and quality, the concept devised by twim achieves the following.
Firstly, the testing teams are provided with comprehensive understanding of the end-to-end processes including interfaces, enabling them to test contextually.
Secondly, with regard to the software landscape, synergies can be used across the entire company organisation and redundancies can be avoided.
And thirdly, test automation is promoted.
Three components to ensure software quality
So much for the effects and side-effects of the holistic testing approach, before moving on to look specifically at its three components. The approach includes a tool to support the software testing teams, relies on a specific process that embeds the testing stream deeper into the project network and involves it earlier in the project, and finally provides added value to the whole business organisation as an end-to-end and testing competence centre.
These measures are carried out by a manager or a small team (depending on the size of the company) who is responsible for the implementation of the measures and their maintenance. In the following, this manager or team is referred to as test support.
Testing Support Tool
At the heart of the concept is a tool that also provides the most obvious and tangible added value for the testing teams. In the concept developed by twim, the tool consists of a sophisticated BPMN 2.0 process visualisation with high information content.
The testers can use the tool to obtain knowledge about the company's software landscape, interfaces and end-to-end processes. In addition, the testing support tool contains each individual process flow in different stages of development and in the context of the individual software tools involved in the process flow. The process flow is presented in a functionality perspective, as such a perspective is relevant for the testing stream (unlike for the business stream, for example).
For each test-relevant functionality, for example, the individual test cases are presented in a high-level version and with the relevant performer profiles, with reference to the underlying stories and acceptance criteria and, if necessary, with reference to any difficulties and risks that need to be considered during the tests or their preparation. Prioritisation and effort estimates, information on the test environments and an indication of any regression tests that may become necessary are also included.
The benefit of this information for test managers on the one hand and for testers on the other is obvious, which is why only a few examples are given here. Test managers use the tool as an orientation aid for their planning with regard to test personnel and time expenditure, but also for completeness control, for example, in order to avoid gaps in software testing. Testers, on the other hand, are relieved of some of the effort, for example, because test variants for a functionality and profiles to be tested are directly visible. Due to the generally better end-to-end understanding of the testers, acceptance also increases, for example on the part of the developers, so that reported bugs are not rejected unseen.
«The Testing Support Tool helps test managers with planning and testers with all testing activities.»
The testing support tool is not only the element of our concept that generates the greatest benefit for the testing stream. It has to be set up, maintained and serviced and thus also involves a relatively high effort. Therefore, it is important that the tool is based on previously defined conventions for structure and modelling. Intelligent naming of processes and test bundles is also indispensable, so that they can be assigned to each other and to different releases or the like. In addition, process models must be linked to each other so that they are embedded in their context in the sense of a map. Such conventions and rules reduce the documentation effort, make it easier to understand and ensure a smooth transition if there is a change of personnel in test support.
In order to reduce the effort of setting up, maintaining and servicing the testing support tool, restrictions are of course conceivable with regard to the information content of the tool.
Testing Support Process
For the benefits of the holistic software testing approach to be fully realised, and for this testing approach to be effectively holistic, the creation and maintenance of the testing support tool must be embedded in a defined process that spans the entire project cycle and ensures that the testing stream is involved earlier and more intensively in the project and development process.
In line with what was said in the first part, according to which software testing must be understood as a component of software development, the early involvement of the testing stream is considered one of the most important success factors for quality assurance.
«The most important success factors for quality assurance are early involvement of the testing stream in the project and close cooperation between the testers and the developers.»
The early involvement of the testing stream in the development process has three concrete effects in our testing concept. Firstly, the testing stream is provided with information and remains up to date. Secondly, the testing expertise can flow directly into the operations and thus support them. Thirdly, errors can already be corrected in the planning phase (for example, through reviews of requirements), as long as they are still easy to correct and do not result in change requests.
Beyond the early involvement of the testing stream, a project and ultimately the resulting software quality also benefits from the testing support process insofar as a continuous exchange between the individual project streams is ensured.
The process of adding new process visualisations to the testing support tool is more comprehensive and costly than the process of maintaining the inventory.
Competence centre
As a side effect, Test Support, and with it the tool it creates and maintains, is evolving as a comprehensive inventory database into a centre of excellence for the entire business organisation. This provides a holistic and context-based understanding of technical components as well as implemented functionalities and acts as a bridging instance between system integration and quality assurance.
twim carried out an elaborate and substantiated analysis of what was presented, taking into account all impact factors of the elaborated concept as well as the evaluation of the testing situation of the past years at the client, in which all test managers of the client company were included for control and verification.
On this basis, assumptions were made with regard to the impact of the holistic testing approach, whereby deliberately conservative estimates were made.
Productivity-enhancing holistic software testing approach: The impact
The improvements achieved by the holistic testing approach affect both workload and efficiency, as explained above in the text. The workload is linked to the testing scope, i.e. the number of requirements to be tested, bug fixes, change requests and regression testing.
Decisive for efficiency, on the other hand, are the testing activities to be carried out and the effort that has to be put into them. For the sake of simplicity, a test case was chosen as the unit of measurement for efficiency, whereby the processing of a test case here not only means its execution, but also includes the necessary knowledge acquisition, training, test preparation and manual regression testing.
«Workload: What/how much testing needs to be done?
Efficiency: How much effort needs to be put into testing?»
With the help of the holistic testing approach, the workload can be reduced from the following points of view. User stories are triaged in advance so that redundancies can be identified and merged early, and incorrect or incomplete stories can be fed back to requirements engineers or architects and corrected.
The number of improvement stories and change requests can be minimised by eliminating inadequacies or misunderstandings in the requirements in advance. Finally, there is less need for bug fixes, as there are fewer bugs, but also less frequent retesting of bug fixes, as the testers' credibility with the developers increases.
The factor that increases efficiency is the reduced effort required for knowledge acquisition and training, for test planning, preparation and execution, and for manual regression testing. The minimisation of effort results, among other things, from the greater overall understanding of the entire software landscape, the prior elimination of misunderstandings and technical obstacles, the clarification of questions in advance and the early visibility of possible effects of changes on interfaces.
Impact of the holistic testing approach
A careful evaluation of the workload and efficiency factors, including a reference project, resulted in the following assumptions: A workload reduction of -1.6% in the first, -8.9% in the second and -11.9% in the third year, and an efficiency increase of +0.5% in the first, +13.6% in the second and +19.5% in the third year can be achieved.
Two additional full-time positions were allocated as test support for the set-up, implementation and maintenance.
After deducting the costs for test support, the holistic testing approach results in a cost saving of 0.05% for the first (current), 21% for the second and 32% for the third year.
As has been shown, restructuring software testing pays off after only a short time. Are you also interested in increasing your software quality? - Contact us for a personal discussion with our testing experts.