Shortening feedback time in continuous integration environment in large-scale embedded software development with test selection

by J (Jarmo) Koivuniemi

Institution: University of Oulu
Year: 2017
Keywords: Information Processing Science
Posted: 02/01/2018
Record ID: 2154747
Full text PDF: http://urn.fi/URN:NBN:fi:oulu-201704121470


Abstract Continuous integration is one of the Extreme Programming practices and is used in agile software development to provide rapid feedback and to have a working system at all times. In continuous integration, a developer commits code to projects mainline at least once a day which triggers automated build and tests. Large projects can struggle with continuous integration because with growing code base the number of tests that need to execute after a developer checks in code also gets bigger. The growing number of tests means that the build times can become long. With embedded systems, the problem can be even bigger as the testing is dependent on target hardware. With long builds, feedback becomes longer thus making it harder to practice continuous integration. Long feedback leads to infrequent integrations which can result in continuous integration becoming a bottle-neck to the development process and can even affect an ability to release software frequently. To shorten the feedback this thesis implements an automated test selection tool to work in continuous integration environment. The test selection tool selects the tests that are relevant for a specific code change instead of executing all tests for each commit. The case company is suffering from long feedbacks and the expensiveness of testing has become a problem. This thesis hopes to solve that problem with the implemented test selection tool. The tool is evaluated using three metrics; feedback time improvement, reduction in number of executed test cases and fault finding capability. The test selection tool was tested with two test suites with different size by collecting data from continuous integration system. Shock suite consisted of about 30 test cases and regression suite about 800 test cases. The average improvement in feedback time for Shock tests was 29.3% and 55.7% for regression suite. The test selection tool reduced the number of test cases executed in Shock by 67.1% and 78.2% for regression suite. Fault finding capability was measured for Shock suite and the tool was able to find same faults as the full suite in 97.8% of the cases. Statistical tests show that the test selection has a significant impact on feedback time. By being able to safely shorten the feedback, the test selection tool can potentially help developers in the case company to practice continuous integration.