Navigating Information Overload Caused by Automated Testing – A Clustering Approach in Multi-Branch Development

Summary, in English

Background. Test automation is a widely used technique to increase the efficiency of software testing. However, executing more test cases increases the effort required to analyze test results. At Qlik, automated tests run nightly for up to 20 development branches, each containing thousands of test cases, resulting in information overload. Aim. We therefore develop a tool that supports the analysis of test results. Method. We create NIOCAT, a tool that clusters similar test case failures, to help the analyst identify underlying causes. To evaluate the tool, experiments on manually created subsets of failed test cases representing different use cases are conducted, and a focus group meeting is held with test analysts at Qlik. Results. The case study shows that NIOCAT creates accurate clusters, in line with analyses performed by human analysts. Further, the potential time-savings of our approach is confirmed by the participants in the focus group. Conclusions. NIOCAT provides a feasible complement to current automated testing practices at Qlik by

reducing information overload.

• Author: Nicklas Erman; Vanja Tufvesson; Markus Borg; Anders Ardö; Per Runeson
• Department/s: Department of Computer Science; Department of Electrical and Information Technology; Software Engineering Research Group; ELLIIT: the Linköping-Lund initiative on IT and mobile communication
• Publishing year: 2015
• Language: English
• Publication/Series: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings
• Full text: - 507 kB; Download statistics
• Links: http://dx.doi.org/10.1109/ICST.2015.7102596
• Document type: Conference paper
• Publisher: IEEE - Institute of Electrical and Electronics Engineers Inc.
• Topic: Electrical Engineering, Electronic Engineering, Information Engineering; Computer Science
• Keywords: software testing; test automation; test result analysis; clustering; case study
• Conference name: International Conference on Software Testing, Verification and Validation, 2015
• Conference date: 2015-04-13 - 2015-04-17
• Conference place: Graz, Austria
• Status: Published
• Project: Embedded Applications Software Engineering; Embedded Applications Software Engineering
• ISBN/ISSN/Other: ISBN: 9781479971251