Understanding and Improving Continuous Experimentation : From A/B Testing to Continuous Software Optimization
Förstå och Förbättra Kontinuerlig Experimentering : Från A/B Testning till Kontinuerlig Mjukvaru Optimering
Summary, in English
The goal of the thesis is to describe current practice and support CE in industry. The main contributions are threefold. First, a review of the CE literature on: infrastructure and processes, the problem-solution pairs applied in industry practice, and the benefits and challenges of the practice. Second, a multi-case study with 12 companies to analyze how experimentation is used and why some companies fail to fully realize the benefits of CE. A theory for Factors Affecting Continuous Experimentation (FACE) is constructed to realize this goal. Finally, a toolkit called Constraint Oriented Multi-variate Bandit Optimization (COMBO) is developed for supporting automated experimentation with many variables simultaneously, live in a production environment.
The research in the thesis is conducted under the design science paradigm using empirical research methods, with simulation experiments of tool proposals and a multi-case study on company usage of CE. Other research methods include systematic literature review and theory building.
From FACE we derive three factors that explain CE utility: (1) investments in data infrastructure, (2) user problem complexity, and (3) incentive structures for experimentation. Guidelines are provided on how to strive towards state-of-the-art CE based on company factors. All three factors are relevant for companies wanting to use CE, in particular, for those companies wanting to apply algorithms such as those in COMBO to support personalization of software to users' context in a process of continuous optimization.
Dissertation 68, 2022
Department of Computer Science, Lund University
- Software Engineering
- ISSN: 1404-1219
- ISBN: 978-91-8039-178-8
- ISBN: 978-91-8039-177-1
4 March 2022
Lecture hall E:A, building E, Ole Römers väg 3, Faculty of Engineering LTH, Lund University, Lund.
- Klaas-Jan Stol (Dr.)