The browser you are using is not supported by this website. All versions of Internet Explorer are no longer supported, either by us or Microsoft (read more here:

Please use a modern browser to fully experience our website, such as the newest versions of Edge, Chrome, Firefox or Safari etc.

Understanding and Improving Continuous Experimentation : From A/B Testing to Continuous Software Optimization

Förstå och Förbättra Kontinuerlig Experimentering : Från A/B Testning till Kontinuerlig Mjukvaru Optimering


Summary, in English

Controlled experiments (i.e. A/B tests) are used by many companies with user-intensive products to improve their software with user data. Some companies adopt an experiment-driven approach to software development with continuous experimentation (CE). With CE, every user-affecting software change is evaluated in an experiment and specialized roles seek out opportunities to experiment with functionality.

The goal of the thesis is to describe current practice and support CE in industry. The main contributions are threefold. First, a review of the CE literature on: infrastructure and processes, the problem-solution pairs applied in industry practice, and the benefits and challenges of the practice. Second, a multi-case study with 12 companies to analyze how experimentation is used and why some companies fail to fully realize the benefits of CE. A theory for Factors Affecting Continuous Experimentation (FACE) is constructed to realize this goal. Finally, a toolkit called Constraint Oriented Multi-variate Bandit Optimization (COMBO) is developed for supporting automated experimentation with many variables simultaneously, live in a production environment.

The research in the thesis is conducted under the design science paradigm using empirical research methods, with simulation experiments of tool proposals and a multi-case study on company usage of CE. Other research methods include systematic literature review and theory building.

From FACE we derive three factors that explain CE utility: (1) investments in data infrastructure, (2) user problem complexity, and (3) incentive structures for experimentation. Guidelines are provided on how to strive towards state-of-the-art CE based on company factors. All three factors are relevant for companies wanting to use CE, in particular, for those companies wanting to apply algorithms such as those in COMBO to support personalization of software to users' context in a process of continuous optimization.

Publishing year





LU-CS-DISS 2022-22


Dissertation 68, 2022

Document type



Department of Computer Science, Lund University


  • Software Engineering




  • ISSN: 1404-1219
  • ISBN: 978-91-8039-178-8
  • ISBN: 978-91-8039-177-1

Defence date

4 March 2022

Defence time


Defence place

Lecture hall E:A, building E, Ole Römers väg 3, Faculty of Engineering LTH, Lund University, Lund.


  • Klaas-Jan Stol (Dr.)