A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering

Context: Systematic literature reviews (SLRs) have become standard practise as part of software engineering research, although their quality varies. To build on the reviews, both for future research and industry practice, they need to be of high quality. Objective: To assess the quality of SLRs in software engineering, we put forward an appraisal instrument for SLRs. The instrument is intended for use by appraisers of reviews, but authors may also use it as a checklist when designing and documenting their reviews. Method: A well-established appraisal instrument from research in healthcare was used as a starting point to develop a quality assessment instrument. It is adapted to software engineering using guidelines, checklists, and experiences from software engineering. As a validation step, the first version was reviewed by four external experts on SLRs in software engineering and updated based on their feedback. Results: The outcome of the research is an appraisal instrument for the quality assessment of SLRs in software engineering. The instrument intends to support the appraiser in assessing the quality of an SLR. The instrument includes 16 items with different options to capture the quality. The item is assessed on a two or three-grade scale, depending on the item. The instrument also supports consolidating the items into groups, which are then used to assess the overall quality of a systematic literature review. Conclusion: It is concluded that the presented instrument may be helpful support for an appraiser in assessing the quality of SLRs in software engineering.