Text size
  • Small
  • Medium
  • Large
  • Standard
  • Blue text on blue
  • High contrast (Yellow text on black)
  • Blue text on beige

    An Evaluation of Quality Checklist Proposals - A participant-observer case study

    13th International Conference on Evaluation and Assessment in Software Engineering (EASE)

    Durham University, UK, 20 - 21 April 2009


    Barbara A Kitchenham, O Pearl Brereton, David Budgen, Zhi Li


    Background: A recent set of guidelines for software engineering systematic literature reviews (SLRs) includes a list of quality criteria obtained from the literature. The guidelines suggest that the list can be used to construct a tailored set of questions to evaluate the quality of primary studies. Aim: This paper aims to evaluate whether the list of quality criteria help researchers construct tailored quality checklists. Method: We undertook a participant-observer case study to investigate the list of quality criteria. The "case" in this study was the planning stage of a systematic literature review on unit testing. Results: The checklists in our SLR guidelines do not provide sufficient help with the construction of a quality checklist for a specific SLR either for novices or for experienced researchers.

    However, the checklists are reasonably complete and lead to the use of a common terminology for quality questions selected for a specific systematic literature review. Conclusions: The guidelines document should be amended to include a much shorter generic checklist. Researchers might find it useful to adopt a team-based process for quality checklist construction and provide suggestions for answering quality checklist questions.


    PDF filePDF Version of this Paper (121kb)