Per-scenario overview of potential pitfalls and how to prevent these when using ASReview in a systematic review
| Potential scenario | Pitfall | Remedy | |
| ① | Only a small (ie, manually feasible*) number of articles (with possibly a high proportion relevant) available for screening | Time wasted by considering AI-related choices, software training and no time saved by using AI | Do not use AI: conduct manual screening |
| ② | Presence of duplicate articles in ASReview | Unequal weighing of labelled articles in AI-supported screening | Apply deduplication methods before using AI |
| ③ | Reviewer’s own opinion, expertise or mistakes influence(s) AI algorithm on article selection | Not all relevant articles are included, potentially introducing selection bias | Reviewer training in title and abstract screening Perform (partial) double screening and check inter-reviewer agreement |
| ④ | AI-supported screening is stopped before or a long time after all relevant articles are found | Not all relevant articles are included, potentially introducing selection bias, or time is wasted | Formulate a data-driven stopping criterion (ie, number of consecutive irrelevant articles) |
| ⑤ | AI-related choices not (completely) described | Irreproducible results, leading to a low-quality systematic review | Describe and substantiate the choices that are made |
| ⑥ | Study selection is not transparent | Irreproducible results (black box algorithm), leading to a low-quality systematic review | Publish open data (ie, extracted file with all decisions) |
*What is considered manually feasible is highly context-dependent (ie, the intended workload and/or reviewers available).