Methodological quality of systematic reviews for identification/selection of studies in human immunodeficiency virus (HIV)

Aim: To assess the methodological quality in the identification and selection of studies of evidence synthesis articles cited in the 2017/2018 U.S. Department of Health and Human Services (DHHS) Human Immunodeficiency Virus (HIV) Guidelines for the Use of Antiretroviral Agents in HIV-1 Infected Adults and Adolescents using the risk of bias in systematic reviews tool (ROBIS).

Background: The number of published systematic reviews has dramatically increased in recent years. The rapid influx of systematic reviews has made keeping up and evaluating primary literature nearly impossible for health care practitioners. Additionally, there has been some concern that a proportionate amount of newly published systematic reviews can be considered low-quality. Suboptimal systematic reviews can be harmful, given their significant prestige and the influence they have on supporting clinical decision making. The concern for low-quality systematic reviews has highlighted the importance of rigorously assessing systematic reviews.

Methods: Our corpus comprised of all literature cited in the 2017/2018 DHHS HIV Guidelines. Two investigators independently conducted a title and abstract screening of the corpus of literature and identified literature that were likely to be categorized as an evidence synthesis article. Full-text screening was then performed independently by two investigators that confirmed the eligibility of evidence synthesis articles included in the study. Evidence synthesis articles were then independently assessed by two investigators for their methodological quality of identifying and selecting studies using the ROBIS tool. Investigators also assessed literature using a questionnaire that looked at the studies’ search-related elements and the methods for article selection and data extraction.

Results: We identified 26 pieces of literature cited in the DHHS HIV Guidelines as evidence synthesis articles. Eleven evidence synthesis articles (42.31%) were considered to have a low risk of bias in the identification and selection of studies by the ROBIS tool and therefore considered to be of high methodological quality. In contrast, 12 evidence synthesis articles (46.15%) were considered to have a high risk of bias in the identification and selection of studies by the ROBIS tool and therefore considered to be of low methodological quality. The remaining three evidence synthesis articles (11.54%) were categorized as unclear in their risk of bias.

Conclusion:  We attempted to establish a benchmark of what is considered the best available evidence through a current guideline.  Using the ROBIS tool, our study identified that more than half of the identified evidence synthesis articles cited by the 2017/2018 DHHS HIV have a high risk of bias in the identification and selection of studies. With these results, it is clear to see that the methodological quality should be considered when evaluating the results of evidence synthesis articles and making guideline recommendations

Published in College of Pharmacy, Virtual Poster Session Spring 2020


  1. Jonny, Calvin, Sonny: This was a very enlightening piece of research. Somewhat appalling that more than 50 % was considered not quality per the standard guidelines used.

    1. Thank you for your comment Jane. I also found it interesting that more than 50% of the systematic reviews cited by a guideline had a risk of bias in their identification and selection of studies.

  2. Jonny, Well done! It is so interesting to see these projects at the completion. These data are quite interesting, as well as somewhat concerning. While you comment that you can’t generalize these findings to other guidelines and the systematic reviews on which they’re based, do you think this is likely a wide problem?

    1. Thank you, Dr. Keefe, for commenting. Although it is hard to say if this applies to other guidelines, I do believe that these results may point to a more wide-spread problem that needs to be addressed.

  3. Interesting work. OVerall, this work addresses an important question and should be presented at Cochrane colloquim conference. I am sure that they will like this methodological research work. I have a few questions for you.

    1) As a reader, I wonder how ROBIS defined criterion for risk of bias of identification/ selection of studies. It would be great if you can say that in the poster or even tell me here. In addition, do you know of exisiting literature on assessing the identification/ selection of studies of other systematic reviews. Do they find the same thing as what you do?

    2) Another thing is that why you chose to use ROBIS? Any particular reasons over AMSTAR?
    3) I wonder if you tried to do analysis based on years of work. You may find that the search/ identification has improved dramatically over years or even performed better once PRISMA or AMSTAR came out. It would be interesting to see.

    Last is my comment. I think 26 SR papers is a small sample and it might be hard to summarize the findings based on this limited sample size.

    1. Thank you for your questions, Dr. Chaiyakunparuk.
      1A: Table 1 includes the questions investigators should use when assessing the risk of bias in systematic reviews for the identification and selection of bias. The investigators can classify whether they believed there was a low vs. high risk of bias based on their questions from the ROBIS questionnaire.
      1B: Through my literature search, I am not aware of existing literature that specifically assesses the identification and selection of studies.
      2A: My project is a subset of a much larger project. The AMSTAR tool was used to assess systematic reviews in the same guidelines for another project. ROBIS was used as another tool that could assess the methodology of systematic reviews.
      3: That would be interesting to see the difference in the search/identification of systematic reviews after one of the tools was published to understand their impact on research.
      4: I agree that due to the small sample size, it might be hard to summarize the findings. Although, I am unsure what pooled pieces of literature besides guidelines would offer more systematic reviews to assess for a large sample size.

  4. I find it very surprising that almost half of the systematic studies included in the guidelines have a high risk of bias and are of low methodological quality. It makes me wonder about the systematic reviews included in other guidelines and whether we should be taking a closer look at those studies as well. Nicely done.

    1. Thank you for your comments Angie. It also makes me wonder about the systematic reviews included in other guidelines as well. I hope that there will be future studies that address the possibility of poor-quality studies being included in guidelines.

  5. Good job Jonathan. You arrived at an important conclusion.

  6. Jonny, Calvin and Sonny,
    Great poster guys. Looks like alot of research went into your presentation. Sure excited you’ll all be finishing soon. Congratulations!

  7. Interesting report, surprising how little this is considered in reviews.

  8. Jonny, this looks like a lot of work, and contributes to fundamental guidance as the world of systematic review continues to expand.

Comments are closed.