Table 3 Risk of bias assessment
Study | Risk Assessment Parameter | Assigned level | Basis of judgment | Assigned overall level |
|---|---|---|---|---|
Afghanistan Cyrus et al., 2015 [19] | Random sequence generation (selection bias) | Low risk | Randomized matched pairs | Low Risk |
Allocation concealment (selection bias) | Low risk | Sequence generation and allocation happened simultaneously | ||
Blinding of participants and personnel (performance bias) | Low risk | Not feasible due to nature of intervention | ||
Blinding of outcome assessment (detection bias) | Low risk | Outcome measures identified before the trial | ||
Incomplete outcome data (attrition bias) | Low risk | Cluster level of analysis with all clusters remaining in trial | ||
Selective reporting (reporting bias) | Low risk | No evidence of selective outcome reporting, presence of a third-party independent evaluator | ||
Argentina Gertler et al., 2014 [20] | Random sequence generation (selection bias) | Unclear | “… Over time the membership of the treatment and control group changes.” | Low risk |
Allocation concealment (selection bias) | Low risk | Based on initial phased randomized clinics assignment | ||
Blinding of participants and personnel (performance bias) | Low risk | Not feasible due to nature of intervention | ||
Blinding of outcome assessment (detection bias) | Low Risk | Difficult to ascertain with multiple outcomes | ||
Incomplete outcome data (attrition bias) | Low risk | Cluster level of analysis with all clusters remaining in trial | ||
Selective reporting (reporting bias) | Unclear | Mix of independent and non-independent consultants | ||
Benin RBFHealth, 2014 [21] | Random sequence generation (selection bias) | Low risk | Quantitative component was based on randomized approach | Medium risk |
Allocation concealment (selection bias) | Unclear | Happened at the same time as sequence generation | ||
Blinding of participants and personnel (performance bias) | Low risk | Not likely to be a source of bias | ||
Blinding of outcome as assessment (detection bias) | Low risk | “… any health staff in the T2 group thought that their bonuses were linked to their performance.” | ||
Incomplete outcome data (attrition bias) | Low risk | Not clear | ||
Selective reporting (reporting bias) | Unclear | Evaluation team composition not clear | ||
Burundi Bonfrer et al., 2013 [22] | Random sequence generation (selection bias) | High risk | “...rolled out at the provincial level in a non-randomized way.” | Medium risk |
Allocation concealment (selection bias) | High risk | “...rolled out at the provincial level in a non-randomized way.” | ||
Blinding of participants and personnel (performance bias) | Unclear | Not done | ||
Blinding of outcome assessment (detection bias) | Unclear | “Facilities receive payments based on the quality of quality of health services provided” | ||
Incomplete outcome data (attrition bias) | Unclear | Attrition not discussed | ||
Selective reporting (reporting bias) | Low risk | Different independent consultants with different affiliations. | ||
Cameroon De Walque et al., 2017 [23] | Random sequence generation (selection bias) | High risk | “… was not feasible given that the Government of Cameroon had already decided and announced which districts would be included in the PBF pilot.” | Medium |
Allocation concealment (selection bias) | High risk | Happened at the same time as sequence generation | ||
Blinding of participants and personnel (performance bias) | Low risk | Not done | ||
Blinding of outcome assessment (detection bias) | Unclear | Difficult to assess given the multiple outcomes | ||
Incomplete outcome data (attrition bias) | Low risk | Not likely to be a source of bias | ||
Selective reporting (reporting bias) | Unclear | Mix of independent and non-independent consultants. | ||
DRC World Bank, 2015 [24] | Random sequence generation (selection bias) | Unclear | Not done | High risk |
Allocation concealment (selection bias) | Unclear | Not done | ||
Blinding of participants and personnel (performance bias) | Unclear | Not done | ||
Blinding of outcome assessment (detection bias) | Unclear | Difficult to assess given the multiple outcomes | ||
Incomplete outcome data (attrition bias) | Low risk | Not likely to be source of bias | ||
Selective reporting (reporting bias) | Unclear | Part of researchers affiliated to the World Bank | ||
Mozambique Rajkotia et al., 2017 [25] | Random sequence generation (selection bias) | High risk | “… attempts to control for selection bias using a two-stage approach. First, a matching algorithm was implemented to construct a matched comparison group for all PBF facilities using propensity scores” | Medium risk |
Allocation concealment (selection bias) | Low risk | Not likely to be a source of bias | ||
Blinding of participants and personnel (performance bias) | Low risk | Not done | ||
Blinding of outcome assessment (detection bias) | High risk | “…. we have no way of determining the extent to which improvements in the intervention group are related to better reporting versus better performance.” | ||
Incomplete outcome data (attrition bias) | Unclear | Not likely to be a source of bias | ||
Selective reporting (reporting bias) | Low risk | Some researchers declared conflict of interest | ||
Rwanda (a) Basinga et al., (2011) [27] | Random sequence generation (selection bias) | Low risk | Randomization was done by coin toss | Low risk |
Allocation concealment (selection bias) | Low risk | Happened at the same time as sequence generation | ||
Blinding of participants and personnel (performance bias) | Unclear | Not done | ||
Blinding of outcome assessment (detection bias) | Unclear | Difficult to ascertain to multiple outcomes | ||
Incomplete outcome data (attrition bias) | Low risk | Not likely to be a source of bias | ||
Selective reporting (reporting bias) | Low risk | No evidence of reporting bias | ||
Rwanda (b) Gertler & Vermeersch, 2013 [28] | Random sequence generation (selection bias) | Low risk | “… evaluation employed a stratified cluster randomized designed where districts were first grouped into pairs with common characteristics and then randomly assigned to treatment comparison groups” | Low risk |
Allocation concealment (selection bias) | Low risk | Happened at the same time as sequence generation | ||
Blinding of participants and personnel (performance bias) | Unclear | Not done | ||
Blinding of outcome assessment (detection bias) | Unclear | Not done | ||
Incomplete outcome data (attrition bias) | Low risk | Not follow up cohort design therefore not likely source of bias | ||
Selective reporting (reporting bias) | Unclear | Mix of independent and non-independent consultants | ||
Rwanda (c) Shapira et al., 2017 [29] | Random sequence generation (selection bias) | Low risk | Sectors (sub-districts) in 19 districts were randomly assigned to different study arms | Low risk |
Allocation concealment (selection bias) | Low risk | Not likely to be a source of bias | ||
Blinding of participants and personnel (performance bias) | Low risk | Not feasible for the design | ||
Blinding of outcome assessment (detection bias) | Low risk | “...to measure outcomes prior to the launch of the program, and to establish internal validity of the study” | ||
Incomplete outcome data (attrition bias) | Low risk | “… because the attrition rates were unbalanced between the treatment arms” | ||
Selective reporting (reporting bias) | Unclear | Mix of independent and non-independent consultants. | ||
Zambia Friedman et al., 2016 [30] | Random sequence generation (selection bias) | Unclear | “… selecting districts for the IE was based on district-matched randomization”, however due to budgetary limitations population-based data was only collected in 18 of the 30 study districts, leading to the possible influence of potential unobserved confounders at the district level” | Low risk |
Allocation concealment (selection bias) | Low risk | Happened at the same time as sequence generation | ||
Blinding of participants and personnel (performance bias) | Unclear | Not done | ||
Blinding of outcome assessment (detection bias) | Unclear | Difficult to ascertain to multiple outcomes | ||
Incomplete outcome data (attrition bias) | Low risk | Not likely source of bias | ||
Selective reporting (reporting bias) | Low risk | No evidence of bias | ||
Zimbabwe World Bank, 2016 [31] | Random sequence generation (selection bias) | High risk | “… These 32 districts were purposively sampled from a total of 64 and then pair matched based on observable factors | Medium |
Allocation concealment (selection bias) | High risk | Follows the same risk as random sequence | ||
Blinding of participants and personnel (performance bias) | Unclear | Not done | ||
Blinding of outcome assessment (detection bias) | Unclear | Difficult considering multiple outcomes | ||
Incomplete outcome data (attrition bias) | Unclear | Not reported | ||
Selective reporting (reporting bias) | Low risk | No evidence of bias | ||
Nigeria Kandpal et al., 2019 [26] | Random sequence generation (selection bias) | High risk | “… design randomly allocated all the 52 LGAs in the experimental states to either the PBF or DFF arms, however while the PBF versus DFF relies on randomized assignment of LGAs to the two arms, the control comparisons are based on purposively selected states and are quasi-experimental in design” | Medium risk |
Allocation concealment (selection bias) | High risk | Happened at the same time as sequence generation | ||
Blinding of participants and personnel (performance bias) | Unclear | Not done | ||
Blinding of outcome assessment (detection bias) | Unclear | Difficult to ascertain to multiple outcomes | ||
Incomplete outcome data (attrition bias) | Unclear | Not reported | ||
Selective reporting (reporting bias) | Low risk | No evidence of bias |