Increasing protocol suitability for clinical trials in sub-Saharan Africa: a mixed methods study
© The Author(s) 2017
Received: 13 September 2016
Accepted: 27 February 2017
Published: 7 April 2017
The trial protocol is the most important document for clinical trials and describes not only the design and methodology of a study, but also all practical aspects. The suitability of the protocol has a direct impact on the execution and results of the trial. However, suitability is rarely addressed in trial practice and research. The aim of our study was to investigate protocol suitability and to identify suitability-enhancing measures for trials in sub-Saharan Africa.
We used an exploratory mixed methods design. First, we interviewed 36 trial staff at different organisational levels in Ghana, Burkina Faso and Senegal. Second, we conducted an online survey among trial staff in sub-Saharan Africa to investigate trial protocol suitability based on the main themes distilled from the interviews.
Protocol suitability surfaced as a prominent topic in interviews with trial staff, critiqued for its lack of clarity, implementability and adaptation to trial participants as well as to the workforce and infrastructure available. Both qualitative and quantitative investigations identified local site staff involvement in protocol development as the most helpful mean of increasing protocol suitability. Careful assessment of the local context, capacity and cultures, and ensuring that staff understand the protocol were also cited as helpful measures.
Our data suggests that protocol suitability can be increased by discussing and reviewing the protocol with trial staff in advance. Involving operationally experienced staff would be most useful. For multicentre trials, we suggest that at least one trial staff member from each of the sites with the highest expected recruitment rates be involved in developing the protocol. Carefully assessing the context prior to study start is indispensable to ensuring protocol suitability and should particularly focus on the workforce and infrastructure available, as well as the needs and availability of trial participants. To allow for protocol suitability enhancing measures, planners must allocate enough time for trial preparation and solicit feedback and information on context at an early stage. Such prospective planning would increase implementability, efficiency and quality of trials in the long run.
Clinical trials are essential for developing new medicines and for improving disease management. From a public health perspective, clinical trials in sub-Saharan Africa (SSA), where high burdens of disease exist, are of particular importance. Trials conducted in this region face particular setting-specific challenges like deficits in infrastructure and skilled workforce, in addition to the already complex task of performing a trial. Specific additional challenges derive from the difficulties of getting patient information and consent  and the frequent involvement of children.
The most important document in a clinical trial is the trial protocol, the key document for planning, conducting, externally reviewing, overseeing and interpreting a study . The trial protocol provides a rational for the trial, defines trial goals, processes and analysis methods and enables scientific and ethical review. A well-designed protocol is paramount for a successful clinical trial for several reasons. First, the study design described in the protocol significantly affects the costs of conducting the trial . Second, protocol deficiencies may lead to amendments  and protocol deviations, which trigger queries and add to already heavy workloads. Protocol amendments are costly , may jeopardize data integrity  and trial participants’ safety, and cause delays and disruptions of the trial . One study found that nearly half of all amendments may be avoidable . Third, the length and complexity of protocols have increased dramatically over the past decades. Higher protocol complexity is directly associated with a greater number of amendments, lower levels of study performance [3, 7] and increases chances of non-adherence and, hence, of risk and low quality. The frequency of procedures per protocol has also increased at an annual rate of 8.7%, which adds to on-site work burdens . The number of protocol deviations is one of the key measures for trial quality  and protocols are the most important instrument for quality risk management. In summary, the protocol largely determines quality, outcomes, efficiency and potential challenges in clinical trials. Getz et al. state that protocol design may hold the key to achieving higher levels of efficiency . Despite the challenges mentioned above and the apparent importance of the protocol, there is little research on how to optimize the conduct of trials in the North as well as in resource-limited countries [10, 11]. Gheorghiade et al. criticise the limited data available to support best trial practices and that we only rely on experience and judgment .
To standardise the content and ensure the quality of trial protocols, the ICH E6 guideline “Good Clinical Practice” contains a full chapter on trial protocols . The SPIRIT 2013 Statement (Standard Protocol Items: Recommendations for Interventional Trials) is a more comprehensive checklist of recommended items to include in a trial protocol . This checklist was developed based on the argument that high-quality protocols facilitate proper conduct, reporting and external review of clinical trials, and that the completeness of trial protocols was often inadequate. In addition, the World Health Organisation’s website offers instructions for designing and formatting a research protocol . TransCelerate Biopharma Inc. developed the freely available “Common Protocol Template” to improve consistency across the increasingly complex protocols . Other free protocol templates are available on the web [16, 17]; selecting the correct template depends on local laws, regulations and the sponsor. All efforts described above mainly focus on the scientific part of the protocol, which is of most interest to researchers and reflects their training.
However, a trial protocol goes beyond describing the research design. It also serves as an operational manual and must satisfy experts from different backgrounds and disciplines . To date, little emphasis has been placed on protocol operationalization. Getz and Campo state that protocol authors often transfer out-dated and unnecessary procedures into next study designs because they are routinely carried over from long-standing protocol templates and operating policies .
A key aspect of operationalization is protocol feasibility, which is customarily assessed after the protocol has been finalised by the sponsor. It is currently common practice in clinical trials to have a site feasibility assessment and/or a pre-study visit. During both visits, facilities are commonly assessed using a standard template in a checklist format that is often used across studies and is not tailored to the specific operational requirements of the trial protocol. On the global health trials webpage, such a protocol feasibility checklist is freely available . To the best of our knowledge, there is only one study covering this topic . This study highlights that protocol design feasibility is a topic of increasing interest to sponsor organizations and recommends more flexible and adaptive trial designs as well as more rigorous upfront planning and simulation.
In contrast to “feasible”, which is defined as achievable and possible, “suitable” is defined as fit for purpose . Protocol suitability goes beyond feasibility and addresses not only technical aspects of the protocol but also considers settings, environments and culture, as well as effectiveness and efficiency of execution. These are of particular importance, as the protocol serves as a manual for health care providers . Protocols that cannot be effectively executed may result in protocol deviations, amendments, quality issues and safety problems. While feasibility of trial sites is routinely assessed, protocol suitability is a new concept and rarely considered. Meeker-O’Connell et al. stress that improving protocol design, trial planning and quality oversight has a direct impact on inefficiencies like high costs and unsustainability . With the rising complexity of trial protocols and the intense pressure on sponsors to accelerate development cycle times, suitability is becoming more important to alleviate execution burdens and ultimately improve trial conduct efficiency .
The study presented here covers protocol suitability for clinical trials in SSA that investigate poverty-related diseases. Ensuring protocol suitability is particularly difficult in these regions due to the geographical separation between sponsors and trial teams. To the best of our knowledge, this is the first study investigating trial protocol suitability in SSA. As clinical research is more established in South Africa and not exactly comparable with other SSA-countries, we excluded South Africa from our study .
Clinical trial staff in SSA implement the trial protocols in practice and can provide valuable insights regarding protocol suitability. Nevertheless, the experience of trial staff is rarely acknowledged in scientific publications. Furthermore, Cullati et al. stressed that more research on trial protocols using qualitative methods could shed light on the factors that facilitate the conduct of clinical research . Hence, we assessed trial staffs’ perspectives by using an exploratory mixed methods approach, combining qualitative and quantitative methods. Mixing two methods has the capacity to strengthen results and conclusions . The aim of our study was to identify how protocol suitability could be improved for clinical trials in SSA.
We used an exploratory mixed methods design, which is an ideal approach to exploring a topic for which no research has been carried out so far . We started with a qualitative part, conducting key informant interviews with clinical trial staff working in SSA, to identify important variables of protocol suitability. In order to quantify identified variables, increase generalizability and explore correlations between variables, we followed up with a quantitative part comprising an online survey targeting trial staff. We used the connection approach, deriving major themes from the qualitative interviews and using them to develop and formulate the questions and answer options in the quantitative survey .
Ethical review exemption for the whole project was granted by the Ethics Committee of North-Western and Central Switzerland (EKNZ), based on the rationale that the research project did not involve access to or collect private, sensitive or health-related data or materials. For the qualitative study, we received full ethical clearance from the Ghana Health Service Ethical Review Committee (GHS-ERC: 18/09/14), the Comité d’Éthique sur la Recherche en Santé in Burkina Faso (N 2014-11-131) and the Comité National d’Éthique pour la Recherche en Santé in Senegal (n12/MSAS/DPRS/CNERS).
We visited clinical research centres in Ghana, Burkina Faso and Senegal as they significantly contribute to public health activities in SSA and because the Swiss Tropical and Public Health Institute (Swiss TPH) has contacts with clinical research centres in these countries. In all three countries, we contacted the major clinical research centres that focus on poverty-related diseases and have a track record of completed clinical trials (no more than four such centres could be identified per country). In every country, we selected the first two research centres that agreed to our visit and ultimately conducted interviews in six centres, four of which were located in an urban setting and two in a rural setting. To ensure anonymity of interviewees, neither the names nor the exact locations of the clinical research centres are mentioned here. Interviews were open to all centre investigators, study coordinators, clinicians and quality assurance professionals with at least six months of experience in clinical research. In each centre, the sample was drawn with the assistance of a clinical researcher working in the centre, who approached eligible participants and acquainted them with this study.
In your experience what is important for a good study protocol that is easy to implement?
Could trial protocols be improved? If yes how and where?
What is the influence of the study protocol on the trial?
Who is writing the protocols you are working with?
All interviews were conducted by the first author of this paper. In Ghana, key informant interviews were conducted in English in December 2014. After translating the interview guide into French (including back-translation and terminology review), we conducted interviews in Burkina Faso and Senegal in March and April 2015. In each country, we considered having reached saturation of information in the number of interviews conducted when few or no new concepts were raised . Unstructured observations, reflections during interviews and informal conversations with external monitors (who were on-site during our visit) were collected and documented in a field diary.
After explaining the purpose of the study and informing the participants of their right to withdraw from the study at any given time, participants gave written consent. Interviews were tape-recorded, transcribed verbatim and analysed using thematic analysis as per Braun and Clarke . After repeated reading of the transcripts, initial coding was performed in MAXQDA 11. The analysis focused on the suitability of trial protocols. To ensure the analysis was reflective, notes were taken. We tested emerging themes from the data interrogation in further interview analyses. Themes were cross-tabulated to explore differences between countries and staff levels before finally defining and naming themes. This study adhered to consolidated criteria for reporting qualitative research (COREQ) .
The survey was based on the key themes that emerged from the qualitative interviews, namely protocol characteristics, context adaptation and involvement of site staff. We developed the survey in a team that included clinical researchers, a statistician and a social scientist and discussed it with and received input from the European & Developing Countries Clinical Trials Partnership (EDCTP), a funder of investigator-initiated trials and active in SSA since 2003. The resulting survey (Additional files 1 and 2) consisted of single and multiple selection questions and ranking of table lists related to the following topics: protocol characteristics, adaptation of procedures and practical aspects in the protocol, measures to increase protocol suitability, and current and most helpful involvement in protocol development. The survey also captured the experience of participants and the degree to which measures were implemented. The survey was deployed using a web-based survey tool developed for researchers at the University of Basel (FlexiForm©).
The survey was piloted among 12 participants who had varying positions in the field of clinical research in SSA. As the relevance of the questions had already been tested in the qualitative interviews, the pilot run focused on the comprehensibility and clarity of questions.
Organisation distributing the survey
Number of trial staff in SSA receiving the survey by email
European and Developing Countries Clinical Trials Partnership (EDCTP)
80 investigators who had previously coordinated an EDCTP grant
Swiss Tropical and Public Health Institute
109 trial staff who worked on the RTS,S malaria vaccine trials in SSA
40 trial staff contacts from Swiss TPH
Two pharmaceutical companies
43 trial staff
European Federation of Pharmaceutical Industries and Associations (EFPIA)
22 trial staff
In the introductory text, we informed respondents that by filling in and pressing the “send” button they were giving consent to participate in the survey. In addition, respondents were assured of their anonymity and that it would not be possible to link the answers to their email-addresses. Respondents were informed that if they could not give a general answer, they should answer the question with reference to an on-going or most recent trial.
Categorical variables were described using absolute and relative frequencies and percentages. Explorative factor analysis (based on principal component analysis) with oblique rotation was performed on the survey items to identify potential associations between the protocol quality score (basing on different assessment variables) and personal characteristics of the respondents. The resulting factor scores were then regressed on personal characteristics of the respondents. Independent variables for the regression analyses were selected based on prior knowledge and experience; other potential covariates were screened but did not improve the model. A p-value smaller than 0.05 was considered statistically significant. Data were analysed using the statistical software STATA 14.
Role and experience of interviewees
(n = 8)
(n = 16)
(n = 12)
Role in trial
(n = 16)
(n = 10)
(n = 6)
(n = 4)
Clinical research experience
0 to 1 year
2 to 4 years
5 to 7 years
More than 7 years
Phase I (a or b)
Type of trial
With high frequency, interviewees reported that a suitable trial protocol has to be clear to avoid methodological and procedural uncertainties that leave room for interpretation. Trial staff emphasized the importance of making protocols understandable for everyone on site, including less skilled staff like field workers, and consistent to avoid ambiguities and contradictions. A bit less frequently, interviewees mentioned the need to make protocols easy to implement, i.e. avoiding too many measurements at the same time. In addition, a few interviewees claimed that a logical flow was sometimes missing and called for well-structured protocols.
“A long document to read can cause a problem. Ideally, summarised protocols that get to the essential points could be better for both the researcher and the ethics committee. This facilitates understanding and implementation on the ground. So that's important.”
— Study coordinator, male, Burkina Faso, centre two
Interviewees from French-speaking countries stated the need to translate the protocol into French, as technical staff are unlikely to understand English. According to interviewees, protocol translations were often of bad quality, leading to errors and ambiguities.
Importance of context adaptation
“I would tell you to try to really adapt to the realities of the countries. If you give us a typical European protocol that has to be reproduced here, I think we are going to have problems. We do not have the same manner of working. We do not have the same tools to work with. So it might be important to really see what is feasible in the country (…) If not, you will have many, many deviations afterwards, because we were not able to do that.”
— Study coordinator, female, Burkina Faso, centre two
Interviewees gave various examples of missing context adaptation. First, their biggest concern was the needs of trial participants, which protocols sometimes failed to consider. Protocols should seek to burden trial participants as little as possible. For example, interviewees in all three countries asserted that trial participants felt uncomfortable with blood drawings; one Senegalese investigator said participants would rather accept four small tubes instead of one big tube hence, it is important to ensure that trials are as non-invasive as possible and to discuss limitations in advance. Another example referred to the heavy agricultural workload of local populations during the rainy season; many trials deliberately take place in this season due to high disease prevalence of malaria, for example. Thus, trial procedures should adapt to the time constraints of its participants. Second, interviewees found that socio-cultural norms and values were sometimes not respected in the protocol. Interviewees gave various examples of this, like asking trial participants about death or sexuality, which are taboo subjects in these settings. A few interviewees mentioned that trial participants would not answer these questions honestly. Other examples included performing HIV tests or pregnancy tests on minors or asking the name of neonates when neonates are not given names in their first seven days of life. One interviewee stressed the importance of having a male and a female area for clinical trials in Muslim environments. Interviewees believed that better adaptation to possibilities and attitudes of trial participants would also improve participants’ adherence to trial protocols and decrease losses to follow-up. Third, a few interviewees reported poor or no adaptation to local capacities, systems and/or the structure of the national health system. Staff experiences revealed that certain laboratory tests or the amount of workforce or expertise (e.g. presence of a psychologist) may not be available on site but were required by the protocol. A few interviewees claimed that the protocol timelines given for patient flow were written for ideal settings and circumstances, but not achievable in practice. Respondents were aware that full adaptation to the site was not possible for multicentre trials. However, they reported that for certain multicentre trials, they were allowed to adapt some sections or details to their setting, such as adapting the formulation of questions, which increased protocol suitability.
According to interviewees, protocols were not adapted to local realities because the ones who elaborated the protocol did not know the context. Hence, some procedures in the protocols were difficult to put into practice. Interviewee experiences revealed that it was best to adapt the protocol to the setting in the development phase, as it is far more challenging to adapt a finalized protocol.
Ideas for improving protocol suitability
“So I think that involving the researcher in writing the protocol allows one to avoid challenges in the field. Because it is him [the researcher] who knows his setting well.”
—Investigator, male, Burkina Faso, centre one
Interviewees had different suggestions on how best to involve the trial team in protocol development. A few participants proposed holding discussions with relevant stakeholders prior to writing the protocol, while others suggested writing the protocol together with the sponsor. A few recommended asking clinical trial staff to review the first draft of the protocol, with the aim of checking trial feasibility and providing added feedback. A few others preferred to wait until the protocol was finalized and then discuss the implementation of the trial in practice with the sponsor. For all trial staff, the objective of their involvement would be to ensure that the trial respected the realities of the setting and centre. Additionally, a few mentioned that their input regarding recruitment was of particular importance and would potentially accelerate recruitment rates.
“There have been protocol meetings on many studies, but it is not all the studies that you get the opportunity to be part of the protocol development and you find out that in instances when you are not part of which and where, you know a training did not trickle down well to the end-users, myself included, there may be errors caused.”
— Study coordinator, male, Ghana, centre one
“One the PIs should have it [the protocol]. And the PI also has the responsibility of sub-delegate (…) if you were the PI, it doesn’t necessarily mean you are the technical person in some of the areas. So it is not enough for the PI to just look at it and say ‘oh the science is ok’, you need the technical people to look at it and then they advise ‘ok this way’.”
— Quality assurance professional, male, Ghana centre one
“When we work with a pharmaceutical company, it's difficult to get everyone involved. But at least the PI may be involved.”
— Investigator, male, Burkina Faso, centre two
Trial staff cited additional benefits of their involvement to trial efficiency. According to interviewees, their involvement would decrease the number of amendments, help to find redundancies in the trial processes and improve the preparation of staff for the trial. They were dissatisfied with only executing protocols and claimed that collaboration was missing. Trial staff was also of the opinion that staff motivation would increase if they were allowed more influence on the protocol.
“When the monitors come for the training, we go through documents and we say when such things in our setting cannot be done like that. Then we have to go back again. This is what I have criticized sometimes. We must amend, go back, start again. Because if you amend, we have to resubmit and so on. Whereas, if maybe we could tolerate that for some studies, you can select the site first and the whole protocol development process is done together with the site. This will allow one to take into account many aspects and once we start the process, we won’t need to go backwards anymore.”
— Quality assurance professional, male, Burkina Faso, centre two
“Then we realized that the test run was our secret. That was our success because we had virtually identified all the possible problems, looked at how they could be resolved before the real test.”
— Quality assurance professional, male, Ghana, centre two
A few interviewees suggested including trial participants’ perspectives in the protocol development, as patient challenges occur very frequently, e.g. during informed consent and follow-up. Involving trial participants in discussions and knowing their perspectives would help to increase protocol adherence, according to interviewees. Having “lessons learned” meetings after trial completion and providing sponsors with information about what went wrong was also deemed to have a positive influence on future trials. According to interviewees, identifying weak spots and finding solutions prior to writing the next protocol would avoid repeating the same mistakes and allow staff to profit from experience.
Role and experience of survey participants
Number of participants, n (%)
Recent primary role in clinical research
Quality assurance professional
Experience in clinical research
0 to 1 year
2 to 4 years
5 to 7 years
More than 7 years
Experience in drug trials
Experience in vaccine trials
Other than pharma companies
I do not know
Distribution of survey participants per country
Number of participants, n (%)
Several countries in SSA
Characteristics and context-adaptation of protocols
not at all [%]
no opinion [%]
Understandable (for all staff levels)
Easy to implement
Clear (no uncertainties)
Consistent (e.g. no ambiguities)
Adaptation of protocol procedures and in-protocol required resources to the setting
marginally adapted [%]
partially adapted [%]
well adapted [%]
no opinion [%]
Informed consent procedure
Inclusion and exclusion criteria
Participant incentives for participation
Data and information to be collected
Medical procedures and decisions
Safety reporting and management
Amount of workforce available
Availability and needs of trial participants
Daily clinical practice
Ethics committee system
Drug regulatory authority system
The majority of respondents (56%) mentioned that protocols were amended an average of three to five times per trial; 7.3% mentioned more than five protocol amendments per trial.
Measures to increase protocol suitability
This result was consistent across countries and positions. A related frequently selected approach to increase protocol suitability was to have committees of investigators in multicentre trials (42.7%), consisting of investigators from all participating sites. On average, respondents were more frequently working on multicentre trials than on single centre trials (77% spent at least 50% of their work time in multicentre trials). The second most helpful option mentioned was careful assessment of local context, capacity and culture by sponsors (49.1%). However, the adaptation of the protocol to site and health care specific systems in a single centre trial (11.8%) was only rarely selected. Overall almost half (47.3%) rated ensuring that everybody understands the protocol and knows his/her role and responsibility in the trial as one of the most helpful options, while the more concrete measures of having a kick-off meeting or a dry run were less frequently chosen (30.9 and 27.3%). Almost half (41.3%) reported to have had a kick-off meeting for all of their trials. Dry runs were less frequently implemented; 24.8% never had dry runs in any trial and 21.1% had dry runs in all trials. Soliciting feedback from the site on what went wrong in previous trials by sponsors was another popular option for respondents (42.7%) to increase protocol suitability. However, 40.4% had never had any “lessons learnt meetings”. Only 1.8% considered that the use of the open source protocol development technique would increase protocol suitability, but 73.4% had never heard of this technique.
Respondents were asked to tick the three options they considered most helpful for increasing protocol suitability. Because multiple selection was possible, 13.6% ticked more than three options. Percentages of answers from those that chose more than three options were compared with the answers of respondents who correctly filled in the survey; the ones that chose more than three options ranked checklist for practical steps, assessment of setting, inclusion of trial participants’ perspective and adaptation to site as slightly higher. However, including the ones that chose more than three options did not change the measures’ ranking order and, due to the small sample size, we included all answers in the final results and figures.
Factor analysis of the different assessment variables was performed to assess potential associations between protocol quality and personal characteristics. The third factor score was most strongly related to personal characteristics of the study participants and described the degree of implementability, understandability, clarity and structure of the trial protocol. The only statistically significant association found was the professional role of survey participants. Compared to other professional roles, the third factor score was on average significantly higher for PIs (difference = 0.43, 95%-confidence interval = 0.04 to 0.83, p = 0.033) and significantly lower for sponsor-investigators (difference = −1.10, 95%-confidence interval = −1.85 to −0.35, p = 0.005) and quality assurance professionals (difference = −1.22, 95%-confidence interval = −2.02 to −0.42, p = 0.003).
This study identified perceived deficits of protocol suitability and yielded several measures to enhance protocol suitability, as put forward by trial staff. According to trial staff, protocol suitability constraints included ambiguity, complexity and poor understandability for all staff levels. Staff mentioned lack of clarity in procedural descriptions and imprecise wording in protocols. Only one third of the survey respondents rated protocols as easy to implement. While context adaptation was a main theme in the qualitative interviews, survey respondents rated most trial aspects as rather well adapted, particularly inclusion and exclusion criteria and follow-up procedures. This finding surprised us and is inconsistent with the literature, which cites follow-up as a challenge [31–33]. The differences in the findings might be explained by the different methodologies used as literature findings base on authors’ personal reflections. In turn, respondents cited poor context adaptation of the protocol to the availability and needs of trial participants as a constraint. The importance of adapting projects to research participants’ cultural norms and values has been described elsewhere [1, 33, 34]. Staff perceived protocols as being too rigid for their settings. An example that was mentioned frequently in interviews was the importance of minimising blood draws. Trial participants in SSA are commonly scared of giving blood as blood is considered sacred, blood sampling is thought to make children ill and there are local rumours surrounding “blood stealing” and “blood selling” [1, 35, 36].
According to trial staff, visit windows and patient flow should also consider trial participants’ obligations, e.g. that harvesting takes place at the end of the rainy season. The lack of protocol adaptation to available site workforce and to daily clinical practice, of particular importance in hospital settings where trial activities are added to routine care, indicates that the work burden of trial implementation was underestimated by protocol developers.
The literature also stresses the importance of context adaptation for easy translation of protocol procedures into practice [37, 38]. Alsumidaie states that, “The sponsors design clinical trials expecting them to fit into study site operational infrastructures which leads to challenges like study procedures that are incompatible with how study sites operate” .
Context adaptation is more challenging in multicentre trials, as procedures must be uniform across sites to enable the required pooled data analysis. However, Thomason et al. state that standardisation of procedures across all sites within a trial in SSA is not always possible due to the differences in resources . We argue that, for multicentre trials, the degree of adaptation to the context has to be considered carefully to avoid protocols that are overly site specific but that consider the cultural, social, economic and political differences between the sites involved. We suggest identifying commonalities among the sites involved and accounting for differing socio-cultural norms, but we acknowledge that this is difficult.
Measures to improve protocol suitability
Involvement of site staff in protocol development
In both qualitative and quantitative results, local trial staff involvement in developing the protocol came up and was rated as the most helpful option for increasing protocol suitability. The importance of site staff involvement in trial planning has also been stressed in the literature [11, 21, 32, 39]. Eastabrook et al. state, “Given the important role of site staff for overall trial success it is critical to understand their preferences and experiences” , while Alsumidaie promotes trial site involvement to create clinical trials that work operationally while reducing risks .
Half of the interviewees and three-quarters of survey respondents (half of survey respondents were investigators) had been previously involved in protocol development in some manner. It is important to carefully choose the composition of the trial team involved in this process. While key opinion leaders may give detailed scientific input, they are often not the ones carrying out the work on site. We agree with the respondents who suggested involving operationally experienced staff as particularly useful to increase suitability. Ideally, technical staff from various functional areas (investigator/clinician, study coordinator, pharmacist, lab coordinator and data manager) should be involved, though this might not be possible for all sites in a multicentre trial. A popular option to increase protocol suitability for multicentre trials, as revealed in the survey, was to form protocol writing committees consisting of one investigator from each site. Indeed, involving an operationally experienced investigator who solicits feedback from his/her team and communicates the outcome to the sponsor would lead to optimized protocols. Multicentre trials on poverty-related diseases do not usually involve an extensive number of trial sites, hence, it should be possible to solicit feedback from all sites in the planning phase of such trials. In cases where it is not possible to involve one investigator from each site, at least the sites expected to have a high enrolment rate should be involved in protocol development. Mbuagbaw et al. recommend selecting national coordinators to participate in the conception of multicentre trials . The assistance of a social scientist would also help to inform protocol development by identifying context specificities. In line with this, Cooper et al. also suggest the use of qualitative research to identify the acceptability of the trial protocol among other things . The meningitis vaccine project shares a positive experience of bringing trial teams together at meetings in the preparatory phase, which empowered the team and fostered communication between sites . In addition, this approach enabled planners to anticipate and resolve operational issues and minimize the number of protocol deviations.
Trial staff rated reviewing and pre-discussing the protocol as the most helpful way to participate in protocol development. This is in line with Alsumidaie, who states that involvement is mainly about obtaining feedback on how to better operate the study . As the sponsor is responsible for the research question and scientific aspects of the protocol, the trial team could provide valuable input in terms of protocol clarity, implementability and adaptation to trial participant needs. The latter is in line with literature stating that staff input would be particularly important for recruitment and follow-up of trial participants  as well as feasibility of scheduled study visits . We consider it essential, that the site, in turn, is transparent and realistic in terms of their capacities.
In addition to increasing protocol suitability, trial staff mentioned several additional advantages of being involved in the protocol preparation process. First, developing an appropriate protocol is a discipline that requires training  and, according to interviewees and the literature, involving local trial staff in this process builds capacity and confidence [31, 43]. As there are only a small number of locally initiated trials  and limited career perspectives in clinical research [1, 45], this is a crucial skill for investigators to acquire. Second, staff saw being involved in protocol development as a way of improving their preparedness for the trial. Third, having an influence on the project would increase trial staff’s motivation, as opposed to having a project forced upon them. In contrast to a top-down approach, a participatory approach fosters ownership of the trial [38, 43] and is also recommended by the transboundary research principles of the Commission for Research Partnerships with Developing Countries (KFPE) . Experiences from the meningitis vaccine project also show that working closely with study staff can be empowering, strengthen team spirit, boost staff motivation and increase everyone’s commitment . Fourth, it has been stated that mobilising collective intelligence of various people for research protocols is a great benefit for research .
Assessment of the context and setting
More careful assessment of the local context, capacity, and culture by the sponsor was rated as the second most helpful measure to increase protocol suitability. Experiences from a trial in the Gambia are in line with this; the authors state that baseline situation assessments are required as each trial and site is unique . For this purpose site assessments and pre-study-visits are commonly performed by sponsors or contract research organizations. While it is common practice to use standard templates (checklists), information collected this way is of limited value since they are frequently focusing on issues influencing trial feasibility. Instead, the questions should be tailored to the trial and the setting, i.e. to the suitability. This includes, for example, a more thorough assessment of the socio-cultural context, local laws and customs where the trial will be conducted and identification of risks and needs upfront. This is in line with current trends toward risk-based approaches, including risk assessments in clinical development .
Our data suggest improvement of both, feasibility and suitability assessments with a focus on workforce, infrastructure (e.g. equipment) available and on availability of trial participants, respectively. Some established clinical research centres in SSA have community advisory boards (CABs) for community engagement and to inform the community appropriately about the study . In view of our findings, it might be helpful to involve the CAB at the conception stage of a trial, to allow for socio-cultural adaptation. Additionally, we found that engaging key staff from different organizational levels during the visits was beneficial. It is important to allocate enough time for such visits and ensure that they are conducted at an early project stage, where changes can still be incorporated. Moreover, it is important that the monitors performing these visits are well qualified, having both the requisite therapeutic knowledge and cultural sensitivity on top of the generally required knowledge of processes, protocols, regulations, laboratories and experience in clinical research.
The two measures “involvement of site staff in protocol development” and “assessment of context and setting” are not only increasing suitability but also feasibility. However, these measures go beyond feasibility through enabling effective and efficient execution of the trial and considering settings and cultures by following the protocol.
Good understanding of trial protocols
Making sure that everyone understands the protocol and knows his/her role and responsibility in the trial is an important factor, as put forward by trial staff and the literature . However, it is challenging in practice, as often staff in SSA, particularly in trials on neglected diseases, have neither a medical background nor prior experience in clinical research . A survey on site initiation visits confirmed our findings, as protocol specific training emerged as a main request by trial staff . The initiation visit presents an ideal opportunity for the site staff to go through each trial procedure in detail and discuss the operationalization of the protocol with the monitor. To coach the team, ensure compliance with the trial protocol and help to correct practices, ideally, the monitor should remain on site during the first few days of recruitment or re-visit the site shortly after recruitment has started. Similarly, Tinto et al. suggested that the Good Clinical Practice trainer supports the trial team in resource-limited settings during trial start .
Advantages of prospective planning
We are aware that the suggestions presented here to increase protocol suitability involve costs and might cause study start delays. However, a recent study by CTTI (Clinical Trials Transformation Initiative) confirmed that to overcome inefficiencies, an approach that emphasizes error prevention over remediation should be the norm . Currently, the intense monitoring, auditing and inspecting processes test for errors during the trial rather than prospectively identifying, preventing and correcting them. In their quality by design project, the CTTI authors suggest, in line with our findings, that protocol issues should be identified early to minimize their impact and to describe the infrastructure, resources and training needs of the site . Another study showed that to ensure data integrity, training and motivating sites is much more cost-effective compared to 100% source data verification . To ensure quality, CTTI encourages critical thinking, addressing implementation challenges proactively and incorporating lessons learned into other trials as a means of continuous improvement. Protocol suitability-enhancing measures may also reduce the number of amendments, minimizing its negative impact on costs, duration, and quality of trials — particularly important given that the majority of survey respondents indicated three to five amendments, on average, per protocol.
Despite the intense pressure on sponsors to accelerate drug development , realistic timelines and sufficient time for trial preparation is important for implementing protocol suitability enhancing measures. This is in line with CTTI’s assertion that, “Rewarding trial teams who minimize the time to first patient enrolled may serve as a disincentive to devoting time to identifying and preventing errors that matter through trial design” .
To ensure that site staff involvement does not delay protocol development and that the process is as efficient as possible, it is important that one person leads the process on the sponsor side and that the protocol development process is clearly defined. The global health trials webpage offers a concept protocol template that provides a format for recording discussions and for presenting a protocol to stakeholders at an early stage . Another promising tool for cost-effective and efficient involvement of stakeholders in protocol development is SWOG, a web-based protocol writing system with integrated support for collaborative reviewing and editing . This tool enables sponsors and trial teams to see each other’s comments and reactions immediately, despite the geographical separation, which is particularly large for clinical trials in SSA . The objective of the software developer was to increase the natural collaborative protocol writing process and facilitate interactions and communications among protocol writers .
Due to time restrictions in this investigation, we covered the site teams’ perspective without incorporating the sponsors’ view. The interviewer of the qualitative interviews is a female Swiss scientist, which may have biased interviewees towards giving a positive answer. The survey sample is a convenience sample, as it is impossible to eliminate nonresponse bias in online surveys. However, triangulation of perspectives through mixed methods approach strengthened our findings. As EDCTP, pharmaceutical companies and Swiss TPH sent out the survey and are simultaneously potential sponsors or funders, survey respondents might have had a tendency to answer questions in a manner that would be viewed favourably by these organisations. However, we tried to mitigate this concern by ensuring anonymity in the survey’s introductory text. Another limitation was incomplete responses received for a few questions. Lastly, we acknowledge that according to the sample size calculation, 200 survey answers would have been required, but despite many efforts we only received 110 answers. Possible reasons were poor internet connectivity and time constraints of trial staff. However, one should consider that the total number of clinical research staff working in SSA is likewise relatively small and we believe that our results are representative for established clinical research centres in SSA.
We speculate that many of our findings are also applicable for Northern settings. Due to the lack of literature on the topic in general and particularly in SSA, the majority of literature cited in this manuscript is based on the Northern setting and is mostly in line with our findings. An example that is equally true for Northern and Southern settings is the practice of sponsors designing clinical trials, expecting them to fit to the trial site . Based on trial experiences in Northern settings, Farrell et al. recommend that differing clinical practices, working environments, and governance regulations should be taken into account  and CTTI recommends involving different levels of site staff to increase the quality of the trial .
To the best of our knowledge, this is the first study on trial protocol suitability for clinical trials in SSA. We encourage further research on trial protocols and their non-scientific parts, in particular. We promote the exploratory mixed methods methodology in the context where little is known about the research topic, as this approach allows new and important themes to emerge and provides the flexibility to adapt to these themes in subsequent steps.
By applying an exploratory mixed methods approach, we identified a lack of clarity, implementability and adaptation to trial participants, workforce and infrastructure as the main constraints of protocol suitability. We found that site staff involvement in protocol development, careful assessment of local context, capacity and culture as well as ensuring that staff understands the protocol are the most helpful measures towards increasing protocol suitability, according to trial teams. Including technical aspects into such preparations and site involvement would simultaneously also enhance the feasibility of trials in the reviewed context. Considering and involving the site’s input at an early stage of protocol development was deemed the best way to increase involvement, as the majority of trial staff did not seek major involvement in protocol development. Our data suggests that the measures presented increase implementability, efficiency and quality of trials in the long run, although it might slightly prolong the protocol development phase. We consider such an approach as particularly useful for clinical trials in SSA, as the protocols are mostly developed by Northern sponsors who might not be familiar with the setting.
Community advisory board
Contract research organization
Clinical Trials Transformation Initiative
European & Developing Countries Clinical Trials Partnership
- Swiss TPH:
Swiss Tropical and Public Health Institute
The authors thank Dr. Christian Schindler for his statistical support. The authors also acknowledge the help of European & Developing Countries Clinical Trials Partnership (EDCTP), particularly Dr. Perry Mohammed, Dr. Pauline Beattie and Dr. Ole F. Olesen for distributing the survey. We are grateful to Amena Briet for editing the manuscript. We thank Prof. Dr. Marcel Tanner who supported the survey distribution. We acknowledge Dr. Julie Catusse for helping with the translations. We would like to thank all clinical trial professionals who participated in our study.
This study was financially supported by the R. Geigy Foundation (http://www.geigystiftung.ch/en/), the Burckhardt-Bürgin-Stiftung (https://www.unibas.ch/de/Forschung/Finanzierung/Doktorat/Burckhardt-Buergin-Stiftung.html) and the Freiwillige Akademische Gesellschaft (http://www.fag-basel.ch/). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Availability of data and materials
Short interview excerpts and the interview guide are available within the text. Qualitative data cannot be made publicly available as study participants consented to interviews with the understanding that their data would remain anonymous and confidential. Further excerpts of study data are available from the corresponding author on reasonable request.
The study was designed by NV, CB, CP. The data were analyzed and interpreted by NV, CB, CP, JK. Manuscript preparation was done by NV, CB, CP, JK. All authors contributed critically and significantly to drafting a final manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Idoko OT, Kochhar S, Agbenyega TE, Ogutu B, Ota MO. Impact, challenges, and future projections of vaccine trials in Africa. Am J Trop Med Hyg. 2013;88(3):414–9.View ArticlePubMedPubMed CentralGoogle Scholar
- Chan A-W, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200–7.View ArticlePubMedPubMed CentralGoogle Scholar
- Getz K. Improving protocol design feasibility to drive drug development economics and performance. Int J Environ Res Public Health. 2014;11(5):5069–80.View ArticlePubMedPubMed CentralGoogle Scholar
- Tufts Center for the Study of Drug Development. Impact Report: Amendments reduce number of patients, but at high cost, longer study times. 2016. Available from: http://csdd.tufts.edu/files/uploads/Summary-JanFebIR2016_.pdf. Accessed 24 June 2016.
- Chan A-W, Tetzlaff JM, Gøtzsche PC, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013;346: e7586. doi:10.1136/bmj.e7586.
- Getz K, Stergiopoulos S, Short M, Surgeon L, Krauss R, Pretorius S, et al. The Impact of Protocol Amendments on Clinical Trial Performance and Cost. Ther Innov Reg Sci. 2016. doi:10.1177/2168479016632271.
- Getz K, Campo R. Drug development study designs have reached the danger zone. Expert Rev Clin Pharmacol. 2013;6(6):589–91.View ArticlePubMedGoogle Scholar
- Getz KA, Wenger J, Campo RA, Seguine ES, Kaitin KI. Assessing the impact of protocol design changes on clinical trial performance. Am J Ther. 2008;15(5):450–7.View ArticlePubMedGoogle Scholar
- Alsumidaie M. Eli Lilly Unveils Innovative Study Design Platform. 2014. Available from: http://www.appliedclinicaltrialsonline.com/eli-lilly-unveils-innovative-study-design-platform. Accessed 24 June 2016.
- Geldenhuys H, Waggie Z, Jacks M, Geldenhuys M, Traut L, Tameris M, et al. Vaccine trials in the developing world: operational lessons learnt from a phase IV poliomyelitis vaccine trial in south Africa. Vaccine. 2012;30(40):5839–43.View ArticlePubMedGoogle Scholar
- Mbuagbaw L, Thabane L, Ongolo-Zogo P, Lang T. The challenges and opportunities of conducting a clinical trial in a low resource setting: the case of the Cameroon mobile phone SMS (CAMPS) trial, an investigator initiated trial. Trials. 2011;12(1):145.View ArticlePubMedPubMed CentralGoogle Scholar
- Gheorghiade M, Vaduganathan M, Greene SJ, Mentz RJ, Adams KF, Anker SD, et al. Site selection in global clinical trials in patients hospitalized for heart failure: perceived problems and potential solutions. Heart Fail Rev. 2012;19(2):135–52.View ArticleGoogle Scholar
- International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use. Guideline for Good Clinical Practice E6. 1996. Available from: http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R1_Guideline.pdf. Accessed 20 Oct 2015.
- World Health Association. Recommended format for a Research Protocol. Available from: http://www.who.int/rpc/research_ethics/format_rp/en/. Accessed 15 Jul 2016.
- TransCelerate Biopharma Inc. Common Protocol Template. Available from: http://www.transceleratebiopharmainc.com/assets/common-protocol-template/. Accessed 24 June 2016.
- NHS. Protocol guidance and template for use in a clinical trial of an investigational medicinal product (CTIMP). 2015. Available from: http://www.hra.nhs.uk/about-the-hra/consultations-calls/closed-consultations/protocol-guidance-template-use-clinical-trial-investigational-medicinal-product-ctimp-consultation-use/. Accessed 24 June 2016.
- FDA and NIH. Clinical Trial Protocol Template. 2016. Available from: http://osp.od.nih.gov/sites/default/files/Protocol_Template_05Feb2016_508.pdf. Accessed 24 June 2016.
- Gennari JH, Weng C, McDonald DW, Benedetti J, Green S. An ethnographic study of collaborative clinical trial protocol writing. Stud Health Technol Inform. 2004;107(Pt 2):1461–5.PubMedGoogle Scholar
- Global Health Trials. Templates. Available from: https://globalhealthtrials.tghn.org/resources/templates/. Accessed 24 June 2016.
- Wikidiff. Feasible vs Suitable - What’s the difference?. Available from: http://wikidiff.com/suitable/feasible. Accessed 19 Jul 2016.
- Meeker-O’Connell A, Glessner C, Behm M, Mulinde J, Roach N, Sweeney F, et al. Enhancing clinical evidence by proactively building quality into clinical trials. Clin Trials. 2016;13(4):439–44.View ArticlePubMedPubMed CentralGoogle Scholar
- Irikefe V, Vaidyanathan G, Nordling L, Twahirwa A, Nakkazi E, Monastersky R. Science in Africa: the view from the front line. Nature. 2011;474(7353):556–9.View ArticlePubMedGoogle Scholar
- Cullati S, Courvoisier DS, Gayet-Ageron A, Haller G, Irion O, Agoritsas T, et al. Patient enrollment and logistical problems top the list of difficulties in clinical research: a cross-sectional survey. BMC Med Res Methodol. 2016;16(1):50.View ArticlePubMedPubMed CentralGoogle Scholar
- Creswell J, Plano CV. Designing and conducting mixed methods research. London: Sage; 2011.Google Scholar
- Zhang W, Creswell J. The use of “mixing: procedure of mixed methods in health services research. Med Care. 2013;51(8):e51–7.View ArticlePubMedGoogle Scholar
- Charmaz K. Constructing grounded theory. A practical guide through qualitative analysis. London: Sage; 2006.Google Scholar
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.View ArticleGoogle Scholar
- Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. International J Qual Health Care. 2007;19(6):349–57.View ArticleGoogle Scholar
- Silverman D. Doing qualitative research. London: Sage; 2013.Google Scholar
- ClinicalTrial.gov. Map of All Studies on ClinicalTrials.gov. Available from: https://clinicaltrials.gov/ct2/search/map/click?map.x=428&map.y=284. Accessed 27 June 2016.
- Mwangoka G, Ogutu B, Msambichaka B, Mzee T, Salim N, Kafuruki S, et al. Experience and challenges from clinical trials with malaria vaccines in Africa. Malar J. 2013;12:86.View ArticlePubMedPubMed CentralGoogle Scholar
- Marchetti E, Mazarin-Diop V, Chaumont J, Martellet L, Makadi M-F, Viviani S, et al. Conducting vaccine clinical trials in sub-Saharan Africa: operational challenges and lessons learned from the meningitis vaccine project. Vaccine. 2012;30(48):6859–63.View ArticlePubMedGoogle Scholar
- Thomason MJ, Spyer MJ, Joffe N, Boles J, Burke A, Wilkes HC, et al. Monitoring randomized clinical trials in Africa; pragmatic approaches and experiences from HIV trials. Clin Investig. 2015;5(6):585–97.View ArticleGoogle Scholar
- Kuepfer I, Burri C. Reflections on clinical research in sub-Saharan Africa. Int J Parasitol. 2009;39(9):947–54.View ArticlePubMedGoogle Scholar
- Peeters Grietens K, Ribera JM, Erhart A, Hoibak S, Ravinetto RM, Gryseels C, et al. Doctors and vampires in Sub-Saharan Africa: ethical challenges in clinical trial research. Am J Trop Med Hyg. 2014;91(2):213–5.View ArticlePubMed CentralGoogle Scholar
- Idoko OT, Owolabi OA, Odutola AA, Ogundare O, Worwui A, Saidu Y, et al. Lessons in participant retention in the course of a randomized controlled clinical trial. BMC Res Notes. 2014;7:706.View ArticlePubMedPubMed CentralGoogle Scholar
- Institute of Medicine (US) Forum on Drug Discovery, Development, and Translation. Transforming Clinical Research in the United States: Challenges and Opportunities: Workshop Summary. 2010. Available from: http://www.ncbi.nlm.nih.gov/books/NBK50892/. Accessed 6 April 2016.
- Farrell B, Kenyon S, Shakur H. Managing clinical trials. Trials. 2010;11:78.View ArticlePubMedPubMed CentralGoogle Scholar
- Alsumidaie M. 2015 a Year in Review: The Year of Clinical Trial Innovation. 2015. Available from: http://www.appliedclinicaltrialsonline.com/good-risk-based-management-begins-site. Accessed 23 Aug 2016.
- Jennifer Eastabrook DM, Shawn Patterson Baker, Ruth Cannata. Enhanced Site Training, Resources, and Communication. 2016. Available from: http://www.appliedclinicaltrialsonline.com/enhanced-site-training-resources-and-communication-during-clinical-trials-site-staff-perspectives. Accessed 28 June 2016.
- Cooper C, O’Cathain A, Hind D, Adamson J, Lawton J, Baird W. Conducting qualitative research within clinical trials units: avoiding potential pitfalls. Contemp Clin Trials. 2014;38(2):338–43.View ArticlePubMedGoogle Scholar
- Clinical Trials Transformation Initiative Quality by Design Project. CTTI Recommendations: Qualtiy by Design. 2015. Available from: https://www.ctticlinicaltrials.org/files/ctti_quality_by_design_recommendations_final_1jun15_1.pdf. Accessed 27 June 2016.
- Devasenapathy N, Singh K, Prabhakaran D. Conduct of clinical trials in developing countries: a perspective. Curr Opin Cardiol. 2009;24(4):295–300.View ArticlePubMedGoogle Scholar
- Franzen SR, Chandler C, Enquselassie F, Siribaddana S, Atashili J, Angus B, et al. Understanding the investigators: a qualitative study investigating the barriers and enablers to the implementation of local investigator-initiated clinical trials in Ethiopia. BMJ Open. 2013;3(11):e003616.View ArticlePubMedPubMed CentralGoogle Scholar
- Ndebele P, Blanchard-Horan C, Shahkolahi A, Sanne I. Regulatory challenges associated with conducting multicountry clinical trials in resource-limited settings. J Acquir Immune Defic Syndr. 2014;65 Suppl 1:S29–31.View ArticlePubMedPubMed CentralGoogle Scholar
- Commission for Research Partnerships with Developing Countries (KFPE). KFPE’s Guide for Transboundary Research Partnerships. 2014. Available from: http://www.naturalsciences.ch/organisations/kfpe/11_principles_7_questions?_ga=1.249050371.561251459.1440766959. Accessed 22 Aug 2016.
- Methods in Research on Research (MiRoR). Individual Research Projects. 2015. Available from: http://miror-ejd.eu/individual-research-projects/. Accessed 28 June 2016.
- Cutts FT, Enwere G, Zaman SM, Yallop FG. Operational challenges in large clinical trials: examples and lessons learned from the gambia pneumococcal vaccine trial. PloS Clin Trials. 2006;1(3):e16.View ArticlePubMedPubMed CentralGoogle Scholar
- Brosteanu O, Houben P, Ihrig K, Ohmann C, Paulus U, Pfistner B, et al. Risk analysis and risk adapted on-site monitoring in noncommercial clinical trials. Clin Trials. 2009;6(6):585–96.View ArticlePubMedGoogle Scholar
- Nyika A, Chilengi R, Ishengoma D, Mtenga S, Thera MA, Sissoko MS, et al. Engaging diverse communities participating in clinical trials: case examples from across Africa. Malar J. 2010;9:86.View ArticlePubMedPubMed CentralGoogle Scholar
- Lang T, Siribaddana S. Clinical trials have gone global: is this a good thing? PLoS Med. 2012;9(6):e1001228.View ArticlePubMedPubMed CentralGoogle Scholar
- Tinto H, Noor RA, Wanga CL, Valea I, Mbaye MN, D’Alessandro U, et al. Good clinical practice in resource-limited settings: translating theory into practice. Am J Trop Med Hyg. 2013;88(4):608–13.View ArticlePubMedPubMed CentralGoogle Scholar
- Clinical Trials Transformation Initiative. Crititical to quality factors. 2015. Available from: https://www.ctti-clinicaltrials.org/toolkit/qbd/introduce-qbd/qbd-principles/explore-ctq-factors. Accessed 27 June 2016.
- Goldfarb N. Are site monitoring and data cleaning a waste of time? Journal of Clinical Research Best Practices. 2006;2(11):1–9.
- Weng C, Gennari JH, McDonald DW. A collaborative clinical trial protocol writing system. Stud Health Technol Inform. 2004;107(Pt 2):1481–6.PubMedGoogle Scholar
- Smith SK, Selig W, Harker M, Roberts JN, Hesterlee S, Leventhal D, et al. Patient engagement practices in clinical research among patient groups, industry, and academia in the united states: a survey. PLoS One. 2015;10(10):e0140232.View ArticlePubMedPubMed CentralGoogle Scholar