Implementation and evaluation of crowdsourcing in global health education
Global Health Research and Policy volume 7, Article number: 50 (2022)
Current global health course is most set as elective course taught in traditional teacher-taught model with low credit and short term. Innovate teaching models are required. Crowdsourcing characterized by high flexibility and strong application-orientation holds its potential to enhance global health education. We applied crowdsourcing to global health teaching for undergraduates, aiming to develop and evaluate a new teaching model for global health education.
Crowdsourcing was implemented into traditional course-based teaching via introducing five COVID-19 related global health debates. Undergraduate students majoring in preventative medicine and nursing grouped in teams of 5–8, were asked to resolve these debates in reference to main content of the course and with manner they thought most effective to deliver the messages. Students' experience and teaching effect, were evaluated by questionnaires and teachers’ ratings, respectively. McNemar's test was used to compare the difference in students' experience before and after the course, and regression models were used to explore the influencing factors of the teaching effect.
A total of 172 undergraduates were included, of which 122 (71%) were females. Students' evaluation of the new teaching model improved after the course, but were polarized. Students’ self-reported teaching effect averaged 67.53 ± 16.8 and the teachers’ rating score averaged 90.84 ± 4.9. Students majoring in preventive medicine, participated in student union, spent more time on revision, and had positive feedback on the new teaching model tended to perform better.
We innovatively implemented crowdsourcing into global health teaching, and found this new teaching model was positively received by undergraduate students with improved teaching effects. More studies are needed to optimize the implementation of crowdsourcing alike new methods into global health education, to enrich global health teaching models.
Global health has attracted wide attention worldwide in many fields . Global health concerns not only the health of people but also the training of health workers . China plays a pivotal role in global health , but it lacks systematic global health education and professionals . The Chinese Consortium of Universities for Global Health (CCUGH) was established in 2013 , which stands for the development of global health education in China . Nevertheless, only few Chinese universities provide systematic global health education . Most of them use the traditional teacher-taught teaching models, and the courses are set as elective courses with low credit and short-term duration [7, 8]. This form of curriculum can hardly trigger students’ learning interest, let alone practice skills learned after the course . Given global health is a multi-disciplinary course with strong knowledge application-orientation, how to facilitate students to get actively engaged and apply knowledge acquired to understand global health related issues is the key teaching need.
To promote students' engagement and improve understanding, prior studies have suggested applying problem-based learning , flipped classroom  and participatory teaching . More recently, crowdsourcing as a novel form of solution collection , characterized by user-centered, high creativity and flexibility , has been used in course delivery . Compared with aforementioned teaching models, crowdsourcing has the advantages of high flexibility and strong application-orientation. Centered on the learning tasks set by the teacher, students are on the drivers’ seats to explore the learning source, learning material, and the best manner to demonstrate their learning outcomes. It can not only hone students' ability to apply theoretical knowledge to solve global health related problems, but also increase their interest in self-driven learning. Crowdsourcing has also been widely used in public health researches to improve diagnosis, surveillance, and education , with satisfying results . The extent to which crowdsourcing may promote global health education has yet been explored.
Our study applied crowdsourcing to global health education for undergraduates, aiming to develop and evaluate a new teaching model for global health that can improve students’ engagement and understanding.
The new teaching model based on the main characteristics of crowdsourcing was applied to a ten-week undergraduate course "Introduction to Global Health" of Sun Yat-sen University from May to July, 2021.
The new teaching model consisted of two parts, crowdsourcing debates and traditional course-based teaching. To enhance students' ability to apply what they learned and to increase their interest in learning, we designed a crowdsourcing debates section. In this part, five COVID-19 related global health topics for debates were provided to all students. Students were free to form teams and work on a topic according to their interests. Each team consisted of 5–8 members including a leader. There were 4–6 teams under each topic with a subject teacher supervising them. The teacher organized a discussion among these teams two-weeks before the end of the course. Students can study the topic from multiple dimensions according to their interests and the main content of the course. Then they would choose a manner that they thought most effective to deliver the messages (e.g. video, comedy, debate, etc.) (Additional file 1: Table 1), and some of the demonstrations were interactive (such as interviews). Additionally, we designed a question-and-answer session to increase the participation of other students. Figure 1 shows the topics of crowdsourcing debates. The traditional course-based teaching part was designed to complement students' knowledge of global health. According to the syllabus of “Introduction to Global Health”, basic concepts, theories, and research methods related to global health were taught by teachers every week (Additional file 1: Table 1). The overall design of the new teaching model in this study is shown in Fig. 2.
“Introduction to Global Health” was a compulsory course for undergraduate students majoring in preventive medicine and nursing. Therefore, the participants of our study were 152 students majoring in preventive medicine in grade 2017 and 49 students majoring in nursing in grade 2019, with a total of 201 students. Exclusion criteria: 1) Students who have not completed the course due to suspension or dropping out; 2) Unwilling to cooperate. We obtained the informed consent of all students and the Ethics Committee of the School of Public Health, Sun Yat-sen University has approved this study (#SYSUNO.085).
Questionnaire and data collection
We collected data from two aspects. One was students’ self-reported questionnaire before and after the course. The other was the score rated by five teachers based on the same criteria which was for students demonstrating their learning outcomes at the end of the course. We regarded this score as the objective teaching effect. The questionnaire was self-compiled, including basic information, students’ experience, and subjective teaching effect.
Basic information includes gender, major, participation in student union, team leader, team participation, and revision time. This part was investigated only once after the course.
The questionnaire on students’ experience was designed based on the Normalization Process Theory (NPT) [18,19,20], including an overall dimension and four sub-dimensions. The overall dimension was concerned with students’ overall view of the new teaching model, and sub-dimensions included coherence; cognitive participation; collective action; reflexive monitoring . The specific items of the questionnaire of students’ experience are shown in Additional file 1: Table 2. This part was investigated before and after the course. Items of the overall dimension were calculated with their mean value, and items of four sub-dimensions were discretized into three categories (“agree”, “neutral”, and “disagree”).
The questionnaire on subjective teaching effect was designed on the Kirkpatrick model [21,22,23]. This part contained a total of 22 items from four dimensions which included reaction, learning, behavior, and results. After calculating the mean value of each dimension, it was summed and converted into a percentage system. This part was investigated only once at the end of the course.
This subjective teaching effect and the objective teaching effect mentioned above together constituted the teaching performance of this study.
We described all the data by using mean ± standard deviation (Mean ± SD) in quantitative data and the frequency and percentage (n (%)) in categorical data. We used Cronbach's α coefficient to evaluate the internal consistency of our questionnaire. Its internal consistency was considered to be good if Cronbach's α coefficient was over 0.7. McNemar's test (or Fisher's exact probability test) was used to compare the differences of each item of the four sub-dimensions in the parts of the NPT Scale on students’ experience, which were investigated respectively before and after the course. Finally, we regarded subjective and objective teaching effects as outcomes. According to Intraclass Correlation Coefficient (ICC) calculated by linear mixed model, we evaluated whether there was clustering between groups. For lower ICC of subjective teaching effect, multiple linear regression was used to explore its influencing factors. And for higher ICC of objective teaching effect, a linear mixed model was adopted to explore its influencing factors by setting groups as high level and individuals as low level. The Pearson or Spearman correlation coefficients between each influencing factor were 0.15–0.55, and the variance inflation factor (VIF) of each model was less than 5, so we regarded that there was no multicollinearity problem. All statistical analysis was performed in R (V4.1.2) and two-side P values < 0.05 was considered statistical significance.
Characteristics of participants
There were 201 students in our course and 172 valid questionnaires were collected, with an effective rate of 85.6%. Table 1 shows the characteristics of all participants. We found that 122 (70.9%) were females, 136 (79.1%) majored in preventive medicine, 55 (32.0%) currently participated in student union, 85 (49.4%) formerly participated in student union, and 24 (14.0%) were team leaders. The average level of team participation was 7.90 ± 1.6. The average revision time was 1.42 ± 1.2 h. Cronbach's α coefficients of all scales in our questionnaire were about 0.9.
The overall dimension of NPT Scales before and after the course were 5.79 ± 2.0 and 6.35 ± 2.2 respectively, and the McNemar's test showed statistically significant differences (P < 0.001). Students’ experience after the course was improved compared with that before the course. Figure 3 shows the investigations before and after the course of each item of the four sub-dimensions. The first sub-dimension was the understanding of the new teaching model. And the number of students who chose “agree” in both cases were more. The second sub-dimension was the participation of the new teaching model. The number of students who chose “agree” in both cases were more, but some students didn’t continue to support the new teaching model after the course. The third sub-dimension was the coordination between the new teaching model and other courses. We found that there was a polarization of students’ ability to adapt to the new teaching model, and the relationship between some students might be disturbed. The fourth sub-dimension was the feedback on the effect of the new teaching model. The number of students who chose “agree” or “neutral” in both cases were more, and students' recognition of the effect of the new teaching model was also polarized.
Teaching effects and influencing factors
The average score for subjective teaching effect was 67.53 ± 16.8, and the average score for objective teaching effect was 90.84 ± 4.9. And the regression results of influencing factors are shown in Table 2. The regression results of the subjective teaching effect showed that the average score of the teaching effect increased by 2.27 (P = 0.004) for each additional 1 h of revision time, and increased by 4.90 (P < 0.001) when the average score of the overall dimension in NPT Scale after the course increased by 1. With ICC of 83.5%, the result of linear mixed model showed that the objective teaching effect were 0.78 higher for preventive medicine than nursing (P = 0.045), 0.79 higher for students who formerly participated in student union than those who didn’t (P = 0.016), and 2.16 higher for team leaders than members (P < 0.001). We also found that the average score of objective teaching effect increased by 0.31 (P = 0.001) when the average score of team participation increased by 1.
We applied crowdsourcing to global health courses, developed and evaluated a new teaching model for global health education. We found that students' feedbacks towards this new teaching model improved after the course. However, there was polarization in students' adaptability to this model. As for teaching effect, students who majored in preventive medicine, formerly participated in student union, were team leaders, spent more time on revision, had higher team participation, and had positive feedback on the new teaching model tended to perform better.
We designed a new global health teaching model based on crowdsourcing, which made full use of its advantages in strong initiative, high flexibility and application-orientation. Previous studies explored different teaching approaches in global health education. For example, the teacher-taught model attached with a World Health Chart could improve the application ability of students in Sweden . In addition to lectures, students were encouraged to communicate with experienced doctors in UK . But these teaching practices were all teacher-centered, and students passively accepted knowledge with poor autonomy. Recently, McNabb et al. provided some strategies based on e-learning tools for global health education under the pandemic of the COVID-19, yet heavily relied on Internet technology . Crowdsourcing, has increasingly been used in teaching field . Geng et al. developed a Crowdsourcing Collaborative Learning Strategy (CCLS) by integrating crowdsourcing and personalized learning in nursing teaching. On the CCLS, students can propose their questions and answer others' questions to learn and obtain customized knowledge from teachers. Compared with the traditional teacher-centered lecture approach, this model had achieved better results . Crowdsourcing can also improve the traditional teaching model by collecting students' questions and targeted teaching . Similarly, Bow et al. developed a platform based on crowdsourcing where students can submit questions and answers and share them with others. Compared with the previous year, students' performance had improved . Generally, there is a lack of authoritative and reliable teaching materials for a new curriculum as global health. Through crowdsourcing, resources can be widely obtained from the students. This method also provides a new idea for curriculum development . Additionally, teachers can rate students more fairly through crowdsourcing . Crowdsourcing can be integrated into education in many forms. However, in these studies, crowdsourcing was mostly used in auxiliary teaching, such as developing platform, collecting learning materials and collecting students' questions. In this model, we fully integrated the course content into crowdsourcing and kept the traditional course-based teaching. The crowdsourcing learning was the main of our course to promote the teaching performance of global health.
Through the implementation of the crowdsourcing teaching model, students' experience improved but polarized. The NPT scale evaluated students' experience in detail through four sub-dimensions. The first (coherence) and fourth (reflexive monitoring) sub-dimensions were the understanding and the feedback on the effect of the new teaching model. The improvements in these domians may result from students' familiarity and recognization of the teaching model, through participation, which also showed in the second sub-dimension (cognitive participation). After the course, however, there was a relative decline in students’ support, in spite of an overall supportive rating. Some research had shown that compared with the traditional teaching model, students were more supportive (collective action) reflecting new teaching model , but some students were dissatisfied with the new teaching model . Through the third sub-dimension of the coordination between the new teaching model and other courses and with other students, we found that there was a polarization of students’ capability, and this model might even affect the relationship between classmates. Therefore, it’s considered that the new teaching model was different from the current other courses, and some students cannot adapt to it well, so they didn’t continue to support this model . Studies have shown that students' differences are the main determinants of education effect, which will produce different responses to teaching [32, 33]. Therefore, there was polarization in students' adaptability, leading to conflicts in groups and unpleasant feelings for students.
Furthermore, we found that the more revision time dedicated to the course, the better the subjective teaching effect. It is considered that the more revision time, the richer the knowledge will acquire, and the better the feedback will be in the self-reported/subjective teaching effect. Additionally, students with better experience tend to report higher subjective evaluation of the teaching effect. In the objective teaching effect, we found that the students majoring in prevention medicine, formerly participated in student union, were team leaders, and had high group participation showed better outcomes. There might be two reasons for the better performance of students majoring in preventive medicine. One is that students majoring in prevention medicine were seniors than students majoring in nursing. The comprehensive ability of students in higher grades is better than in lower grades [34, 35]. The other is that prevention medicine major is more relevant to global health . Students who formerly participated in student union performed better, may be due to their abilities of leadership. They were more likely to be the team leaders or play a major role in the teams, and thus were more likely to gain extra awarded points in the scoring criteria. Team collaboration in the new teaching model also played an important role. Prior studies showed that students prefer team collaboration, turned to achieve better outcomes in innovative teaching models . Good team collaboration could allow excellent individual to drive the whole team to study together and to achieve a better teaching effect.
Strengths and limitations
In this study, crowdsourcing was applied to the global health teaching model design, combining the advantages of crowdsourcing and the characteristics of global health education. We evaluated students’ experience through NPT from the view of implementation science, and examined the teaching effect by the Kirkpatrick model. These methods were widely used in education [37,38,39,40,41,42], which generate standardized and valid evaluation on the application of the new teaching model to provide new evidence for global health education.
However, there were still some limitations in this study. Firstly, this study is a first attempt of applying crowdsourcing to global health education, the design of the teaching model only regarded crowdsourcing as the guiding ideology and utilizied some crowdsourcing elements. A feasible teaching model that is more consistent with the form of crowdsourcing needs to be further tried. Secondly, due to ethical consideration, we didn’t set a control group. Instead, the teaching effects and influencing factors were explored through regression models and self-comparison of participants before and after the course. Thirdly, the period of this course was just ten weeks, which was too short to evaluate the teaching model for a longer term.
We applied crowdsourcing to global health courses for undergraduate students, developed and evaluated a new teaching model for global health education. It is found that most students supported this new teaching model and achieved good results. However, the new teaching model is quite different from traditional ones and some students may not have difficulties to adapt to it. To promote the application of crowdsourcing in global health education, it is necessary to streamline the teaching model and highlight the core of global health and crowdsourcing, with better connection of the course content with the teaching model, and improved coordination with other courses. More attention should be paid to the guidance and supplement learning content, encourage students majoring in preventive medicine or formerly participated in student union to play a core role in teams.
Availability of data and materials
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
Crowdsourcing collaborative learning strategy
Chinese Consortium of Universities for Global Health
Intraclass correlation coefficient
Normalization process theory
Sun Yat-sen University
Health IOMU. The US Commitment to Global Health: Recommendations for the New Administration. Washington (DC): National Academies Press (US); 2009.
Koplan JP, Bond TC, Merson MH, Reddy KS, Rodriguez MH, Sewankambo NK, Wasserheit JN. Towards a common definition of global health. Lancet. 2009;373(9679):1993–5.
Liu P, Guo Y, Qian X, Tang S, Li Z, Chen L. China’s distinctive engagement in global health. Lancet. 2014;384(9945):793–804.
Fu J, Jiang C, Wang J, Xing Y. To establish a talent pool for global health in China: from political will to action. Glob Health Action. 2018;11(1):1489603.
Guo Y. Development of global health research in China. J Glob Health. 2017;7(2):20102.
Kwete X, Tang K, Cheng F, Chen Y, Hao YT, Mao Z, Ren R, Wang Y, Wang Y, Wu C, et al. Research capacity of global health institutions in China: a gap analysis focusing on their collaboration with other low-income and middle-income countries. BMJ Glob Health. 2021;6(7):555.
Bozorgmehr K, Menzel-Severing J, Schubert K, Tinnemann P. Global Health Education: a cross-sectional study among German medical students to identify needs, deficits and potential benefits (Part 2 of 2: knowledge gaps and potential benefits). BMC Med Educ. 2010;10:68.
Kickbusch I, Franz C, Holzscheiter A, Hunger I, Jahn A, Koehler C, Razum O, Schmidt J. Germany’s expanding role in global health. Lancet. 2017;390(10097):898–912.
Myers KR, Fredrick NB. Team investment and longitudinal relationships: an innovative global health education model. Acad Med. 2017;92(12):1700–3.
Havemann M, Boesner S. Global Health as “umbrella term” - a qualitative study among Global Health teachers in German medical education. Globalization Health. 2018;2:14.
Hew KF, Lo CK. Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ. 2018;18(1):38.
Kiessling A. Participatory learning: a Swedish perspective. Heart. 2004;90(1):113–6.
Geng Y, Huang P, Huang Y. Crowdsourcing in nursing education: a possibility of creating a personalized online learning environment for student nurses in the post-COVID Era. Sustainability-Basel. 2021;13(6):74.
Wazny K. “Crowdsourcing” ten years: a review. J Glob Health. 2017;7(2):20602.
Penciner R. Crowdsourcing: an instructional method at an emergency medicine continuing education course. Can J Emerg Med. 2015;17(4):433–6.
Wazny K. Applications of crowdsourcing in health: an overview. J Glob Health. 2018;8(1):10502.
Day S, Li C, Hlatshwako TG, Abu-Hijleh F, Han L, Deitelzweig C, Bayus B, Ramaswamy R, Tang W, Tucker JD. Assessment of a crowdsourcing open call for approaches to university community engagement and strategic planning during COVID-19. JAMA Netw Open. 2021;4(5):e2110090.
Finch TL, Girling M, May CR, Mair FS, Murray E, Treweek S, Mccoll E, Steen IN, Cook C, Vernazza CR, et al. Improving the normalization of complex interventions: part 2-validation of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol. 2018;2:18.
Rapley T, Girling M, Mair FS, Murray E, Treweek S, Mccoll E, Steen IN, May CR, Finch TL. Improving the normalization of complex interventions: part 1-development of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol. 2018;5:18.
Lewis NV, Dowrick A, Sohal A, Feder G, Griffiths C. Implementation of the Identification and Referral to Improve Safety programme for patients with experience of domestic violence and abuse: a theory-based mixed-method process evaluation. Health Soc Care Commun. 2019;27(4):E298–312.
Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE guide no. 67. Med Teach. 2012;34(5):e288–99.
Yoon HB, Shin JS, Bouphavanh K, Kang YM. Evaluation of a continuing professional development training program for physicians and physician assistants in hospitals in Laos based on the Kirkpatrick model. J Educ Eval Health Prof. 2016;13:21.
Johnston S, Coyer FM, Nash R. Kirkpatrick’s evaluation of simulation and debriefing in health care education: a systematic review. J Nurs Educ. 2018;57(7):393–8.
Bateman C, Baker T, Hoornenborg E, Ericsson U. Bringing global issues to medical teaching. Lancet. 2001;358(9292):1539–42.
Mcnabb S, Magowe M, Shaw N, Berrian AM, Wilkes M, Shaikh A, Gachuno O, Perrone LA, Murray BL, Berman E, et al. Delivering modern global health learning requires new obligations and approaches. Ann Glob Health. 2021;87(1):68.
Bow HC, Dattilo JR, Jonas AM, Lehmann CU. A crowdsourcing model for creating preclinical medical education study tools. Acad Med. 2013;88(6):766–70.
Kpokiri EE, John R, Wu D, Fongwen N, Budak JZ, Chang CC, Ong JJ, Tucker JD. Crowdsourcing to develop open-access learning resources on antimicrobial resistance. BMC Infect Dis. 2021;21(1):914.
Chan C, Holosko MJ. An overview of the use of mechanical turk in behavioral sciences: implications for social work. Res Social Work Prac. 2016;26(4):441–8.
Liu H, Jing D, Shafi K. A study on innovative methods used in teaching finance course. Proceedings of the Eighth International Symposium-Education Innovation and Historical Culture Research. 2015:233–241
Finkelstein ND, Pollock SJ. Replicating and understanding successful innovations: Implementing tutorials in introductory physics. Phys Rev Spec Top Ph. 2005;1(1):666.
Ramage C. Negotiating multiple roles: link teachers in clinical nursing practice. J Adv Nurs. 2004;45(3):287–96.
Basch CE. Healthier students are better learners: a missing link in school reforms to close the achievement gap. J Sch Health. 2011;81(10):593–8.
Basch CE. Healthier students are better learners: high-quality, strategically planned, and effectively coordinated school health programs must be a fundamental mission of schools to help close the achievement gap. J Sch Health. 2011;81(10):650–62.
Stillman PL, Wang Y, Ouyang Q, Zhang S, Yang Y, Sawyer WD. Teaching and assessing clinical skills: a competency-based programme in China. Med Educ. 1997;31(1):33–40.
Xia T, Kang M, Chen M, Ouyang J, Hu F. Design training and creativity: students develop stronger divergent but not convergent thinking. Front Psychol. 2021;12:695002.
Wang J, Xu X, Shi X, Zhang J, Gao F, Li J, Jia S, Xu J, Zhang J, Peng L, et al. Literature-based learning and experimental design model in molecular biology teaching for medical students at Tongji University. Biochem Mol Biol Educ. 2021;49(2):189–97.
Odonnell A, Kaner E. Are brief alcohol interventions adequately embedded in UK primary care? a qualitative study utilising normalisation process theory. Int J Environ Res Public Health. 2017;14(4):2224.
Maddineshat M, Hashemi M, Besharati R, Gholami S, Ghavidel F. The effectiveness of clinical teaching of mental health courses in nursing using clinical supervision and Kirkpatrick’s model. Electron Phys. 2018;10(1):6265–72.
Chambers S, Boydell N, Ford A, Eadie D. Learning from the implementation of Universal Free School Meals in Scotland using normalisation process theory: lessons for policymakers to engage multiple stakeholders. Food Policy. 2020;95:101936.
Cook C, Finch T, Sharma S, Girling M, Rapley T, Vernazza CR. Developing oral health risk assessment as routine practice during early stages of clinical careers: a cross-sectional study of dental students using the NoMAD questionnaire. Eur J Dent Educ. 2020;24(1):169–76.
Keenan J, Poland F, Manthorpe J, Hart C, Moniz-Cook E. Implementing e-learning and e-tools for care home staff supporting residents with dementia and challenging behaviour: a process evaluation of the ResCare study using normalisation process theory. Dementia (London). 2020;19(5):1604–20.
Pinilla S, Cantisani A, Kloppel S, Strik W, Nissen C, Huwendiek S. Introducing a psychiatry clerkship curriculum based on entrustable professional activities: an explorative pilot study. Acad Psychiatry. 2021;45(3):354–9.
We thank all the students and teachers who participated in this course “Introduction to Global Health”. Professor Yuantao Hao gratefully acknowledges the support of K.C.Wong Education Foundation.
This work was supported by Higher Education Teaching Reform Project in Guangdong Province (Guangdong Education High Letter  No.20).
Ethics approval and consent to participate
The Ethics Committee of the School of Public Health, Sun Yat-sen University has approved our study (#SYSUNO.085).
Consent for publication
The authors declare that they have no competing interests.
About this article
Cite this article
Cai, H., Zheng, H., Li, J. et al. Implementation and evaluation of crowdsourcing in global health education. glob health res policy 7, 50 (2022). https://doi.org/10.1186/s41256-022-00279-7
- Global health
- Teaching model
- Effect evaluation