Skip to main navigation menu Skip to main content Skip to site footer
##common.pageHeaderLogo.altText##

Research Article

Vol. 21 No. 1 (2023)

Evolving a conceptual framework and developing a new questionnaire for usability evaluation of blended learning programs in health professions education

DOI
https://doi.org/10.26443/mjm.v21i1.961
Submitted
February 11, 2022
Published
2022-12-06

Abstract

Background: Blended learning programs (BLPs) have been widely adopted across health professions education (HPE). To bolster their impact on learning outcomes, the usability of BLPs should be rigorously evaluated. However, there is a lack of reliable and validated tools to appraise this dimension of BLPs within HPE. The purpose of this investigation was to evolve a conceptual framework for usability evaluation in order to initially develop the Blended Learning Usability Evaluation – Questionnaire (BLUE-Q).

Methods: After the completion of a scoping review, we conducted a qualitative descriptive study with seven purposefully selected international experts in usability and learning program evaluation. Individual interviews were conducted via videoconferencing, transcribed verbatim, and analyzed through thematic analysis.

Results: Three themes were identified: (1) Consolidation of the multifaceted ISO definition of usability in BLPs within HPE; (2) Different facets of usability can assess different aspects of BLPs; (3) Quantitative and qualitative data are needed to assess the multifaceted nature of usability. The first theme adds nuance to a previously established HPE-focused usability framework, and introduces two new dimensions: ‘pedagogical usability’ and ‘learner motivation.’ The latter two provide guidance on structuring BLP evaluations within HPE. From this followed the development of the BLUE-Q, a new questionnaire that includes 55 Likert scale items and 6 open-ended questions.

Conclusions: Usability is an important dimension of BLPs and must be examined to improve the quality of these interventions in HPE. As such, we developed a new questionnaire, solidly grounded in theory and the expertise of international scholars, currently under validation.

References

  1. Garrison DR, Kanuka H. Blended learning: Uncovering its transformative potential in higher education. The internet and higher education. 2004;7(2):95-105. DOI:10.1016/j.iheduc.2004.02.001
  2. Garrison DR, Vaughan ND. Blended learning in higher education: Framework, principles, and guidelines. John Wiley & Sons; 2008. DOI:10.1002/9781118269558
  3. Williams C. Learning on-line: A review of recent literature in a rapidly expanding field. Journal of further and Higher Education. 2002 Aug 1;26(3):263-72. https://doi.org/10.1080/03098770220149620
  4. Watson J. Blended learning: the convergence of online and face- to-face education. North American Council for Online Learning: Promising Practices in Online Learning; 2008. Retrieved from: https://files.eric.ed.gov/fulltext/ED509636.pdf
  5. Hrastinski S. What Do We Mean by Blended Learning? TechTrends. 2019; 63, 564–569. https://doi.org/10.1007/s11528-019-00375-5
  6. Abrosimova G, Kondrateva I, Voronina E, Plotnikova N. Blended Learning in University Education. Humanities & Social Science Reviews. 2019; 7(6), 6-10. https://doi.org/10.18510/hssr.2019.762
  7. Cronje J. Towards a new definition of blended learning. Electronic journal of e-Learning. 2020 Feb 1;18(2):pp114-121. DOI: 10.34190/EJEL.20.18.2.001
  8. Armellini A, Rodriguez BC. Active Blended Learning: Definition, Literature Review, and a Framework for Implementation. Cases on Active Blended Learning in Higher Education. 2021:1-22. DOI: 10.4018/978-1-7998-7856-8.ch001
  9. Leidl DM, Ritchie L, Moslemi N. Blended learning in undergraduate nursing education–A scoping review. Nurse Education Today. 2020 Mar 1;86:104318. DOI: 10.1016/j.nedt.2019.104318
  10. Bersin J. The blended learning book: Best practices, proven methodologies, and lessons learned. John Wiley & Sons; 2004 Sep 24.
  11. Graham CR, Woodfield W, Harrison JB. A framework for institutional adoption and implementation of blended learning in higher education. The internet and higher education. 2013 Jul 1;18:4-14. https://doi.org/10.1016/j.iheduc.2012.09.003
  12. Zainuddin Z, Haruna H, Li X, Zhang Y, Chu SK. A systematic review of flipped classroom empirical evidence from different fields: what are the gaps and future trends?. On the Horizon. 2019 Jun 3. DOI: 10.1108/OTH-09-2018-0027
  13. Awidi IT, Paynter M. The impact of a flipped classroom approach on student learning experience. Computers & Education. 2019 Jan 1;128:269-83. DOI: 10.1016/j.compedu.2018.09.013
  14. Rafiola R, Setyosari P, Radjah C, Ramli M. The Effect of Learning Motivation, Self-Efficacy, and Blended Learning on Students’ Achievement in The Industrial Revolution 4.0. International Journal of Emerging Technologies in Learning (iJET). 2020 Apr 24;15(8):71-82. DOI: 10.3991/ijet.v15i08.12525
  15. Vallée A, Blacher J, Cariou A, Sorbets E. Blended learning compared to traditional learning in medical education: systematic review and meta-analysis. Journal of medical Internet research. 2020;22(8):e16504. DOI: 10.2196/16504
  16. Khalil MK, Abdel Meguid EM, Elkhider IA. Teaching of anatomical sciences: A blended learning approach. Clinical Anatomy. 2018 Apr;31(3):323-9. DOI: 10.1002/ca.23052
  17. Oguguo BC, Nannim FA, Agah JJ, Ugwuanyi CS, Ene CU, Nzeadibe AC. Effect of learning management system on Student’s performance in educational measurement and evaluation. Education and Information Technologies. 2021 Mar;26(2):1471-83. https://doi.org/10.1007/s10639-020-10318-w
  18. Pereira JA, Pleguezuelos E, Merí A, Molina‐Ros A, Molina‐Tomás MC, Masdeu C. Effectiveness of using blended learning strategies for teaching and learning human anatomy. Medical education. 2007 Feb;41(2):189-95. DOI:10.1111/j.1365-2929.2006.02672.x
  19. Fernandes RA, de Oliveira Lima JT, da Silva BH, Sales MJ, de Orange FA. Development, implementation and evaluation of a management specialization course in oncology using blended learning. BMC medical education. 2020 Dec 1;20(1):37. https://doi.org/10.1186/s12909-020-1957-4
  20. Sáiz-Manzanares MC, Escolar-Llamazares MC, Arnaiz González Á. Effectiveness of blended learning in nursing education. International journal of environmental research and public health. 2020 Jan;17(5):1589. DOI: 10.3390/ijerph17051589
  21. Muresanu D, Buzoianu AD. Moving Forward with Medical Education in Times of a Pandemic: Universities in Romania Double Down on Virtual and Blended Learning. Journal of Medicine and Life. 2020 Oct;13(4):439. DOI: 10.25122/jml-2020-1008
  22. Theoret C, Ming X. Our education, our concerns: The impact on medical student education of COVID‐19. Medical education. 2020 Jul;54(7):591-2. DOI: 10.1111/medu.14181
  23. Torda AJ, Velan G, Perkovic V. The impact of the COVID-19 pandemic on medical education. Med J Aust. 2020 Aug 1;14(1). DOI: 10.5694/mja2.50705
  24. Steelcase. COVID-19 Accelerates Blended Learning; The recent crisis is showcasing the upside of a blended learning approach to education. N.D. Retrieved from: https://www.steelcase.com/research/articles/topics/education/covid-19-accelerates-blended-learning/
  25. Bordoloi R, Das P, Das K. Perception towards online/blended learning at the time of Covid-19 pandemic: an academic analytics in the Indian context. Asian Association of Open Universities Journal. 2021 Feb 16. DOI: 10.1108/AAOUJ-09-2020-0079
  26. Adel A, Dayan J. Towards an intelligent blended system of learning activities model for New Zealand institutions: an investigative approach. Humanities and Social Sciences Communications. 2021 Mar 16;8(1):1-4. https://doi.org/10.1057/s41599-020-00696-4
  27. Kim JW, Myung SJ, Yoon HB, Moon SH, Ryu H, Yim JJ. How medical education survives and evolves during COVID-19: Our experience and future direction. PloS one. 2020 Dec 18;15(12):e0243958. https://doi.org/10.1371/journal.pone.0243958
  28. Lala SG, George AZ, Wooldridge D, Wissing G, Naidoo S, Giovanelli A, et al. A blended learning and teaching model to improve bedside undergraduate paediatric clinical training during and beyond the COVID-19 pandemic. African Journal of Health Professions Education. 2021 Mar 1;13(1):18-22. https://doi.org/10.7196/AJHPE.2021.v13i1.1447
  29. Cheng SO, Liu A. Using online medical education beyond the COVID-19 pandemic–A commentary on “The coronavirus (COVID-19) pandemic: Adaptations in medical education “. International Journal of Surgery (London, England). 2020 Nov 16. DOI: 10.1016/j.ijsu.2020.05.022
  30. Metz AJ. Why conduct a program evaluation? Five reasons why evaluation can help an out-of-school time program. Research-to-Results Brief. Chilid TRENDS. 2007 Oct;4. Retrieved from: https://www.childtrends.org/wp-content/uploads/2013/04/child_trends-2007_10_01_rb_whyprogeval.pdf
  31. Cleveland-Innes M, Wilton D. CHAPTER 8: Evaluating Successful Blended Learning. Guide to Blended Learning . Athabasca University. 2018. Retrieved from: https://openbooks.col.org/blendedlearning/chapter/chapter-8-evaluating-successful-blended-learning/
  32. Arora A. Usability in blended learning programs within health professions education: A scoping review. Available from Dissertations & Theses at McGill University; ProQuest Dissertations & Theses Global; 2019. Retrieved from https://proxy.library.mcgill.ca/login?url=https://www.proquest.com/dissertations-theses/usability-blended-learning-programs-within-health/docview/2516891062/se-2
  33. Arora AK, Rodriguez C, Carver T, Teper MH, Rojas-Rozo L, Schuster T. Evaluating usability in blended learning programs within health professions education: a scoping review. Medical science educator. 2021 Jun;31(3):1213-46. Retrieved from: https://link.springer.com/article/10.1007/s40670-021-01295-x
  34. Bowyer J, Chambers L. Evaluating blended learning: Bringing the elements together. Research Matters: A Cambridge Assessment Publication. 2017;23:17-26. Retrieved from: https://www.cambridgeassessment.org.uk/Images/375446-evaluating-blended-learning-bringing-the-elements-together.pdf
  35. Singh H. Building effective blended learning programs. InChallenges and Opportunities for the Global Implementation of E-Learning Frameworks 2021 (pp. 15-23). IGI Global. DOI: 10.4018/978-1-7998-7607-6
  36. Mohammed Abdel-Haq E. The Blended Learning Model: Does It Work?. Sohag University International Journal of Educational Research. 2021 Jan 1;3(3):29-40. DOI: 10.21608/SUIJER.2021.122458
  37. International Organization for Standardization. ISO 9241-11:2018(en) Ergonomics of human system interaction — Part 11: Usability: Definitions and concepts. 2018. Retrieved from https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
  38. Ifinedo P, Pyke J, Anwar A. Business undergraduates’ perceived use outcomes of Moodle in a blended learning environment: The roles of usability factors and external support. Telematics and Informatics. 2018 Apr 1;35(1):93-102. 10.1016/j.tele.2017.10.001
  39. Ventayen RJ, Estira KL, De Guzman MJ, Cabaluna CM, Espinosa NN. Usability evaluation of google classroom: Basis for the adaptation of gsuite e-learning platform. Asia Pacific Journal of Education, Arts and Sciences. 2018 Jan;5(1):47-51. Retrieved from: http://apjeas.apjmr.com/wp-content/uploads/2017/12/APJEAS-2018.5.1.05.pdf
  40. Nakamura WT, de Oliveira EH T, Conte T. Usability and user experience evaluation of learning management systems-a system- atic mapping study. In International Conference on Enterprise Information Systems. 2017;2:97–108. DOI: 10.5220/0006363100970108
  41. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. International journal of social research methodology. 2005 Feb 1;8(1):19-32. https://doi.org/10.1080/1364557032000119616
  42. Green J, Thorogood N. Qualitative methods for health research. sage; 2018 Feb 26.
  43. Marshall MN. Sampling for qualitative research. Family practice. 1996 Jan 1;13(6):522-6. DOI: 10.1093/fampra/13.6.522
  44. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. https://doi.org/10.1191/1478088706qp063oa
  45. Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, Burroughs H, Jinks C. Saturation in qualitative research: exploring its conceptualization and operationalization. Quality & quantity. 2018 Jul;52(4):1893-907. https://doi.org/10.1007/s11135-017-0574-8
  46. Precel K, Eshet-Alkalai Y, Alberton Y. Pedagogical and design aspects of a blended learning course. International Review of Research in Open and Distributed Learning. 2009;10(2). DOI: https://doi.org/10.19173/irrodl.v10i2.618
  47. Sandars J. The importance of usability testing to allow e-learning to reach its potential for medical education. Education for Primary Care. 2010 Jan 1;21(1):6-8. DOI: 10.1080/14739879.2010.11493869
  48. Zhang H, Schuster T. Questionnaire instrument development in primary health care research: A plea for the use of Bayesian inference. Canadian Family Physician. 2018 Sep 1;64(9):699-700. Retrieved from: ncbi.nlm.nih.gov/pmc/articles/PMC6135135/pdf/0640699.pdf
  49. Zhang H, Schuster T. A methodological review protocol of the use of Bayesian factor analysis in primary care research. Systematic Reviews. 2021 Dec;10(1):1-5. https://doi.org/10.1186/s13643-020-01565-6
  50. Haynes SN, Richard D, Kubany ES. Content validity in psychological assessment: A functional approach to concepts and methods. Psychological assessment. 1995 Sep;7(3):238. https://doi.org/10.1037/1040-3590.7.3.238

Downloads

Download data is not yet available.