Research Scientist, Google Research.
Creating a novel benchmark dataset of 25,000 question-answer pairs to rigorously evaluate 20+ LLMs on African healthcare, spanning 32 clinical specialties, contributed by 1k+ African clinicians across 15 countries
Training data bias: LLMs are trained on vast amounts of web data largely sourced from developed countries. Consequently, models are exposed to healthcare problems that are prevalent in these geographies. However, the burden of disease varies significantly in other parts of the world.
In Sub-Saharan Africa communicable diseases, maternal, and neonatal disease dominate while non communicable diseases are prevalent in the US. It is therefore important to investigate if LLMs trained on datasets sourced predominantly from data rich regions exhibit bias when applied in LMICS. see more
AfriMed-QA creates a unique, inclusive, pan-African, multi-region, multi-institution dataset of 25,000 Africa-focused question-answer pairs to comprehensively evaluate LLM capabilities across diverse facets of African healthcare. It seeks to spotlight where these models excel, identify potential pitfalls, and explore scenarios where they could prove pivotal or potentially risky.
More than just a benchmarking exercise, this project stands as a guide for African academic, clinical, biomedical, and research communities. see more
Expert Generated MCQ’s with Answers
Crowdsourced MCQ’s and SAQ with Answers and explanations
Consumer Health Queries with Clinician Answers
Research Scientist, Google Research.
Co-Founder, Sisonkebiotik
CEO, Intron
Kenya Lead, FAMSA Liaison
University of Cape Coast, GhanaNLP
CS PhD Candidate, Georgia Institute of Technology
The dataset creation will leverage professional medical questions– multiple choice
questions (MCQ) and short answer question (SAQ)– crowdsourced from exam prep
material popularly used by medical schools across Africa as well as consumer health
questions, similar to questions typed in a google search query. see more
This work is licensed under a creative commons Attribution-NonCommercial -ShareAlike 4.0 International License.
Access the data, benchmarking code, and results on github.
Want to use AfriMed-QA for commercial purposes? Contact us to obtain a commercial license.
Learn to benchmark open or closed LLMs on Africa dataset
The dataset’s geographical and clinical diversity facilitates robust contextual evaluation of LLMs in African healthcare and provides a sufficiently large corpus to finetune LLMs to mitigate biases discovered.
It rigorously documents evidence in the context of African healthcare highlighting use cases or clinical specialties where LLMs shine as well as situations where they fall short or have a high potential for harm.
The project will be a timely and invaluable resource guiding the African academic, clinical, biomedical, and research communities on the utility of LLMs in African see more
healthcare at a scale that not only enables robust and rigorous LLM evaluation but provides a sufficiently large dataset to mitigate biases – by finetuning these LLMs, adequately exposing model weights to African healthcare data in context. Such a rigorous evaluation could also uncover desirable and highly valuable but unexplored applications of LLMs in African healthcare, enabling African healthcare professionals to use LLMs in novel and relevant ways that improve patient outcomes.
The initial success of the project will stimulate broader research, eliciting more interesting questions on LLM behavior resulting in deeper investigation into specific clinical specialties of interest, demographic or sociocultural issues, as well as possible expansion of the dataset methodology to other domains beyond healthcare.
We are seeking clinicians, medical students, data scientists, data engineers, MLOPs engineers, data science students and enthusiasts. Your contributions make a difference in improving LLMs for use in African healthcare to improve patient outcomes: By contributing, you will be helping to reduce the burden on healthcare providers, allowing them to focus on delivering better patient care and improving health outcomes and strengthening healthcare in Africa. Your input will shape and influence the direction of our project and help us create a more effective and user-friendly platform.
