Why Educators Cannot Only Rely on Established Review Organizations for Product Research Validation
Co-Author: Apoorva Shivaram, Northwestern University
Education leadership is tightening its flexibility by using curriculum products that do not have evidence of effectiveness. Yet, millions of children nationwide still use educational products without evidence of effectiveness. While some states pass laws and create vetted lists (summary of such laws as of November 2022), many district and school leaders must review the research to find evidence-based products. Unfortunately, research is often difficult to access and understand. Government and non-profit organizations have stepped in to help. Still, educators cannot rely solely on these reviews as only a tiny fraction of research studies are reviewed by these organizations.
What do we mean by ‘evidence-based’?
‘Evidence-based’ refers to the product used by students and teachers in classrooms and schools. The companies that develop these products create a description of ‘fidelity of implementation,’ or how the product should work, and then follow groups of students or schools to provide evidence that the program is effective. These reports essentially show the public that the product can produce the promises of growth the company is claiming.
Why is reviewing research necessary?
Conducting research is complicated, with each step of the process involving several decisions made by the researchers and their teams. These decisions, individually or taken together, tremendously impact the study’s outcome and resulting implications. Therefore, the research must be reviewed by appropriate expert researchers (“peers”).
Who reviews the research?
In the United States, there are three main organizations that review research that assists educators with selecting practical educational tools: What Works Clearinghouse (WWC), run by the federal Department of Education through the Institute for Education Science, Evidence for ESSA, run by John Hopkins’ Center for Education Reform, and the National Center for Intensive Intervention (NCII) at the American Institutes for Research. These organizations review research, determine whether they meet specific rigorous standards, and summarize the findings to provide clear and authoritative information about educational tools that improve student success.
Confused about all the different organizations that review products and research? This glossary can help!
What research has been reviewed?
Evidence for ESSA and NCII review any submitted research studies and do not seek out research to review. Alternatively, WWC conducts a broad literature review and database search to find related studies, scores and ranks the interventions, and reviews the related research (WWC’s description of this process). Because of these different approaches, these websites may have overlapping and contradictory information or have different products reviewed compared to the others.
How many research studies have been reviewed?
All three organizations follow strict standards to examine the validity of education-based study methodologies and findings for causal inferences. They also emphasize the importance of utilizing interventions in the classroom that are backed by high-quality research. Yet, only a tiny percentage of the conducted studies are reviewed. For instance, over the last five years, nearly 20,982 studies on literacy or reading were eligible to be reviewed in the last five years on the education database ERIC (Education Resource Information Center for Publications). WWC has reviewed 80 of those 20,982 studies (0.38%), while Evidence for ESSA has reviewed only 16 of those 20,982 studies (0.06%). Similarly, NCII has only reviewed 82 studies related to reading over the past three decades.
Why do these organizations review only such a small percentage of research?
Some strict standards and criteria need to be met to be considered for review by any of these organizations. For instance, Evidence for ESSA screens studies based on when the study was conducted, a specific set of types of research design, length of study duration, sample size requirements, and outcome measures, amongst other factors. Similarly, WWC screens studies based on, amongst other factors, whether it meets all requirements related to availability, completeness, timeframe, and language used. It also uses a specific set of types of research design, population, interventions, and outcome measures. These relatively strict criteria screen out many studies, so these two organizations review very few studies.
What has been done so far to fill the large literature review gap?
On an individual level: In April 2023, the Institute of Education Service partnered with WestEd Research to produce guidance on building evidence to guide EdTech adoption in schools by providing background knowledge for each of the four tiers of evidence and a case study for each. In addition, WestEd’s quick-reference guide from August 2021 describes how to select an evidence-based intervention by evaluating the study source, design, findings, and its relevance to a given context. These guides provide insight into evaluating a single study. In conjunction with these guides, the Evidence Evaluation Framework developed by LXD Research guides evaluates educational products’ and interventions’ relevance and efficacy to ensure they fit the learning context.
On a state level: School, district, and state leaders have started to create curricular and policy changes that ensure only evidence-based and evidence-informed curricula are implemented in educational settings. For instance, Utah’s legislation to improve the state’s early literacy outcomes is a broad sweeping goal that supports evidence-based practices in the science of reading instruction. As summarized by Dr. Jennifer Throndsen, Director of Teaching and Learning at Utah State Board of Education, the bill that was passed into law in March 2022 and that took effect in May 2022 is one of the most significant state-wide investments in teacher education around evidence-informed practice and using an evidence-based curriculum. As defined in the legislation, ‘evidence-based’ refers to a strategy that, amongst other criteria, produces an effect size of at least 0.4, includes random or quasi-random sampling, a curriculum that aligns with the Utah State Board of Education’s Science of Reading Criteria, and the participants of the study have to be in at least two grades between K-3. These criteria, although stringent, give solution providers guidance on what to aim for in terms of research design and outcomes.
Another example is from the Arizona State Department of Education. Arizona’s Move on When Reading (MOWR) was first enacted in 2010 and implemented in 2013 to identify children struggling with reading goals and provide targeted and evidence-based interventions that enable them to read at or above their grade level by the end of 3rd grade. Over the past decade, Arizona’s Department of Education has created a vetted list of interventions and supplemental program materials with linked research as evaluated by the Evidence for ESSA. This resource could be a starting point for educators and legislators to utilize software, programs, or tools that have been screened in their classrooms.
So, where does this leave us?
As educational professionals and stakeholders in the current education system, we must implement evidence-based products and evidence-informed curricula in our classrooms. Although the What Works Clearinghouse, Evidence for ESSA, and NCII organizations evaluate existing research using strict criteria, several studies conducted with educational products don’t pass the screening criteria. It is, therefore, essential to provide educators with the tools necessary to evaluate any product before it is used in the classroom to provide high-quality education to students. After all, our students are the future of the world and our greatest asset.
About LXD Research
LXD Research is an independent evaluation, research, and consulting division within Charles River Media Group, LLC focusing on educational technology. We design rigorous research studies, multifaceted data analytic reporting, and dynamic content to disseminate insights. Visit www.LXDResearch.com.
Recommended citation: Shivaram, A., & Schechter, R. (2023, June 2). Why educators cannot only rely on established review organizations for product research validation. LXD Research. https://lxdresearch.com/product-review-orgs-limitations