A new set of non-regulatory guidance came from the US Department of Education on September 28, 2023, about using evidence to purchase educational products. The guidance emphasizes What Works Clearinghouse (WWC) criteria as the highest level of quality and rigor. There are over 10,000 products used in a given school district. Educators need more info about the research on the products they already have. Other designs provide meaningful information about product performance and expectations. How are educators to know how these other studies measure up? LXD Research has created ESSA Alignment Seals to meet this very need.
LXD Research recognizes that when you ask 98 researchers to comment on a study, you will get 98 different responses. Our team of experts reviews studies that do not meet WWC criteria and provides documentation for educators to understand the research questions, how those questions were investigated, and the findings. Most importantly, we also share our experts’ questions about the study’s design and execution. We explicitly point out some characteristics the study would have needed to have met WWC criteria for the educator to contextualize and the company to know as they plan their future studies.
Here is how LXD Research’s ESSA-Aligned Seals address the most common factors that cause product studies to fail to meet WWC criteria.
More Flexible Evaluation Criteria
The WWC requires studies to have a control group, such as randomized controlled trials (RCTs) or quasi-experimental designs, with sufficient sample sizes and appropriate data analysis techniques. Most educational research uses a cohort design or a before-and-after data story (no control group). Finding districts willing to delay using a new product to meet the research design criteria can be difficult. Cohort and correlational designs are not eligible for review by WWC or the Evidence for ESSA website. Studies without a control group may qualify for LXD Research ESSA Level 3, Promising, if the study shows meaningful student outcomes growth.
Improved Research Accessibility
The WWC requires research studies to be housed in the ERIC database to be reviewed. While the database accepts online submissions, the authors must submit the paper. Publishers are not allowed to submit reports on behalf of others. LXD Research studies that meet WWC criteria are all uploaded into ERIC (review all of Dr. Schechter’s submissions to ERIC). LXD Research will review studies that are not currently in ERIC.
Leveraging Time and Resources
Conducting high-quality research studies to evaluate educational interventions is time-consuming and resource-intensive. Many educational organizations and developers may not have the capacity or funding to undertake long or large evaluations. LXD Research accepts small studies for review and allows for combining sample sizes across products for an overall ESSA level for the product. LXD Research also considers whether the assessment was standardized or embedded in the product. While LXD Research limits ESSA Level 1, Strong ratings for standardized measures, ESSA Level 2, Moderate ratings can use publisher-created measures provided they are given through the computer and have research evidence for validity.
Evolving Field of Research
Education is a dynamic field with new products and interventions continually being developed. LXD Research allows companies to review more of their research so experts can evaluate these new offerings, leading to a wider selection of listed products than if a district were to use WWC alone. WWC does not review products upon request, making it nearly impossible for a company to have their product reviewed even if they meet all the criteria. Additionally, publicly available data can help companies provide evidence for the impact of their products. LXD Research ESSA ratings allow for public data for Levels 2 or 3.
Focus on Evaluating More Than Just Study Design
The WWC only includes interventions with rigorous study design and then estimates the product’s effectiveness. LXD Research considers the study design, the number of studies, and its impact on student learning in its ratings. If a study has a randomized assignment but lacks meaningful differences in student outcomes regarding growth trajectories or having more students reach grade level goals, LXD Research ESSA rating may use Level 2 or 3. Studies demonstrating a large impact with compelling evidence with a cohort design can meet the LXD Research ESSA Level 2 rating. (Cohort designs are not eligible for Evidence for ESSA.)
LXD Research ESSA-Aligned Seals Can Help
In summary, the WWC’s commitment to rigorous research standards and evidence-based practices means that only a select number of educational products meet their criteria for inclusion. This limited list gives educators and policymakers confidence in its effectiveness. Still, WWC’s framework and timeframe are impractical for a world where over 10,000 products are used in a given school. LXD Research ESSA-Aligned Seals are here to help. The next time you ask for a company’s product research that doesn’t say the ESSA level, ask them to contact LXD Research.
Thanks to the help of OpenAI in getting the outline for this article!
OpenAI (2023). ChatGPT (September 25 Version) [Large language model]. https://chat.openai.comCopy