Show Me the Research: How to Evaluate AI Tools Before You Buy
At MassCUE, Dr. Paul Chase and I led a session called “Navigating the Evidence Clearinghouses.” We started with a simple observation: districts are adopting AI tools at unprecedented rates, often on the strength of a vendor demo alone.
The session wasn’t meant to discourage that adoption. It was about adopting wisely, and about the free resources that let a leader verify the evidence before signing a contract.
Why Isn’t a Great Demo Evidence?
A sales demo shows a tool at its best, with a prepared example and a friendly audience. Evidence shows what happens to student learning when real classrooms use the tool over time. I used to build the sample data for demos at companies I worked for, so every feature had something to show. A demo can reflect what a classroom looks like three months in, not the first week, and it was never designed to answer whether students learn more.
District leaders have told us that privacy and security have been the first screen for the last couple of years, and many vendors now clear it. That moves the question to the next stage. Of the tools that are safe and secure, which will be effective in a given context? Massachusetts districts already see this logic in curriculum procurement, where implementation efficacy accounts for half of a bid’s score; we break that down in our guide to Massachusetts curriculum efficacy requirements.
Look for documentation, not a demonstration. A good vendor can show what implementation actually looks like at the start of use, including for students with different ability levels.
What Does an Evidence Badge Really Tell You?
Badges answer two different questions, and it helps to know which one you’re looking at. Design badges are the blueprint review: was the product built on sound research? Effectiveness badges are the building inspection: does it work in real classrooms? A product can have one without the other.
Most effectiveness badges follow the four tiers of the Every Student Succeeds Act (ESSA):
| Tier | What it requires |
|---|---|
| Tier I (Strong) | A well-designed randomized experiment |
| Tier II (Moderate) | A well-designed study with a comparison group but no random assignment |
| Tier III (Promising) | A correlational study with statistical controls (small experiments can land here too) |
| Tier IV (Demonstrates a Rationale) | A research-based logic model and a plan to study the effects |
For any claim you encounter, three questions help you decide how much weight it deserves:
How rigorous is the study design?
How large are the effects on student learning?
Has it been replicated in settings like mine?
Built to Work, or Shown to Work?
Every badge you see in a vendor’s materials is a claim about one of two things. Keeping them apart is the most useful habit a district buyer can build.
Design badges review the blueprint. Was the product built on sound learning research, with a logic model that explains how it is meant to help?
Effectiveness badges inspect the building. Does it improve outcomes for students in real classrooms, compared with similar students who didn’t use it?
A new AI tool may reasonably have the first without the second. That’s a reason to ask questions and plan a pilot, and not a reason to reject the tool.
Which Clearinghouses Should You Check?
No single source covers every product, which is why we suggest checking more than one. The nonprofit validators and the researchers who award their own badges each answer a slightly different question, and the ISTE EdTech Index collects many of the badges in one searchable place. Use it to build a shortlist, then open the studies behind each badge.
| Source | What it helps you answer |
|---|---|
| Evidence for ESSA | Which studies exist for a program, and at which ESSA tier? Run by Johns Hopkins University. |
| Digital Promise Product Certifications | Was the design based on research (Research-Based Design: ESSA Tier IV), and is there a well-designed study (Evidence-Based Edtech: ESSA Tier III)? |
| EduEvidence | Which efficacy and effectiveness badges (Bronze, Silver, or Gold) has a product earned? Browse the EduEvidence global list. |
| Leanlab Education | Does the product hold Codesign or ESSA evidence badges from Leanlab? |
| Instructure (LearnPlatform) ESSA Evidence Badges | Has a study met the requirements for a specific ESSA level, and what year was it completed? |
| LXD Research Published Studies | Which products have LXD Research studies and ESSA badges, and what did each study find? |
| ISTE EdTech Index | Which badges has a product earned across these organizations, in one place? |
For research beyond the badges, we also pointed attendees to ResearchGate, ERIC, and AI chatbots, which can help you find and summarize published studies. Always open the original paper before relying on a summary. If you want a step-by-step process, see our guide, Five Steps to Finding Research Studies.
What Should You Note on Every Search?
During the session we searched Edpuzzle across the platforms and compared what turned up, then invited the room to suggest tools their districts are considering. We came back to the same four things each time.
- Is it listed? No listing is a prompt to ask questions, not a verdict.
- Which badge, and from whom? Design badges and effectiveness badges answer different questions.
- Who was studied? Compare the grades, subjects, and student groups to your own.
- How many studies, and how recent? A portfolio across settings beats a single study.
The student-group question deserves a closer look. A study may report that 10 percent of its sample belonged to a given group, but that doesn’t mean the study analyzed that group separately. Check whether the subgroup results are broken out. A single study shows promise in one setting, while several studies across settings suggest results can travel, even when each study is Tier III.
Ask for the full report. Who conducted the study, who the comparison group was, what usage produced the results (time on task, training, and support), and whether the methods are shared are all fair questions for any vendor.
What If the Tool Isn’t Listed Anywhere?
Many AI tools are new and won’t appear anywhere yet. That is common, and it isn’t proof the tool doesn’t work. Ask the vendor for studies and a logic model, then plan a pilot with three stages:
- Before: define the question and success measures, and collect a baseline.
- During: track usage and implementation, and compare with a similar group.
- After: check results for all student groups, then decide whether to scale.
A few red flags are worth watching for: percent-improvement claims with no study behind them, no comparison group, company-written research with no methods, only engagement or satisfaction data, and reluctance to share the full report.
How Do You Know the Measure Itself Is Any Good?
A pilot or a vendor study is only as good as what it measures. Many vendors report gains on an internal metric that no one has validated. If students went from 10 percent to 40 percent proficient on a measure like that, you can’t tell whether it reflects learning you care about, whether it’s reliable across students and over time, or whether it was tested with 50 students in one setting.
When LXD Research validates an assessment, we ask whether it makes sense internally, whether students grow on it, and whether it tracks trusted outside measures like MAP or iReady for the same students in the same year. We describe the results in three levels. The first is preliminary: strong correlations, but nothing too robust. The second shows the measure works in the context where it’s used, though that context is limited. The third requires enough data, ideally nationally representative or close to it, to show the measure indexes the intended skills across subgroups and under stress testing. For a real example of this kind of work, see the 95 Phonics Screener for Intervention validity study report.
For assessment developers preparing for review by the National Center on Intensive Intervention, a research-validated measure is also a useful stepping stone. Read more about our Research-Validated Measures service.
What Should You Do on Monday?
We closed with three takeaways that work as a checklist:
- Verify. Check more than one source, and look past the badge to the study behind it.
- Match. Ask whether the students and settings in the research look like yours.
- Pilot. When evidence is thin, define success up front and measure before you scale.
Or, as our last slide put it: Show me the research!
Want to research a product in your district?
Book a meeting with Dr. Schechter to see what might be possible, or send us an email.
Book a Meeting with Dr. Schechter Email UsBuilding the evidence for your own product? See how to get ESSA certified or our Research-Validated Measures service.