Skip to content
Home » Show Me the Research: How to Evaluate Math Tools Beyond the Demo

Show Me the Research: How to Evaluate Math Tools Beyond the Demo

Show Me the Research: Evaluating Math Tools Beyond the Demo — LXD Research
NCTM 2026 Session Recap · November 2026

Over a million people have watched a video I made in 2005 about carving a watermelon basket. I opened my session at the 2026 NCTM Annual Meeting & Exposition in Denver with that story because it holds the idea behind everything that followed: a tool can be delightful and still be unproven, and the two are easy to confuse. The basket video worked because anyone who followed my directions ended up with a gorgeous watermelon basket. It delivered the outcome it promised.

This post recaps the session, “Evidence-Based Decision Making: Empowering Math Leaders to Evaluate Digital Tools,” for anyone who could not be in the room. At LXD Research, we spend our days producing and reviewing the studies behind edtech claims, so the talk moves from three personal stories to a practical way of reading evidence, using one real product as the worked example.

What is a demo video actually for?

A demo video has one job: to show what is possible. I know this because I have made hundreds of them. The watermelon basket was my audition tape in 2005, before the iPhone, for a company hiring people to make how-to videos for About.com, which the New York Times owned at the time. It was the only video of mine that really took off.

The job that followed was less glamorous. I made about a hundred cooking videos, and I am not a chef, so I was inventing recipes and filming myself following them. One day the croissants came out raw in the middle. We put them under the broiler so they would look cooked on camera, and on screen they were perfect.

That croissant is what I think about when I watch a polished edtech demo. It is easy to make anything look good on camera, and the basket had to work in someone else’s kitchen while the croissant only had to work on screen. A demo shows what is possible, so ask what happens with real students in a real classroom, and ask for documentation of the testing that was done with them.

What does user testing actually change?

My second story is from spring 2001, when I was a production intern at Nickelodeon working on the first and second seasons of Dora the Explorer. Every episode had learning objectives it had to hit, and every episode was tested with kids in three rounds before it aired: a puppet show to test the story, a black-and-white version to test the script, and then the final cut, each with groups of New York City preschoolers.

Dora opens by asking the viewer a question to pull them in. In one test, the question was “Do you like to get mail?” We expected cheers and got crickets. After that group left, I raised my hand and said I did not think many four-year-olds get mail. The team asked what we should change it to, and I suggested “Do you like to go to the post office?” because I remembered going with my mother, with the line, the boxes, and the stamps on the wall. The next group heard the new question and answered with an enthusiastic yes. That is what user testing does: it finds the gap between what the designers assume and what children experience, before the show airs.

Plenty of edtech companies say they built a tool with teachers or tested it with students, and that is a good start. The harder question is what came after. Unlike a television episode, a digital tool keeps changing, so the testing, the findings, and the changes have to keep happening too.

My third story is about that cycle. After finishing my PhD coursework, I joined the research team at Lexia, which was building Core5, a new online adaptive platform to teach children to read. We did user testing, but the first version of Core5 was not the best version. In the first year, after the program expanded to hundreds of schools, the feedback rolled in. One teacher wrote “I hate the lady’s voice!” Children cried when they logged in and found their work was gone.

Those turned out to be symptoms of fixable problems. It was not the actress’s tone but the directions repeating over and over, so we stopped repeating directions a child had already heard that day. Students lost progress when they left an activity midway, so we saved it. We also rebuilt what happens after a wrong answer, so a child gets a simpler version and a short explanation before returning to the original question. After the changes, children completed activities about 8 percent faster. We tested again after the fixes, which is the part people skip.

What this means for a math leader: “We tested it with teachers” is the start of the conversation. Ask what the company changed because of what it learned, how it checked that the change worked, and what happens in the product when a student gets an item wrong.

Is a polished math tool the same as an effective one?

No, and the EdTech Quality Indicators help explain why. They were developed by the EdTech Quality Collaborative, a group that includes 1EdTech, CAST, CoSN, Digital Promise, ISTE, and SETDA, and they cover five areas: safe, interoperable, usable, inclusive, and evidence-based. I set safe and interoperable aside for this session and focused on three questions. Is it usable, meaning can teachers and students work it easily? Is it inclusive, meaning can every student get to it? And is it evidence-based, meaning what research is it built on, and what evidence shows it works, with whom, and in what contexts? A pilot answers the first question quickly. Easy to use is not the same as effective.

Math facts practice makes the point well. Two apps can both be called math facts practice. In one, the child sees the answer and taps to continue. In the other, the child has to pull the fact from memory and then gets feedback. The first can look wonderful and feel like a game, but learning science favors the second, which is called retrieval practice. The What Works Clearinghouse practice guide on helping students who struggle with mathematics recommends spending about ten minutes of each intervention session on building fluent retrieval of basic facts. That guidance is written for intervention settings, so it is not proof for every student or every tool, and it says nothing about any particular product. What it gives you is the question to ask a vendor: what is the child actually doing?

In the session, I asked everyone to try this with a neighbor, using a retrieval tool they know, at two zoom levels. In one moment, is the child recalling or looking and tapping, and what happens after a wrong answer? Across the week, what comes back, and when?

How do the four ESSA tiers work if you think of them as a recipe?

I am not a good cook, and I find recipes incredibly helpful, so I explain the Every Student Succeeds Act (ESSA) tiers of evidence through cooking. The tiers are a ladder, and the lower the number, the stronger the evidence.

TierThe kitchen analogyWhat it means
Tier IV: Demonstrates a RationaleThe recipeA logic model built on research: what students do, why that should work, and what you would expect to see. It does not yet show that it worked.
Tier III: PromisingThe recipe in other people’s kitchensA correlational study with statistical controls, asking whether students who used the tool more grew more after accounting for where they started.
Tier II: ModerateA formal test, with someone choosing the groupsStudents who used the tool are compared with similar students who did not. Teachers volunteer, or the district decides who goes first.
Tier I: StrongA formal test, with a lotteryThe same comparison, but the researcher randomly assigns who uses the tool now and who waits.

For Tier IV, the logic model is the recipe for the product and the promise of what it will do. In our free guide to evaluating a product logic model, we describe five parts: the problem and goal, the resources, the activities, the outputs, and the outcomes. The warning signs are vague goals, parts that do not connect to each other, and unclear outcomes. Because the tier asks only for a general review of the research, the quality of that rationale varies, so look for one that starts from the research rather than from features the company already had.

For Tier III, the phrase to remember is “apples to apples.” Groups that use a tool more often are not always alike. They may start higher or have more support, so the study accounts for where each student began before asking whether heavier users grew more. Tiers II and I both use a standardized test rather than the company’s own quiz, and Evidence for ESSA’s current standards ask for at least 350 students across multiple sites, with at least two treatment schools and two comparison schools when schools are compared.

A higher tier gives more confidence that the tool caused the results. It does not mean a product is better for every school, and a lower tier does not mean a product fails to work. Several Tier III studies in different settings can be more persuasive than a single study at a higher tier, because they suggest the results can travel. You can read more on our pages for Tier IV, Tier III, Tier II, and Tier I.

What does it look like when one product climbs the ladder?

To make the tiers concrete, I walked through Classtime, a practice tool created for the California state test (CAASPP). A disclosure: LXD Research worked with Classtime on its logic model and conducted both studies described here, so you may read this as a case from our own files. I chose it because the results are mixed in instructive ways. You can browse more of our work on the published studies page.

The recipe. Classtime’s logic model does not stop at listing features. Its outputs section says what teachers and students actually do: teachers focus instruction on class-wide and individual gaps in strategic thinking, and students take an active part in discussing the strategic thinking types a math problem needs. That is the kind of named behavior to look for.

The recipe in context. The first study, from 2022 to 2023, compared one Classtime school with two comparison schools, covering 667 students in grades three to six. Among students in grades four to six, the comparison schools grew 46 points on CAASPP. Students with low use of Classtime looked about the same, and students with more use grew more.

Comparison schools, no Classtime+46
Low use (fewer than 6 sessions)+48
Moderate use (6 to 12 sessions)+73
High use (more than 12 sessions, average 19)+97

Growth in CAASPP points, grades four to six, 2022 to 2023. This is a correlation, not proof of cause, because teachers chose how much to use the tool and classrooms that used it more may have differed in other ways. It still told the company something useful about dosage: roughly two to three sessions a month.

The formal test. The second study, from 2024 to 2025, compared 10 Classtime schools with 18 similar comparison schools, covering 6,937 students. It used i-Ready in the fall and CAASPP in the spring, with an analysis that controlled for baseline scores and demographics. Overall, Classtime schools scored about nine points higher, roughly a month and a half of additional growth, with an effect size of about +0.05. That is positive and modest. The grade-level picture is where the lesson sits.

Third grade +8 2414 comparison vs. 2422 Classtime
Fourth grade +7 2452 comparison vs. 2459 Classtime
Fifth grade +1 2492 comparison vs. 2493 Classtime
Sixth grade +23* 2496 comparison vs. 2519 Classtime

CAASPP scale scores by grade, 2024 to 2025. The asterisk marks the only statistically significant difference. Every grade was higher, but only sixth grade showed a clear difference, which suggests implementation was uneven when the program scaled from one school to ten. This matters because Evidence for ESSA’s standards say a statistically significant positive result in a single grade can qualify as a positive effect, provided that grade-level analysis independently meets the other requirements. One grade can carry a rating, which is why you should ask for results by grade, not just the average.

Two reviewers, two ratings: Evidence for ESSA lists Classtime as Promising, while our firm rates the same study Moderate. Reviewers apply different rules, so check more than one source before you decide what a rating means.

So where does Classtime stand against a reasonable bar? I offered a rule of thumb for state and district leaders: look for Tier III studies in many settings, and at least one Tier II study with a positive, meaningful result on state scores or common assessments. “Meaningful” means more than statistically significant, so ask how big the result is. This is a rule of thumb rather than a state requirement, so check your own state’s rules. Classtime has one Tier III and one Tier II study, both in the same California district, and the Tier II result is modest. It is promising, not proven, and the many-settings piece is not yet met, which shows the checklist working on a product we know well.

The next rung would be Tier I: finding new interested districts and randomly assigning at least four schools, two to use the tool and two to wait. Randomized studies are rarer and usually narrower, which is why several studies in several settings matter so much.

Delight draws students in. Evidence shows how and when it works.

Remember the watermelon basket? People watched because it looked delightful, and they kept coming back because it delivered what I promised. Math tools are the same: students ask for the app that is fun, and fun is worth having. It just answers a different question from the one a school leader needs answered.

Delight
Why students come to it

Polish, game design, kid appeal, price, and popularity. Valuable, and none of it says students learn more.

Evidence
Why it earns a place

What students did, who they were, how much they used it, how big the change was, and what the company changed afterward.

The only thing we can count on in education, and in life, is that things will change. Products update, students change, and standards move. So we can never stop asking for research.

Where does the evidence live, if a search will not surface it?

It is rarely where a quick search takes you. As a live exercise in the session, I searched for the best apps for math fact practice. The AI overview described a good app as balancing quick drills with engaging design, followed by a list with a line each about price and features. The sponsored results emphasized popularity, kid appeal, and price. Further down was a roundup of teacher favorites from Edutopia. Teacher-tested means teacher-recommended, which is useful, and it is different from evidence that students learn more. None of the results I reviewed pointed to a study, and neither the EdTech Index nor Evidence for ESSA appeared. That does not mean no studies exist. It means those pages will not tell you, and you have to know where to go.

Here is the order I suggest. Start with the ISTE EdTech Index, the searchable directory built around the five quality indicators, which lists each product’s third-party validations. Under Evidence-Based, the validators include Digital Promise, Instructure’s ESSA Evidence, Leanlab, EduEvidence, and my own firm, LXD Research, which is why I walk people through several sources rather than only mine. On a product page, a dark check means an outside reviewer validated that indicator, while a lighter bar means the company reported it itself. Click into any badge to see what it covers and who issued it. Next, check Evidence for ESSA from Johns Hopkins, which includes math programs. If a product is on neither, ask the vendor for its studies and check the tier yourself. EduEvidence, which awards Bronze, Silver, and Gold, appears on the Index as an international option.

Not listed is a result, not a failure. Finding nothing does not mean a product is ineffective. It means these directories do not show evidence for it, so the next step is to ask the vendor.

What should you ask before you buy?

In the session, we used the phrase “Show me the research,” and I sorted the questions into three steps: verify, match, and pilot.

Verify. What studies exist, and can I read them? Who conducted them, who paid for them, and who reviewed them? Who was the comparison group? The right answer is similar students who did not use the tool.

Match. Do the students and settings in the study match mine? Check the math skill progression and grade level, and ask for results by group, including multilingual learners, students with disabilities, and students from low-income families, ideally in more than one district. An average can hide the students you most need to reach.

Pilot. What usage produced the results, in terms of how often, for how long, and with what support? How big was the effect in plain terms, and what has the company changed since? Then test the tool in your own classrooms, with checks before, during, and after.

The Unlocking Evidence Checklist from the session handout walks through the same ideas. If you build or sell math tools, these are also the questions your buyers are learning to ask, and clear answers come much more easily when the research path is planned. The overview of how to get ESSA certified is a good place to begin, and our Levels of Evidence PDF puts the tiers on a single page.


Not Sure What Level of Evidence Your Product Has?

Get a research-path recommendation for your product, timeline, and procurement goals.

Book a Free Conversation See the ESSA Certification Path

Or email us at info@lxdresearch.com