The Score Went Up. Did the Learning?

AI can help students perform better. Schools still need to know what students can do when the tool is no longer helping.

Schools are beginning to face a new measurement problem because of AI. A student can work with an AI tutor during a difficult math unit, complete more problems correctly, get immediate help when something goes wrong, and finish the unit with much stronger scores than where the student began. Every one of those improvements may be real, and the tool may be providing useful support. The question is whether the school is measuring what the student learned or what the student and the tool were able to do together.

That difference is easy to miss because the information schools receive from these tools often shows what happened while the support was available. The student answered more questions correctly, accuracy improved, more work was completed, and the student moved into harder material. A teacher looking at that progress may reasonably believe that understanding has improved, and a school leader reviewing the same data may see evidence that the tool is working.

The numbers are not necessarily wrong. They may be showing exactly what happened while the student was using the tool. The problem begins when schools treat those results as proof of what the student can now understand, explain, and do without the same level of support.

AI is changing what student performance means because some of the performance schools are measuring now includes a much larger contribution from the tool itself.

Consider what this can look like during a regular math unit. A teacher assigns an AI-supported practice tool because students are having trouble with the material. The tool adjusts questions as students work, gives explanations when they get stuck, points out mistakes quickly, and gives students another chance to try. Over several weeks, the dashboard shows steady improvement. Students who struggled at the beginning of the unit are answering more questions correctly and moving into harder material, so the teacher begins reducing the amount of reteaching and prepares to move the class forward.

That may be the right decision. The students may have developed a stronger understanding and may no longer need the same amount of support. But the dashboard alone cannot tell the teacher that. Some students may have learned the math well enough to work more independently. Others may be succeeding because the tutor gives a hint at the exact moment they get stuck, corrects an error before they continue, or shows the next step when they are unsure. Those students can produce similar scores while having very different levels of understanding once the support changes.

If the school never looks at the skill under another condition, those differences can stay hidden. The class moves forward, the dashboard records success, and the gap may not appear until students meet the same concept later without the tool. The next teacher may receive students who look successful in the data but struggle when asked to use the skill without that particular support.

The same issue can show up at the school level. A principal comparing two tutoring products may receive dashboards showing time on task, questions completed, accuracy, engagement, and growth during practice. Both products may appear to be helping students, and those measures can tell leaders whether students are using the tools and whether the support is helping them complete the work.

What those reports may not show is what students can do after the support is reduced or removed. Can students solve a similar problem without the tutor? Can they explain why the steps work? Can they find an error without the system pointing it out first? Can they use the skill in a different kind of problem? Does the improvement remain when the specific support being studied is no longer available?

Those questions matter because two tools can both improve supported performance while affecting student learning in very different ways.

The issue becomes even more important during district purchasing and renewal decisions. A company may present data showing that students improved while using the product, and those numbers may come from the district's own students. Leaders may see better accuracy, more completed work, or stronger scores over time and reasonably view those results as evidence that the tool is useful.

The district still needs to know what those numbers actually measure. If the tool is being purchased because the district wants students to become stronger in math, improve reading comprehension, write more independently, or remember concepts for longer periods of time, then data collected only while the tool is helping students answers only part of the question. It shows what students can do with the support. It does not automatically show what they learned well enough to carry forward without it.

A parent can experience the same difference without ever seeing the dashboard. Grades may rise. Homework may become easier to complete. A student who once needed a great deal of help may begin finishing work more easily because the AI tutor is available when the teacher is not. From the family's point of view, that can look like growing independence. It may be growing independence, or it may mean the student has become very good at working with the support.

Schools need enough evidence to tell the difference before families and students are told that a skill has been mastered.

This is not an argument that students should be expected to do everything without help. Schools have always provided support, and some supports are meant to remain. A student with a disability may use technology that provides access to the curriculum rather than temporary help the student is expected to outgrow. A multilingual learner may need language support while showing understanding of a science or math concept. Removing those supports in the name of measuring independence may create another inaccurate result because the school ends up measuring the barrier instead of the learning.

The more useful question is which support the school is trying to evaluate and what students are expected to be able to do when that particular support is no longer present. If an AI tutor is being used to help students develop independent skill with equations, the school needs some evidence of whether students can later solve similar equations without the tutor supplying hints, corrections, or next steps. If AI is providing an accessibility support the student is expected to keep using, then that support may be part of the learning environment rather than something that should be removed during assessment.

The measurement has to match the learning goal.

Why It Matters

Schools use student performance to make decisions long before final grades are issued. Teachers decide whether to reteach a concept, move to the next unit, change a group, provide intervention, or reduce support based on the evidence in front of them. When AI-supported performance is treated as independent mastery, students can move forward before the underlying skill is secure.

That matters especially for students who needed the support most in the first place. A student who has struggled with math may experience real relief when an AI tutor makes the work more manageable and immediate feedback helps the student keep going. That support can be valuable. But if the school decides from better-assisted performance that the student no longer needs additional instruction, the tool can unintentionally hide the need it was brought in to address.

The student does not experience that later as a problem with data. The student may experience a sudden drop in performance in a subject that seemed to be going well. The work looked manageable while the support was present, and then a later assessment, a new course, or a different kind of problem shows that the understanding was not as strong as everyone thought.

That is why schools need this information while teachers can still respond to it. An end-of-year assessment may eventually show that students did not retain as much as expected, but that result comes too late to help with the unit where the learning was supposed to develop. A more useful check happens closer to the AI-supported practice, while the teacher can still adjust instruction.

Recent controlled research gives schools a reason to pay attention to this difference. In a large study involving nearly 1,000 high school math students, students using two different AI tutoring systems performed much better during supported practice. When the tools were removed for a later exam, those gains did not carry over in the same way. Students who had used a more open ChatGPT-style tutor performed worse than students who had practiced without AI, while students who used a more structured tutor designed with learning safeguards performed about the same as the comparison group.

The study should not be read as evidence that AI tutoring is bad for learning. It points to something more useful for schools. The design of the support matters, and strong performance while a tool is available does not automatically mean students developed the same level of skill without it.

A system that gives students answers quickly can create a very different learning experience from one that keeps more of the reasoning with the student. One tool may provide the next step right away, while another may ask a question that requires the student to continue thinking. One may correct the work before the student has to find the problem, while another may give a hint and require another attempt. Those design choices affect how much thinking remains with the learner.

They also affect how schools should read dashboard data. A platform may accurately report that a student completed more problems correctly, but leaders still need to know how much help was built into that performance. The stronger the support becomes, the more important it is to know what the school is actually trying to measure.

This does not mean schools should distrust the data from AI tutoring tools. It means schools should use the data for what it actually shows. If the goal is to know whether a tool helps students stay engaged longer, complete more practice, receive faster feedback, or work more successfully with support, then assisted performance is useful information. If the goal is to claim that students have developed stronger independent skill, the school needs another measure.

Teachers can begin without creating another testing system. When an AI tutor or practice tool plays a major role in a unit, a teacher can schedule one short follow-up within the next week or two using the same basic skill without the specific AI support being evaluated. Students might solve several related problems, explain a concept, complete a short retrieval task, or use the skill in a slightly different situation.

The check does not need to be high stakes, and it does not need to repeat the unit assessment. Its purpose is to give the teacher another piece of information about what remained after the supported practice. If students perform well during the AI-supported work and again during the later check, the teacher has stronger evidence that the learning carried beyond the tool. If performance changes a great deal, the teacher has useful information about where more independent practice, explanation, or instruction may be needed.

Either result gives the teacher more information than assuming.

Instructional leaders can bring the same thinking into data meetings. When a product report shows improvement, leaders can ask what conditions produced the number. Were students receiving hints? Could they ask the tool for an explanation or a solution? Was the score produced while the tutor was correcting mistakes? Has the school looked at the same skill after the support changed?

Those questions do not dismiss the data. They help the school understand it more accurately.

District leaders can bring that same expectation into purchasing and renewal conversations. When a vendor presents evidence of student growth, the district can ask whether the measure reflects performance while students used the tool, performance afterward without the tool, or both. Leaders can ask how students were assessed, whether there was a comparison group, how long after the supported practice learning was measured, and whether the evidence shows improvement in the underlying skill.

The point is not to demand that every company prove every possible benefit. It is to make sure the evidence matches the claim the district is using to support the purchase. If a company says the tool increases student engagement, engagement data may be enough to support that claim. If the claim is that students learn math more effectively, the district needs evidence connected to math learning.

Families can ask a simpler version of the same question. When a child uses an AI tutor often and grades or practice scores improve, families can ask what the child can now understand or do that could not be done before and how the school knows. That question does not challenge the tool or the teacher's choice to use it. It keeps the conversation focused on the student rather than on the technology.

Schools also need to avoid turning this into a choice between support and independence. Students learn with help. Teachers model, prompt, question, scaffold, reteach, and change the amount of support as understanding grows. AI tutoring belongs inside that same instructional question.

The difference is that AI can provide support continuously and respond to students at a level schools have never been able to provide to every learner at once. That is part of what makes the technology valuable. It is also what makes measurement more complicated because the support can become so closely connected to the student's performance that the school stops seeing where one ends and the other begins.

That is the shift schools need to notice. AI is not only changing how students receive help. It is changing what successful performance can tell teachers, leaders, families, and students about learning.

Schools do not need to wait for perfect research or a new accountability system to respond. They can begin by pairing AI-supported performance with small opportunities to see what students retain, transfer, explain, or perform under the conditions the school actually cares about.

For a classroom teacher, that may mean one short follow-up after a heavily AI-supported unit. For an instructional leader, it may mean asking what a dashboard actually measures before using it as evidence of mastery. For a district, it may mean making sure an adoption or renewal discussion clearly separates performance with the product from evidence of learning beyond it.

The practice is simple, but the habit matters. AI tools will continue getting better at helping students complete work, and the numbers produced while students use them will probably become more impressive. That makes it even more important for schools to know when a higher number reflects stronger learning, stronger support, or both.

A school does not have to choose between appreciating what the tool helps students accomplish and asking what students can now do for themselves. It needs both pieces of information.

The score going up is worth noticing. The learning is what the school still has to determine.

This is one of many hidden shifts AI is introducing into education. I write these newsletters to help educators and school leaders see them early and respond intentionally. If that matters to you, stay with the series.