We have two very interesting stories that both came out within a day of each other. Together, they expose a hole in how districts govern AI.
On Wednesday, Common Sense Media's Youth AI Safety Institute rated the AI built into Google search, AI Overviews and AI Mode, an unacceptable risk for kids, its lowest possible rating. I don’t know about your district, but that AI sits on every Chromebook in America by default. Nobody decided. Nobody procured it. It’s unlikely your school lawyer vetted it. Yet, there it sits on your network. The day before, Anthropic launched Claude for Teachers, a year of free premium AI for any verified educator, joining OpenAI, Microsoft, and Google, which have all rolled out free teacher offerings of their own in recent months. Those accounts arrive one teacher at a time, through an inbox, and never cross a district desk either. Governance costs extra.
Your vetting rubric, your DPAs, your board-approved tool list: all of it works through procurement. These two apps route around that moment entirely. No one procured them at all. This issue is about extending your governance to the AI you didn’t choose and probably don’t want.
Some folks have reached out to us asking how to subscribe themselves or their colleagues to this newsletter. Click here to do so. We send a newsletter every two weeks about the latest happenings with K-12 and AI.
IN THIS ISSUE:
What independent testing found in the AI search tools students use by default
Why the free-teacher-account era demands a district answer before September
2,600+
Test searches Common Sense Media ran against Google's AI search tools using child accounts with the strictest safety settings. Source
100%
Share of hypothetical homework assignments the AI completed for testers posing as students. Source
75%
US tweens and teens who already use AI-generated search results, per Common Sense Media's census. Source
43%
Repeated history questions that got different answers on different tries in testing. Source
4
Major AI companies now offering teachers free premium accounts: Google, OpenAI, Microsoft, and Anthropic. Source
The Common Sense Media review is worth reading in full, because its methodology anticipates the most obvious objections. Researchers ran more than 2,600 searches and audited over 2,100 cited sources, using accounts configured as an 11-year-old with active parental management and a 15-year-old, with SafeSearch on. Essentially, they tested the safest version available. It still failed all five of the institute's severe-harm red lines. Worried about kids losing critical thinking? The tools also completed every homework assignment testers asked them to do. Worse yet, the Gemini-powered search engine missed clear crisis signals including explicit statements of suicidal ideation, pointed a struggling tester to an eating disorder helpline that was disconnected in 2023, and cited social media posts alongside peer-reviewed research with no distinction between them. Robbie Torney, who leads the institute, put the practical takeaway simply: “It can be helpful, but it can also be wrong.”
Google disputes the findings, saying the queries were contrived, that it could not replicate many results, and that its own testing shows higher quality. You can decide for yourself. But two facts aren’t in dispute. First, these features are the default on the world's dominant search engine, which means they reach students on district devices without any adoption decision ever being made. AI Overviews cannot be meaningfully disabled for schools while keeping search itself. Second, this is not about one company, and it is not even about the underlying model. The same reviewers have rated the standalone Gemini chatbot a full tier better than the search features built on it, and have rated a string of major chatbots as high risk in previous assessments, including ChatGPT and Claude. The pattern is the point: the failures concentrate in the default integrations, and consumer AI, delivered by default and tested independently, keeps failing tests that education-grade tools would be required to pass before a single student touched them.
OUR TWO CENTS
Someone at Google made a decision for your district. They decided. Your governance team never got a vote. We recommend you take the vote back. Start by inventorying the ambient AI in your environment, meaning everything students encounter that no one approved: AI features inside search, inside browsers, inside the free apps that ride along on every device. Then make deliberate calls. Decide what elementary research looks like, and whether it starts from librarian-vetted databases instead of an open search box, which is exactly what Common Sense recommends. Configure what can be configured, and document what cannot, because "we assessed it and cannot disable it" is a defensible answer to a parent, while "we never looked" is not. And use the report itself as teaching material. Have students run the same prompt five times and compare the answers. The flaws in these tools are the best AI literacy curriculum anyone has written this year.
Add a section for default and ambient AI to your tool inventory, alongside the tools you actually adopted.
Set an explicit district position on AI search for each grade band, and route elementary research through vetted databases.
Brief your counselors: AI search tools failed crisis-detection tests, so make sure students know the humans and hotlines that actually work.
Turn the report into classroom material. Comparing inconsistent AI answers teaches more skepticism than any slide deck.
Within the past year, OpenAI, Microsoft, Google, and now Anthropic have each launched free premium AI offerings aimed directly at teachers. The newest, Claude for Teachers, gives verified educators a year of free access, with curriculum libraries mapped to standards in all 50 states and, notably, privacy commitments: teacher conversations are not used for model training, and the company says student information protections are built for FERPA compliance. A version for schools and districts is, in the company's words, coming soon.
The vetting process requires teachers to upload their paystubs, in some cases, and requires the teacher to consent to a DPA. You read that right. Teachers are reading legal documents and signing up online.
That sequence, individual teachers first, district offering later, is worth scrutinizing. It is, also, the industry's standard playbook. Free accounts drive bottom-up adoption. By the time the district version arrives, the district is negotiating about tools its staff already uses. None of this requires bad faith by anyone, and the privacy commitments attached to these offerings are improvements from the commercial versions. But it shows a lack of understanding of how districts actually work. The district, not the individual teacher, is the steward of student data. When a teacher uploads assessment results or IEP notes to a personal account on any platform, however reputable, the district owns whatever comes next: the investigation, the scope determination, the parent communication, the deletion request, and, sometimes the lawsuit. The teacher clicked upload. The district inherits the fallout.
OUR TWO CENTS
I run an AI company that sells to districts, so read this knowing where I fit into the puzzle. The free teacher account playbook works because it treats your staff as an acquirable user base and your district as a later conversation (who can pay the freight). Governance is not an enterprise feature to be unlocked at the paid tier. It is the prerequisite, and districts should say so.
Everyone signing up for free AI tools should remember that the trillions of dollars being invested in AI data centers means that it’s unlikely Google, OpenAI, Anthropic, and the like are going to give their product away for free. Remember: many of these companies are burning billions of dollars; if you get the product for free, you are the product.
The companies aren’t looking out for your district, despite their claims. They’re here for shareholders.
So what to do? Publish your position on teacher-facing free AI accounts: which are approved, under what terms, and what may never touch them, starting with anything resembling student records. And put the question to every vendor, including ClassCloud: is governance included or is it extra? Districts should reward the companies that get this right.
Publish a one-page position on free teacher AI accounts before school starts: approved, conditional, and prohibited uses.
Draw the brightest line at student data: no student records, IEP content, grades, or rosters in any individual account, on any platform.
Build the fast lane. A 5-day tool review process is the single best defense against workarounds.
Rehearse the incident play: who investigates when district data reaches an unapproved app, who determines scope, who notifies, and who demands deletion.
Vetting Generative AI Tools for Use in Schools by the Future of Privacy Forum
A departure from our usual book pick, because this issue calls for a working document. FPF, a nonprofit that has spent a decade on student data privacy, wrote the checklist your technology director needs for the next free tool that shows up: how FERPA and COPPA apply to generative AI, what to ask about whether student inputs train the underlying model, and which contract terms separate an education-grade tool from a consumer product wearing an education label. Print it, and hand it to whoever answers your teachers' tool requests.
Chat with ClassCloud
We’re listening. Let’s Talk!
This newsletter works best when it’s a conversation, not a broadcast. If you want to talk through how any of this applies to your district specifically—or if you have feedback on what would make this more helpful—just hit reply. We read and respond to everything.
Schedule a Virtual Meeting
Thanks for reading,
Russ Davis, Founder & CEO, ClassCloud ([email protected])
Tracy Walters, Director of Customer Growth ([email protected])
ClassCloud is an AI company, so naturally, we use AI to polish our content.
