GEO for Education: Six University Sites, Zero llms.txt Files
What we found on six education sites: one llms.txt file, two bot walls
We scanned each homepage, its robots.txt, and the llms.txt path with plain HTTP requests — no JavaScript, no browser. That's the closest approximation of what a text-only AI crawler actually receives, and the results were stark:
| Site | llms.txt | AI crawler rules in robots.txt | Structured data on homepage |
| Harvard | missing (404) | none — blanket allow-all | Organization, WebPage, Person |
| MIT | missing (404) | none — blanket rules only | none |
| Stanford | missing (404) | none — blanket allow-all | none |
| Oxford | unreachable | unreachable — Cloudflare challenge blocks robots.txt | unreachable |
| Coursera | published (86 lines) | GPTBot, ClaudeBot, PerplexityBot allowed; /lecture/ blocked | Organization, FAQPage (7 Q&A) |
| Khan Academy | soft-404 — HTTP 200, HTML challenge shell | soft-404 — same challenge HTML | none — JS-gated |
The other five sites have no policy at all. Harvard and Stanford run plain allow-all robots.txt files. MIT disallows a few stale internal paths and nothing else. There's no answer to "may AI read this site?" because the question was never asked.
Why the institutions with the most to gain ship the least structured data
Students already ask LLMs the questions these schools care about: "best computer science programs," "MIT vs Stanford," "how selective is Oxford," "is Coursera worth it." The models quote rankings, Reddit threads, and niche blogs because the institutions don't hand them anything better. The facts a prospect wants — acceptance rates, deadlines, tuition, program requirements — live in PDFs and JavaScript tabs, the two formats language models read worst.
Our GEO statistics scan across six industries puts the average site at 45 out of 100. Nothing in this education check suggests the sector scores higher. Harvard carries Organization, WebPage, and Person schema on its homepage — the most complete of the four universities — and still skips CollegeOrUniversity, the type that tells a model what kind of institution it is. Stanford's homepage is 779 KB of HTML with no structured data inside it. MIT's homepage has none either. A model that reads all of those bytes walks away knowing the schools exist and almost nothing about what they offer.
Khan Academy surprised us the most. Its mission is learning for everyone, yet its entire site — homepage, robots.txt, llms.txt — responds to non-browser clients with a 3 KB Fastly "Client Challenge" page and an HTTP 200 status. An AI crawler following the llms.txt convention sees a 200, tries to parse HTML as markdown, and silently moves on. Nobody blocked GPTBot on purpose here. The bot wall went up as infrastructure protection, and no one checked what it does to AI crawlers.
Oxford sits behind the same kind of wall, built by Cloudflare: its robots.txt returns 403 with a "Just a moment..." challenge, and a Googlebot user agent drew the same response. Text-only crawlers can't execute JavaScript, so the challenge never resolves for them. Oxford never answers whether AI crawlers may enter — the wall swallows the request before the question exists.
How to fix it: five changes, in the order they matter
- Publish llms.txt this week. It's a text file with no build step. List your top 30 pages — admissions, programs, tuition, financial aid, one page per faculty — and check the format with the llms.txt checker. Our llms.txt guide covers anatomy of a good file. Coursera proved this move alone puts you a year ahead of every university we tested.
- Add CollegeOrUniversity schema to the homepage and FAQPage to admissions pages. Admissions teams already maintain the Q&A students ask — acceptance rate, deadlines, transfer credits, housing. Wrap those blocks in schema so a model treats them as answers instead of prose. The schema checker catches syntax errors before deploy, but start by just marking up the 5-10 questions your admissions office answers weekly. Harvard's missing CollegeOrUniversity shows this is a gap even at the top of the market.
- Your bot protection may already be blocking AI crawlers. If you run a Fastly, Cloudflare, or Akamai challenge, pull up the AI crawler check from a server, not a browser. You'll see exactly what Khan and Oxford show right now: 200s that carry HTML shells, or 403s that never end. Verify GPTBot and friends can still reach robots.txt, llms.txt, and your core pages.
- Write answer blocks for the questions students actually ask. Not brochure copy. Open each admissions page with a 134-167 word passage that says what the program is, who it's for, and what it costs — then let the rest of the page elaborate. Our how to get cited by ChatGPT walkthrough explains why models quote that shape and that length.
- AI search is a recruitment channel now, whether or not you staff it. A prospective student who gets a good answer in ChatGPT that names your program is further along than one who saw a billboard. The schools that hand the model clean, quotable facts will get the mentions; the ones that keep their facts in PDFs will get skipped.
The split is consumer vs. institution, and it's not close
Coursera treats AI answers as a funnel — its llms.txt names the products, its FAQ schema answers the objections, its robots.txt protects only what's paid. Every university we checked treats AI search as something that happens somewhere else. The gap is fixable in an afternoon for the file and a sprint for the schema, which is exactly why the schools that move first will own the answers for years. Run the free GEO audit on your admissions page and homepage today — you'll get a six-dimension score and a fix list ordered by impact.