AEO is just SEO with content negotiation

If you've been on LinkedIn this year, you've seen the AEO posts. Answer Engine Optimization. Generative Engine Optimization. LLM Optimization. The pitch is always some flavour of "SEO is dead, AEO is the future, here's a course."

It's not. AEO is mostly SEO. The fundamentals that worked in 2024 still work: clear titles, accurate meta descriptions, structured content, internal links, real backlinks, no boilerplate. Write things humans (and now machines) want to cite.

There's one genuinely new part. Agents are a different kind of visitor. They don't render JavaScript reliably, they hate bloated HTML, and their context windows are small. The fix is small too: serve them a cleaner format when they ask for one. That's content negotiation, and it's been in the HTTP spec since 1996.

This post is the practical version. What we kept doing on robotostudio.com (SEO), what we added (content negotiation), and what we ignored (most of the AEO discourse). It's largely the same playbook Vercel laid out in Making agent-friendly pages with content negotiation, with a few extra details from doing it on a real production site.

What's actually new

Agents are a new class of visitor. They:

• can't run JavaScript reliably
• choke on ads, navigation, footers, and cookie banners
• have small context windows, so wasted tokens cost real money
• prefer structured text they can quote verbatim

Our HTML pages are fine for browsers. Styles, scripts, fonts, and analytics all add up. The markdown twin of any page on robotostudio.com is roughly 94% smaller than the HTML version on the same URL. No JSX components, no tracking, no nav.

That's the whole problem AEO is actually solving. Not "how do I get cited by ChatGPT," but "how do I serve a few KB of clean text instead of a hundred KB of markup when the visitor is a machine."

The SEO half (nothing changed)

Before any AEO plumbing, do the SEO that's been working since the Panda update. Quick summary, since you've probably read this before:

• Titles target the actual query, sentence case, under 60 characters
• Meta descriptions are written for humans and include the keyword
• Schema markup where it earns rich results: BlogPosting, FAQPage, Service
• Internal links from blog posts to relevant service pages, not just topic clusters (our generative engine optimisation service is a good example of a destination worth linking to from AEO-adjacent content)
• Real backlinks from people who chose to cite you, not directories
• Content that says something specific, not "the ultimate guide to [topic]"

Three observations from doing this for a few years:

1. Your title tag is the single biggest lever. Position 8 with a 0.2% click-through is not a content problem, it's a title problem. Fix titles before you write a word of new content.
2. Internal linking is undervalued and free. We added contextual CTAs from every blog post to the most relevant service page. Took an afternoon. The blog cluster now feeds traffic into services rather than dead-ending at "related posts".
3. Most "AEO checklists" are SEO checklists with the word "answer" added. If you skip the SEO basics, no amount of llms.txt will save you.

The new half (content negotiation)

Now the actually new bit. Here's what each piece does on robotostudio.com.

The Accept header tells you who's asking

The HTTP Accept header is how a client tells the server which formats it can handle. Browsers send text/html. Agents that want markdown send text/markdown. Claude Code happens to send text/plain. Same URL, different format, no client-side knowledge required.

In next.config.ts:

const acceptMarkdown = {
  type: "header" as const,
  key: "accept",
  value:
    "(?=.*(?:text/plain|text/markdown))(?!.*text/html.*(?:text/plain|text/markdown)).*",
};

const beforeFiles = [
  { source: "/blog/:slug",      has: [acceptMarkdown], destination: "/api/md/blog/:slug" },
  { source: "/services/:slug",  has: [acceptMarkdown], destination: "/api/md/services/:slug" },
  { source: "/migration/:slug", has: [acceptMarkdown], destination: "/api/md/migration/:slug" },
  { source: "/case-study/:slug",has: [acceptMarkdown], destination: "/api/md/case-study/:slug" },
  { source: "/careers/:slug",   has: [acceptMarkdown], destination: "/api/md/careers/:slug" },
  { source: "/",                has: [acceptMarkdown], destination: "/api/md/pages/home" },
];

The regex looks scary but it's defensive. It matches text/markdown or text/plain only when those tokens appear before text/html in the Accept header. Browsers that happen to list text/markdown;q=0.5 after HTML still get HTML. Agents that ask for markdown first get markdown. Borrowed from vercel-labs/markdown-to-agents.

Pretty URLs and .md twins both work

For agents that don't speak the Accept header (yet), every page also has a .md twin:

• https://robotostudio.com/blog/some-post (HTML)
• https://robotostudio.com/blog/some-post.md (markdown)
• https://robotostudio.com/index.md (homepage markdown)

Both URLs route to the same handler. Whichever the agent prefers, it works. Belt and braces.

Discovery via Link headers

We advertise the markdown twin in an RFC 8288 Link header on every content page:

Link: </blog/some-post.md>; rel="alternate"; type="text/markdown"

The homepage adds a few more pointers:

Link: </llms.txt>; rel="describedby"; type="text/plain",
      </llms-full.txt>; rel="service-doc"; type="text/plain",
      </sitemap.xml>; rel="sitemap"; type="application/xml",
      </index.md>; rel="alternate"; type="text/markdown"

An agent crawling the homepage gets a structured map of where everything lives without parsing HTML.

llms.txt is just a sitemap with intent

/llms.txt is the convention proposed by Jeremy Howard. A markdown file at the root that tells agents what's on the site, in priority order, with both HTML and markdown URLs:

# Roboto Studio

> Our mission is to create the best editorial experiences on the web...

## For AI agents

Every URL on this site has a markdown twin: append `.md` to any page URL,
or send `Accept: text/markdown` against the same URL.

## Services
- [Sanity CMS](https://robotostudio.com/services/sanity) - [markdown](https://robotostudio.com/services/sanity.md)

## Blog Posts (Recent)
- [Latest post](https://robotostudio.com/blog/...) - [markdown](https://...md)

Think of it as a sitemap that talks. We also publish llms-full.txt, which is the entire content of every page concatenated. Useful for agents that want to embed the whole site once instead of re-fetching.

The markdown actually has to be clean

This is the part nobody talks about. If your content is MDX (markdown with React components inside), you can't just serve the file. Agents will choke on <InlineCTA> and <Newsletter> tags.

Our markdown-route.ts walks the MDX:

• Strips JSX components, keeping their text children
• Preserves fenced code blocks intact (delimiter-length-aware, so a 3-backtick example inside a 4-backtick fence survives)
• Preserves inline code spans (CommonMark equal-delimiter rule)
• Removes import/export statements
• Renders the frontmatter as a readable header (title, description, author, date)

Then it serves with:

Content-Type: text/markdown; charset=utf-8
Cache-Control: public, max-age=3600, s-maxage=86400
X-Robots-Tag: noindex, nofollow

The X-Robots-Tag matters. Without it, Google indexes both the HTML and .md versions and they compete for ranking. We don't want a .md URL outranking the canonical HTML page on a content set this size.

What to skip

A non-exhaustive list of things being sold as AEO that you can ignore:

• "AEO schema." There's no separate schema for AEO. Schema.org is the same standard SEO uses. Add FAQPage if you have FAQs, HowTo for genuine how-tos, Article for posts. That's it.
• AEO courses. The playbook is two paragraphs of HTTP and a sitemap. Don't pay for it.
• FAQ stuffing. Writing fake FAQs for the sake of FAQ schema is the new keyword stuffing. Google restricted FAQ rich results to authoritative government and health sites in 2023, so the rich-result payoff is gone for most sites anyway. Write FAQs because they're useful, not because they get cited.
• Worrying about which agent's leaderboard you're on. ChatGPT, Claude, Perplexity, and Gemini all use slightly different retrieval strategies. The signal that matters across all of them is the same one Google's used for fifteen years: do real people cite you?

Did it work?

It's been a few weeks. Two observations.

Agent traffic is up but small. Around 14 visits per month from Claude, Perplexity, and ChatGPT combined according to our PostHog data. The point of doing this work now isn't to ride a wave that hasn't arrived. It's to be ready for the slope.

The work itself was small. A few rewrites in next.config.ts, a route handler, a markdown converter, and an llms.txt generator. If your site is on Next.js 16 it's an afternoon's work. If it's on something else, the same pattern works on any framework that supports header-conditional routing.

That's the whole AEO playbook from someone who's actually done it. Skip the courses.