Make Your Site Readable to AI: The AEO Technical Setup Guide

"Our website looks great — so why, when a buyer asks ChatGPT which Taiwanese suppliers make this part, are we nowhere in the answer?" It is the question HappyCXO Studio hears most often from export manufacturers. In most cases the content is not the problem. The problem is that AI cannot read the site at all: crawlers are blocked at robots.txt, the page only fills in after JavaScript runs, the entire spec sheet lives inside a JPEG, and there is not a single line of structured data anywhere. We covered the strategic case for GEO in the GEO-era B2B SEO survival guide, and the whole-site audit in the manufacturer website SEO checklist. This piece does one job only: walk through the AEO technical setup in the right order, and show you how to verify it actually worked. Everything here applies to any CMS — WordPress, Shopify, Wix, or a custom Next.js build.

Three things usually block AI from reading your site

Direct answer: when AI cannot read your site, it is almost always one of three technical gates — robots.txt does not explicitly allow AI crawlers, the page content only appears after client-side JavaScript runs, or there is no structured data, so AI has to guess who you are, what you sell, and whether to trust you. None of these require writing a single new word of content. Most teams can clear all three in half a day to two days.

The clearest way to think about it is a three-stage funnel: crawl → render → understand. If stage one blocks you, nothing downstream matters. If stage two yields an empty shell, the richest copy in the world is invisible. If stage three has no structured data, AI receives a pile of text but cannot assemble "this company is named X, based in Y, makes Z, written by this person, updated on this date" — it may still cite you, but it is far more likely to cite a competitor that spelled all of that out.

The funnel has a brutal property: it multiplies, it does not add. Any stage at zero makes the whole thing zero. So sequence matters. Fix robots.txt first (cheapest, fastest, highest impact), then rendering (most expensive, but it sets your ceiling), then backfill structured data (moderate effort, and you can do it page by page). Plenty of companies get this backwards — three months of schema work while a stray "Disallow: /" from the staging site still sits in production robots.txt.

Worth busting one very common myth: most owners assume AEO means "publish more articles," so step one is hiring a blog writer. The opposite is true. Every article published before the technical foundation exists is written on paper the machines cannot see. Content matters enormously, but it is a multiplier, not a base. When the base is zero, multiplying changes nothing. Spend two days on the foundation first, and the identical content budget produces a completely different outcome.

Set expectations on timing, too. Technical changes do not take effect instantly. Google generally caches robots.txt for up to 24 hours, and longer when it hits timeouts or 5xx errors — see the Google robots.txt documentation. Other AI crawlers publish very little about their revisit cadence. So the sane expectation after a change is to watch in weeks, not refresh in hours.

One organizational reality that quietly kills these projects: the work spans marketing, IT, and an outside web vendor. Marketing knows it matters but cannot edit anything. IT can edit but was never told why. The vendor has no line item for it in the contract. The most effective move is to hand the checklist in section seven directly to your vendor as acceptance criteria, rather than asking them to "do AEO," which nobody can quote a price for.

robots.txt: explicitly welcome GPTBot, ClaudeBot, PerplexityBot

Direct answer: in the robots.txt at your domain root, use named User-agent blocks to Allow each AI crawler you want, instead of relying on a single User-agent: block — most crawlers obey only the group that matches their own name most specifically. This is the highest-return ten minutes in the entire setup.

Some background most people miss: the Robots Exclusion Protocol ran informally for nearly thirty years and was only standardized by the IETF as RFC 9309 in 2022. Even standardized, it remains voluntary — it stops nothing malicious, but every major AI vendor publicly commits to honoring it. Which means what you write in robots.txt genuinely governs the AI systems you actually want citing you.

The crawlers worth naming explicitly fall into three purposes:

• OpenAI: GPTBot (model training), OAI-SearchBot (ChatGPT search indexing), ChatGPT-User (live fetch triggered by a user question). Three different purposes you can decide on separately — see the official OpenAI bots documentation.
• Anthropic: ClaudeBot, Claude-User, and Claude-SearchBot, documented in Anthropic crawler guidance.
• Perplexity: PerplexityBot for indexing and Perplexity-User for user-triggered fetches, per the Perplexity bots documentation.
• Google: Googlebot handles ordinary search, while Google-Extended is a separate control governing whether your content feeds generative products like Gemini. It does not affect Search ranking — details in the Google crawlers overview.
• Others worth allowing: Bingbot (Microsoft Copilot draws on the Bing index), Applebot and Applebot-Extended, Amazonbot, meta-externalagent, and CCBot (Common Crawl, the training source behind many open models).

A minimum viable file you can copy:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /studio

Sitemap: https://www.example.com/sitemap.xml

The roster changes, so do not treat one pass as permanent. The open-source ai.robots.txt project maintains a continuously updated list worth diffing against quarterly. On the HappyCXO Studio site itself we name 13 AI crawlers with explicit allow rules and block only the CMS admin path.

Four failure modes we have personally found on live client sites. One: robots.txt allows the crawler, but the Cloudflare or WAF "block AI bots" toggle is switched on — the firewall wins, and your edit did nothing. Two: the staging "Disallow: /" shipped to production, closing the entire site. Three: robots.txt returning 404 is harmless, but returning 5xx persistently is not — after roughly a month of continuous errors Google may fall back to treating the site as disallowed. Four: rules written under "User-agent: " on the assumption that named crawlers inherit them; they do not, because a crawler reads only the most specific matching group.

Whether to let training crawlers in is a genuine business decision for the owner, not a technical one. The line is clean: retrieval crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) are the pipe through which you get cited, and there is almost no reason to block them; training crawlers (GPTBot, Google-Extended, CCBot) mean "use my content to train a model." For export manufacturers our default recommendation is to allow both. Your spec tables and application notes are not creative works with meaningful licensing value, so the marginal loss from training is tiny, while the value of a model reliably associating your brand and part numbers with a product category is large. What actually needs protecting — pricing, customer lists, CAD source files — should never have been public on the website in the first place.

Verify by impersonating the crawler:

curl -s -A "GPTBot" -o /dev/null -w "%{http_code}\n" https://www.example.com/
curl -s https://www.example.com/robots.txt | head -40

A 200 means you are through. A 403 nearly always means a CDN or WAF is blocking, and the fix belongs in the firewall settings, not in robots.txt.

What llms.txt is, and what belongs in it

Direct answer: llms.txt is a Markdown file served at your domain root (/llms.txt) that tells language models, in a structured list, who this site is, which pages matter most, and where to get the full text. It was proposed by Jeremy Howard in September 2024, with the spec published at llmstxt.org, to address a real constraint: model context is limited, and raw HTML is full of navigation, ads, and script noise.

The format is deliberately minimal — an H1 with the site or company name, a blockquote summarizing in one or two sentences who you are and what you do, several H2 sections each holding a list of "link plus one-line description," and an optional section for secondary resources. The whole file usually fits in under a hundred lines.

# Example Precision Co., Ltd.

> Taiwan-based manufacturer of CNC-machined components for
> medical and semiconductor equipment. Founded 1998. ISO 13485.

## Core pages
- [Capabilities](https://www.example.com/en/capabilities): equipment list, tolerances, materials
- [Quality](https://www.example.com/en/quality): certifications, inspection process
- [Contact](https://www.example.com/en/contact): RFQ form, lead times

## Guides
- [Material selection guide](https://www.example.com/en/blog/material-guide): stainless vs titanium

## Optional
- [Full text export](https://www.example.com/llms-full.txt)

In practice you ship two files. /llms.txt is the curated map — only the pages you most want cited, ordered by importance and freshness. /llms-full.txt is a full-text export that concatenates the plain text of your main pages so a model can ingest everything in one request. The HappyCXO Studio site publishes both, and orders /llms.txt so that stable entity facts come first and time-sensitive content follows.

An honest caveat, because the internet is full of posts treating this as magic: no major AI vendor has publicly committed to reading llms.txt, and Google has said plainly that it does not use it. It is a community proposal, not a search-engine standard. So why bother? Three reasons: it costs about an hour to produce, it is genuinely useful to humans and internal tools (sales reps use it as a link directory), and if any vendor does start reading it, you are already listed. Treat it as cheap insurance, not as foundation — the foundation is always robots.txt, server-side rendering, and structured data.

Maintenance discipline matters more than the initial file. A stale llms.txt is worse than none: it points models at discontinued product lines and moved URLs, actively manufacturing wrong answers. The practical fix is to write "update llms.txt" into your publishing checklist, adding a line whenever you ship a service page or significant article. Better still, if your site is dynamically generated, render llms.txt from the CMS so it can never go stale.

One last common error: renaming sitemap.xml and calling it llms.txt. The two have opposite goals. A sitemap is an exhaustive URL list for crawlers, optimized for coverage. llms.txt is a curated guide for models, optimized for signal-to-noise. Dumping three hundred URLs into it says nothing at all. Twenty to forty links, each with a one-line description, outperforms a complete but undifferentiated list every time.

Structured data: how to mark up Organization, Article, and FAQPage

Direct answer: three layers of JSON-LD is enough — one site-wide Organization block (who you are), one Article or BlogPosting block per content page (what this page covers, who wrote it, when it was updated), and a FAQPage block wherever you have a real Q&A section. Google explicitly recommends JSON-LD over microdata scattered through your HTML; see the Google structured data documentation.

Organization is the most important block and the one most often broken. It belongs in the shared site layout, appearing exactly once, with name, url, logo, description, address, contactPoint, and — most critically — sameAs, listing your LinkedIn, Facebook, YouTube, and even Alibaba storefront URLs. sameAs is the heart of entity disambiguation: it tells AI that these scattered accounts are one company, letting signals from across the web resolve to a single entity. The full property list lives on the schema.org Organization type page.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Precision Co., Ltd.",
  "url": "https://www.example.com",
  "logo": "https://www.example.com/logo.png",
  "description": "CNC-machined components for medical and semiconductor equipment.",
  "sameAs": [
    "https://www.linkedin.com/company/example-precision",
    "https://www.youtube.com/@exampleprecision"
  ],
  "contactPoint": {
    "@type": "ContactPoint",
    "contactType": "sales",
    "email": "sales@example.com",
    "availableLanguage": ["en", "zh-Hant"]
  }
}

Article / BlogPosting goes on every content page, with headline, image, datePublished, dateModified, author, and publisher. Two practical notes. First, use a Person type for author with a real name and title — it supports E-E-A-T far better than attributing everything to the company. Second, dateModified has to actually move. Countless sites hardcode it equal to the publish date, which tells AI directly that the page has never been maintained. Freshness is a real selection signal for generative engines, and if a 2023 technical article is still accurate in 2026, the right move is to review it and let dateModified reflect the true date.

FAQPage comes with a detail worth stating plainly: since August 2023, Google has sharply narrowed FAQ rich result display, in practice reserving it for authoritative government and health sites (official Google announcement). Many people concluded that FAQPage markup is now pointless. That conclusion is wrong. What Google withdrew was the right to display it as a rich result on the SERP, not the data itself. Generative engines parse the structured data actually present on the page, and a clean question-and-answer pairing remains the single most extractable format there is. Phrase the questions the way buyers really type them, keep each answer self-contained, ship four to eight of them, and it stays one of the best-return markup decisions available.

Manufacturers should consider a few more types: Product or ProductGroup (turning specs into machine-readable fields), BreadcrumbList (so AI understands site hierarchy), ItemList (for product listing pages), and DefinedTermSet (glossaries — the HappyCXO marketing glossary uses exactly this type, and its citation value is surprisingly high because models love citing definitions). The schema.org vocabulary spans hundreds of types; do not deploy them all. Pick the ones Google supports that also mean something to a buyer, using something like the Moz structured data guide as a filter.

The four most common markup errors. One: marking up content that is not visible on the page — explicitly prohibited by Google structured data policy and grounds for manual action. Two: an Organization block on every page with contradictory data (old address here, old phone number there), so no consistent entity can be assembled. Three: copying JSON-LD from another site and forgetting to change the URL and company name, effectively marking up a competitor. Four: marking the same fact in both microdata and JSON-LD so the two disagree. None of these require engineering skill to catch — the how is in the final section.

Server-side rendering vs pure JS: why crawlers see a blank page

Direct answer: if your product specs are fetched from an API by JavaScript after the page loads, then any AI crawler that does not execute JavaScript sees an empty shell. Googlebot does render JavaScript, but only after queueing the page for a second pass; most AI crawlers do not render at all. This is the most expensive layer of the funnel, and the one that sets your ceiling.

Google spells the mechanism out in its JavaScript SEO basics documentation: crawling and rendering are separate stages with a queue between them. For Google that queue is a delay. For the overwhelming majority of AI crawlers it is total blindness — they take the raw HTML and leave without waiting for React to mount.

You can self-diagnose in thirty seconds with no tools at all:

curl -s https://www.example.com/products/model-a | grep -c "Part No"
curl -s https://www.example.com/products/model-a | wc -c

If the grep returns 0, or the entire HTML response is only two or three kilobytes (typically an empty div plus a few script tags), your content does not exist as far as crawlers are concerned. This test beats any SEO tool, because it is literally what the crawler receives.

Fixes, cheapest first. The ideal is SSR or SSG — Next.js, Nuxt, and Astro emit complete HTML by default, and platforms like WordPress, Shopify, and Wix are server-rendered out of the box, so they are usually fine. The highest-risk case is a custom React or Vue single-page app, especially one built with create-react-app a few years back. If rebuilding is unrealistic, the pragmatic middle path is prerendering your critical pages — home, product, capabilities — so crawlers get static HTML while the rest of the app stays as-is. We work through the architecture choices in the export website build guide.

Several variants cause the same invisibility and get overlooked. Client-side pagination and infinite scroll: nothing past page one is ever crawled unless you also expose real paginated URLs. Tabbed content: fine if all tabs are in the HTML and merely hidden with CSS, broken if switching tabs triggers a fetch. Modal spec sheets: content that loads on click does not exist. The rule never changes — is it in the HTML source? If not, it is not there.

There is a related but distinct issue: server response time. Crawlers budget how much they fetch per site, and a slow server directly reduces how many pages get crawled per visit. If your product pages have a two-to-three-second time to first byte, even complete HTML may leave deep pages perpetually uncrawled. This bites hardest on manufacturer sites that put a thousand-page catalog online.

Specs buried in images do not exist to AI

Direct answer: spec tables, certification numbers, and dimension drawings that exist only inside a JPEG or a scanned PDF are invisible to text-based crawlers. Moving specs out of images and into HTML tables is the single highest-return action on this entire list. It needs no architecture change and no engineer — marketing or a sales assistant can do it.

Three "content black holes" show up in nearly every manufacturer site audit we run. First, the entire spec table rendered as one image — thirty part numbers and twelve columns crammed into a 1200-pixel-wide JPEG that humans have to zoom into and crawlers get nothing from. Second, catalogs offered only as PDF downloads, often scanned with no text layer, so even the PDF itself is unsearchable. Third, certificates as scan images only, turning ISO numbers, validity dates, and issuing bodies into pixels.

The fixes are straightforward:

• Move specs into an HTML table: one row per part number, one column per parameter (dimension, material, tolerance, torque, certification). Text that can be selected is text that can be cited.
• Keep the image but write a real alt: alt text describes the content for people and machines who cannot see it; it is not a keyword bin. "Stainless hex bolt M8 x 40, DIN 933, A2-70" is good alt text. "screw bolt taiwan manufacturer factory supplier" is not.
• Give PDFs a text layer: export natively instead of scanning, and better still, restate the key figures in an HTML page with the PDF as a supplementary download rather than the only carrier.
• Name files meaningfully: model-a-spec-sheet.pdf beats 20240612_final_v3.pdf.

There is a higher-order issue with outsized impact: part-number consistency. When the same product is "A-100" on the website, "A100" in the catalog, and "Model A 100 series" on your Alibaba storefront, humans shrug and AI sees three different things. Entities can only be assembled if naming is consistent — the same logic as sameAs in the Organization block above. Your job is to actively connect the dots for AI, not to hope it guesses correctly.

One second-order effect people rarely anticipate: moving specs from images into HTML helps your sales team as much as it helps AI. Once specs are text, site search finds them, reps paste them straight into quote emails, buyers Ctrl+F for a part number on the page, and translating to a second language no longer means retyping the whole table. The most immediate effect we see is a measurable drop in inbound questions of the "do you have this size?" variety — questions that were always answered on the image, but that nobody could read. That is an operational win from a technical change, and it pays back before any AI citation arrives.

Technical checklist: right way vs common mistake

Direct answer: the table below compresses everything above into twelve verifiable items. The intended use is to hand it to your web vendor as acceptance criteria, or run it as a quarterly internal self-check. Every row can be verified in ten minutes with free tools.

If you cannot do all twelve, the priority order is: robots.txt and WAF (rows one through three) → rendering (rows five and six) → specs into HTML (row seven) → Organization and Article schema (rows nine and ten) → llms.txt and the rest. The reason goes back to the multiplying funnel: when the early rows are zero, nothing later multiplies into anything.

A cost reframe worth stating: not one of these twelve items requires rebuilding the website. Most are config-file or template-level changes, adding up to roughly one to three engineering days, plus the data-entry effort of converting spec sheets to HTML (typically two to five days of a marketing or sales assistant, depending on how many part numbers you carry). Against six-figure quotes for a full site rebuild, this is a different order of magnitude entirely — and it is a prerequisite for a rebuild, not a substitute. When you do rebuild, these twelve items belong verbatim in the specification you hand the vendor.

How to verify you got it right

Direct answer: four free tools and about fifteen minutes cover a full verification round — curl impersonating a crawler, Google Rich Results Test for structured data, Schema Markup Validator for the complete JSON-LD, and Search Console URL Inspection for the HTML Google actually received. Add server logs and you have the only first-party proof that AI crawlers really reach you.

Step one: curl as a crawler. This is the most honest test, because it is exactly the raw response a crawler receives:

curl -s -A "GPTBot" -o /dev/null -w "%{http_code}\n" https://www.example.com/
curl -s -A "ClaudeBot" https://www.example.com/products/model-a | wc -c
curl -s https://www.example.com/llms.txt | head -20
curl -s https://www.example.com/ | grep -c "application/ld+json"

Those four lines check, in order: whether AI crawlers get a 200, whether the HTML carries real content, whether llms.txt exists, and how many JSON-LD blocks the page ships. Whichever line comes back wrong tells you which layer is broken.

Step two: Google Rich Results Test. Paste a URL and it reports which structured data types Google recognizes and which required fields are missing. Note it only validates Google-supported types, so a type it does not report is not necessarily broken — it just is not used for rich results.

Step three: Schema Markup Validator. This fills the gap left by step two, validating the full schema.org vocabulary including types Google never renders. Running both is what makes structured data verification complete.

Step four: Search Console URL Inspection. Enter the URL, test the live URL, then view the crawled page HTML. What you see there is the HTML Google actually got after rendering, and it is the most authoritative evidence for diagnosing JavaScript problems. If content appears there but not in curl, you are dependent on rendering: Google can absorb that, AI crawlers cannot.

Step five, the least-practiced and most valuable: read your server logs or CDN analytics. Filter for requests whose User-agent contains GPTBot, ClaudeBot, or PerplexityBot, and check whether they actually arrived, which pages they fetched, and what status codes they received. This is the only first-party data in the whole process. Steps one through four verify "it should work"; the logs verify "it did." Weeks of logs with zero AI crawler hits sends you straight back to the WAF.

Step six, output-side verification: just ask the AI. Use the questions buyers really type — not your brand name — in ChatGPT, Perplexity, and Google AI Overviews, and record whether you are mentioned and which page gets cited. It is slow, noisy, and never identical twice, but it is the only measure of the actual objective. We break the tracking method down in the GEO-era B2B SEO survival guide, and the growth trajectory of generative AI usage is tracked on the Statista generative AI topic page.

Two measurement traps to avoid. First, record a baseline before you touch anything — run the full verification once and save the output, or three months later you will be unable to prove anything happened. Second, do not expect linear growth. There is no Search Console for AI citations; samples are small and every generation is stochastic, so a single result means nothing. Watch four-to-eight-week trends, and cross-check two datasets: whether logs show crawlers fetching, and whether sampled questions mention you. Technical health usually improves within weeks (crawl frequency up, fetch errors down); citation rate moves much more slowly.

A sensible cadence: run all six steps on day one after launch, again on day seven to confirm crawlers returned, then a quick check monthly and a full check quarterly. Always rerun after a redesign, a CDN change, a CMS migration, or a new firewall rule — more than half the technical regressions we find were introduced by an unrelated project, not by never having done the work.

Once the technical foundation is in place, content is the next step: titles built from questions buyers actually ask, and engineering knowledge written in passages AI is willing to quote. More on the same theme lives on the website SEO topic hub. If you would rather have someone run these twelve checks on your site before deciding what to change, see our website and SEO service, or simply talk to us about where your site stands today.

Further reading: Product Page AEO: Make AI Read Your Spec Sheets · AI Search Engines Compared: Where Exporters Should Start