TopFoundAEOBot

TopFoundAEOBot is the web fetcher run by TopFound. It reads a website's homepage and public files to build a free AEO (answer engine optimization) report, which shows how visible that site is in AI assistants such as ChatGPT, Gemini, Claude, and Perplexity.

TopFoundAEOBot is not a crawler. It does not discover links or revisit sites on a schedule. It fetches a site only when someone enters that site's address on https://www.topfound.io/aeo-report to request a report.

Each report reads at most eight URLs on the site:

• robots.txt: read first, to honor opt-outs.
• The page entered: usually the homepage.
• llms.txt and sitemap.xml: at the site root.
• Up to four inner pages: linked from that page, such as services, pricing, or about pages.

It sends GET requests only, never submits forms or logs in, follows at most five redirects per URL, gives up on a URL after ten seconds, and reads no more than 1.5 MB of any response.

What it reads is used only to build the report that was requested: the page title, description, headings, structured data, and visible text describe what the business offers, and robots.txt, llms.txt, and the sitemap are checked as technical signals.

The content is not used to train AI models, is not sold, and is not republished.

5.1 User agent

Mozilla/5.0 (compatible; TopFoundAEOBot/1.0; +https://www.topfound.io/bot)

5.2 Web Bot Auth signatures

Every request is signed with HTTP Message Signatures (RFC 9421) under the IETF Web Bot Auth drafts. Requests carry Signature, Signature-Input, and a Signature-Agent header pointing to our key directory, so a site can verify a request really came from us rather than trusting the user agent alone.

Public key directory: https://www.topfound.io/.well-known/http-message-signatures-directory

To stop TopFoundAEOBot from reading your site, add this group to your robots.txt:

User-agent: TopFoundAEOBot
Disallow: /

With that rule in place, TopFoundAEOBot reads robots.txt and nothing else. Because it only visits on request, a general "User-agent: *" rule does not apply to it; name it directly.

Questions, problems, or requests about TopFoundAEOBot can go to rushik@topfound.io.