Access
Review robots.txt rules for the named crawler and the HTTP responses observed by the audit. Check server logs to investigate the real crawler’s activity.
Publisher website SEO and crawler audit
Keep your articles accessible. Choose your crawler policy.
Review article pages, indexing signals, and crawler access. Separate search access from training policy, then check what your website actually returns.
Category scores and selected findings free. No email required.
Detailed findings and export files: $10 once.
Different uses. Separate rules.
- GPTBot
- Model training
- OAI-SearchBot
- Search in ChatGPT
- Googlebot
- Google Search
Choose your policy by purpose. Verify the provider’s current documentation.
Make an explicit crawler policy
Check the reported rules for search and training separately. A publishing policy should be deliberate.
Where to look
Access, indexing, and citation are separate questions.
The scan shows its actual coverage and applicable checks. Review the evidence before deciding which changes to make.
Indexing
Inspect canonical signals, internal links, and indexing directives. Use Search Console to review how Google processed specific URLs.
Citation
Publish useful reporting with clear sources, dates, and authorship. Website heuristics can prompt editorial review; they do not measure whether an AI answer cites an article.
Before you scan
Questions from publishers.
What the audit can tell you, and what needs a separate review.
Do my robots.txt settings actually control what AI crawlers can access?
Robots.txt communicates crawling rules to compliant crawlers. It is not authentication. A server can reject an otherwise permitted request, and a crawler can use different infrastructure from an audit probe. Review both the rule and the response, then investigate logs before assigning a cause.
Is llms.txt necessary for a blog or media site?
It is an optional proposal, not a requirement for Google Search or its AI features. Google says no special AI text files or schema are required. Do not treat the file as an access control or assume that adding it changes indexing or citation priority.
How do I get my content cited by ChatGPT or Perplexity?
There is no guaranteed procedure. Make useful original content accessible under the relevant search policy and identify sources clearly. OpenAI documents OAI-SearchBot separately from GPTBot; allowing model training is not required to allow search access.
Why is an AI overview not available for my content on Google?
Eligibility and selection differ. Google says a supporting page must be indexed and eligible to show a search snippet, with no additional technical requirements. An eligible page is still not guaranteed to appear in an AI Overview for a given query.
What makes a page quotable to an AI answer engine?
Clear, specific statements with attributable sources are useful to readers. AuditLamp’s answer-readiness checks are heuristics for reviewing that writing. Failing a homepage heuristic does not establish that the rest of a publication is uncitable.
Check the source.
Start with your website
Scan my publication.
Review the free preview. Choose Full Report when you need the detailed findings and export files.
Compare the plansCategory scores and selected findings free. No email required.
Detailed findings and export files: $10 once.