Technical guide
Make your website accessible to search and AI crawlers
Crawler access starts with successful public responses and consistent technical signals. The same visible content should be available to users, search crawlers, and answer engine crawlers.
Access checklist
- Return a successful response for every canonical public page.
- Allow search crawlers and OAI SearchBot in robots.txt when discovery is desired.
- Publish an XML sitemap using absolute canonical URLs.
- Add a self referencing canonical URL to every indexable page.
- Keep important facts in text instead of placing them only inside images.
- Use JSON LD structured data that accurately represents visible content.
- Link important guides and service pages from a crawlable navigation or hub.
What to monitor
Monitor response errors, accidental blocking, duplicate URLs, stale sitemaps, and changes in rendered content. IndexNow can notify participating search engines when important URLs are added, updated, or removed.
See what an agent can understand about your site.
Run the free public page audit, then use the evidence to prioritize the next improvement.
Audit your site