Crawlability is the degree to which a website's pages and resources are accessible to automated crawlers — including search engine bots and AI crawlers such as OAI-SearchBot, GPTBot, and PerplexityBot. A crawlable website has no blocking directives that prevent important content from being indexed, correctly configured robots.txt and llms.txt files, no technical barriers such as JavaScript-only rendering for critical content, and a clear internal link structure that allows crawlers to discover all relevant pages. In the context of AI Search Readiness, crawlability determines whether AI systems can access the content they need to evaluate and cite a website.
