← Back to glossary

A text file in your site's root directory instructing search engine crawlers which areas to access or avoid. It can block crawling of admin sections, duplicate content, or resource-heavy areas that waste crawl budget. However, robots.txt is a suggestion, not a security measure—sensitive content needs proper access controls. We configure robots.txt during every build to guide crawlers efficiently through your site structure.