robots.txt

Text file used by websites to instruct web crawlers which parts of the site should not be indexed or accessed.

"Disallow: " directories may be of interest for us

source: https://moz.com/learn/seo/robotstxt