Question

What is robots.txt file?

Answer 1

Robots.txt is a text file within your website’s top-level directory that instructs search engines how to crawl your pages.

Answer 2

robots.txt is a text file that instructs web robots (search engine robots) about how to crawl pages on their website. It is used to manage crawler traffic to the website. 

Answer 3

Robots.txt is a text file. It is through this file. It gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.