Robots.txt With Search Engine
Robots.txt with search engine behaves like an intermediator. You can tell to communicate or block communication through this with Google or any search engine.
In the robots.txt have one thing that most probably confuses everyone. That is noindex with Disallow. You can not come to SERP. if you use one of them. It is not recommended to use until you want to block any URL or Directory in SERP.
Robots.txt with a search engine.
Never you will come in the search by using both noindex and Disallow.
Note: If your URL is read in webmaster to do crawling then Disallow will reject to crawl and when URL is read by itself on the internet then noindex will tell to reject for indexing. Moreover, if you use only noindex then the webmaster will crawl the URL but will not index due to noindex meta tag. Also, Disallow can not read noindex because crawling is blocked.
Like and Share