Block Content Access For Search Engine

Blocking content access in Google and other search engines is not easy to understand who are beginners. Retrieving web pages on your site by the crawler is monitor through the robots.txt file.

Allow: /URL is used when you have one directory is blocked but you one of URL to access by search engine crawler.

But Blocking URL comes under the Disallow: /

Disallow: /search/label/SEO

In the above code label, SEO is blocked for indexing and crawling.

My robots.txt file

User-agent: Mediapartners-Google 

Disallow: User-agent: * 

Disallow: /search Allow: / 

Sitemap: https://www.mssalotra.com/sitemap.xml

Read more about robots.txt file with Google article.

You can block URL with NoIndex also. But the crawler will visit on URL and then understand the NoIndex. Finally, your URL will not come in search.

You can also say "NoSnippet" to search crawler and with this, you can block your content access in SERP.

For any help ask a question.

Stay Happy

Like and Share

Popular posts from this blog

SEO Vocabulary - The SEO Dictionary

What IS SEO: Optimizing The Content For Search Engine

Mismatched URL - Title and Description Gap

SEO True False Statements

SEO Redirect: How it help us?

High Quality Website

Multiple Sitemaps: How To Submit A Sitemap Index

Canonical Tag - Mobile or Desktop Version Teller

HTTP Status Code IN SEO

Log Files On Server and IN SEO

Navigate the blog from the labels. This is for your help to easily navigate the articles according to the category. Moreover, on each post, you will find the internal links that will help you to navigate the related articles.
Protected by Copyscape

Popular posts from this blog

Log Files On Server and IN SEO

What IS SEO: Optimizing The Content For Search Engine

HTTP Status Code IN SEO