Web Design / Items

Blocking Search Engines Crawlers using Robots.txt File

Get Feed

Description

You require a robots.txt file only if you don’t want searchbots to crawl and therefore index certain pages. i.e you want to block particular pages of your website from coming in search results of search engines. If you want your entire site to be indexed, you don’t need robots.txt file in the first place.

Robots.txt is simply an ASCII text file placed at the root of your domain.

For example, http://www.domainname.com/robots.txt.

Check the link below for more information..

Original URL: http://www.kittuk.com/2009/08/blocking-search-engines-crawlers-using-robotstxt-file/

http://www.kittuk.com/2009/08/blocking-search-engines-crawlers-using-robotstxt-file/

Comments

Public Comments
- kittu k
  
  4 months ago
  
  Make the search engines see your website the way you want it to..
  
  Make the search engines see your website the way you want it to..
  Commented in: Internet Marketing, Web for Social Change, SEO, Web Industry Trends, Web Design, Online marketing and advertising, Web Design and Development, Social Media, SEO Resources, Search and Discovery, SEO, Organic Search Engine Optimization, Search Engine News, Online Marketing, Microblogging, Social Media Marketing, Marketing Guide, SEO, SEO, Search Engine Optimization, Web Development, Web Development, Web 3.0 - Semantic Web, Technology Trends, Internet Marketing, Google
- Moh Ata
  
  4 months ago
  
  But why would someone need to use it ??
  Anyone who publishes something online will want it to be found, right ??
  
  But why would someone need to use it ?? <br />Anyone who publishes something online will want it to be found, right ??
  Commented in: Web 3.0 - Semantic Web
- seonext
  
  4 months ago
  
  Great info...Like the post, Good to know information for business as well as SEO's...Thanks for sharing.
  
  To Moh Ata - by making this file we ask search engine to not to flow the link juice to irrelavant pages like - our legal terms and conditions, our policies and may be your internal confidential details like contracts which you won't like them to be showcased on searchengines...Hope this will clear your doubts
  
  Great info...Like the post, Good to know information for business as well as SEO's...Thanks for sharing. <br /><br />To Moh Ata - by making this file we ask search engine to not to flow the link juice to irrelavant pages like - our legal terms and conditions, our policies and may be your internal confidential details like contracts which you won't like them to be showcased on searchengines...Hope this will clear your doubts
  Commented in: SEO
Add a Comment

Report This

Twine is about discovering, collecting and sharing the content that interests you. Learn More

Join Twine

Stats

26 Twines
3 Comments

First Posted By

kittu k

First Comment By

kittu k

Who's Interested In This?

kittu k added to Internet Marketing, Google, Web Design, Web Industry Trends, Internet Marketing, Web Development, Search Engine Optimization, Web Development, SEO, Social Media, Marketing Guide, Technology Trends, Online Marketing, Web 3.0 - Semantic Web, Search Engine News, Search and Discovery, SEO, SEO, Microblogging, SEO, Organic Search Engine Optimization, Online marketing and advertising, Web Design and Development, SEO Resources, Social Media Marketing, Web for Social Change 4 months ago

HTML Tag	Attributes
<a href="URL"><a href="#">link</a></a>	'href', 'target', 'rel'
<strong>strong</strong>
<b>bold</b>
<blockquote> blockquote </blockquote>
<em>emphasis</em>
<i>italic</i>
<img src="URL">	'src', 'width', 'height', 'alt', 'title'
<u>underlined</u>
<s>~~strike~~</s>
<del>~~deleted~~</del>

Web Design / Items

Blocking Search Engines Crawlers using Robots.txt File

Comments

Public Comments

kittu k

4 months ago

Moh Ata

4 months ago

seonext

4 months ago

Stats

First Posted By

First Comment By

Tags

Community Tags

Who's Interested In This?

Web Design / Items

Blocking Search Engines Crawlers using Robots.txt File

Comments

Public Comments

kittu k 4 months ago

Moh Ata 4 months ago

seonext 4 months ago

Stats

First Posted By

First Comment By

Share and Collect

Share With

Tags

Community Tags

Who's Interested In This?

kittu k

4 months ago

Moh Ata

4 months ago

seonext

4 months ago