Controlling how search engines access and index your website

<span class="byline-author">Posted by Dan Crow, Product Manager</span><br /><br />I'm often asked about how Google and search engines work. One key question is: how does Google know what parts of a website the site owner wants to have show up in search results? Can publishers specify that some parts of the site should be private and non-searchable? The good news is that those who publish on the web have a lot of control over which pages should appear in search results.<br /><br />The key is a simple file called <span style="font-family:Courier New;">robots.txt</span> that has been an industry standard for many years. It lets a site owner control how search engines access their web site. With <span style="font-family:Courier New;">robots.txt</span> you can control access at multiple levels -- the entire site, through individual directories, pages of a specific type, down to individual pages. Effective use of <span style="font-family:Courier New;">robots.txt</span> gives you a lot of control over how your site is searched, but its not always obvious how to achieve exactly what you want. This is the first of a series of posts on how to use <span style="font-family:Courier New;">robots.txt</span> to control access to your content.<br /><br /><span style="font-weight: bold;">What does robots.txt do?</span><br /><blockquote><span style="font-style: italic;">The web is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is. I mean, you might think it's a lot of work maintaining your website, but that's just peanuts to the whole web.</span> (with profound apologies to Douglas Adams)</blockquote>Search engines like Google read through all this information and create an index of it. The index allows a search engine to take a query from users and show all the pages on the web that match it.<br /><br />In order to do this Google has a set of computers that continually crawl the web. They have a list of all the websites that Google knows about and read all the pages on each of those sites. Together these machines are known as the Googlebot. In general you want Googlebot to access your site so your web pages can be found by people searching on Google.<br /><br />However, you may have a few pages on your site you don't want in Google's index. For example, you might have a directory that contains internal logs, or you may have news articles that require payment to access. You can exclude pages from Google's crawler by creating a text file called <span style="font-family:Courier New;">robots.txt</span> and placing it in the root directory. The <span style="font-family:Courier New;">robots.txt</span> file contains a list of the pages that search engines shouldn't access. Creating a <span style="font-family:Courier New;">robots.txt</span> is straightforward and it allows you a sophisticated level of control over how search engines can access your web site.<br /><br /><span style="font-weight: bold;">Fine-grained control</span><br />In addition to the <span style="font-family:Courier New;">robots.txt</span> file -- which allows you to concisely specify instructions for a large number of files on your web site -- you can use the robots <span style="font-family:Courier New;">META</span> tag for fine-grain control over individual pages on your site. To implement this, simply add specific <span style="font-family:Courier New;">META</span> tags to HTML pages to control how each individual page is indexed. Together, <span style="font-family:Courier New;">robots.txt</span> and <span style="font-family:Courier New;">META</span> tags give you the flexibility to express complex access policies relatively easily.<br /><br /><span style="font-weight: bold;">A simple example</span><br />Here is a simple example of a <span style="font-family:Courier New;">robots.txt</span> file.<br /><blockquote><pre>User-Agent: Googlebot<br />Disallow: /logs/</pre></blockquote>The <span style="font-family:Courier New;">User-Agent</span> line specifies that the next section is a set of instructions just for the Googlebot. All the major search engines read and obey the instructions you put in <span style="font-family:Courier New;">robots.txt</span>, and you can specify different rules for different search engines if you want to. The <span style="font-family:Courier New;">Disallow</span> line tells Googlebot not to access files in the <span style="font-family:Courier New;">logs </span> sub-directory of your site. The contents of the pages you put into the <span style="font-family:Courier New;">logs</span> directory will not show up in Google search results.<br /><br /><span style="font-weight: bold;">Preventing access to a file</span><br />If you have a news article on your site that is only accessible by registered users, you'll want it excluded from Google's results. To do this, simply add a <span style="font-family:Courier New;">META</span> tag into the html file, so it starts something like:<blockquote><pre><html><br /><head><br /><meta name="googlebot" content="noindex"><br />...</pre></blockquote>This stops Google from indexing this file. <span style="font-family:Courier New;">META</span> tags are particularly useful if you have permission to edit the individual files but not the site-wide <span style="font-family:Courier New;">robots.txt</span>. They also allow you to specify complex access-control policies on a page-by-page basis.<br /><br /><span style="font-weight: bold;">Learn more</span><br />You can find out more about robots.txt at <a href="http://www.robotstxt.org/" target="_blank">http://www.robotstxt.org</a> and at <a href="http://www.google.com/support/webmasters/bin/topic.py?topic=8843" title="Google's Webmaster help center,">Google's Webmaster help center,</a><span class="q"> which contains lots of helpful information, including:<span style="font-weight: bold;"><br /></span></span> <p></p> <p> <span class="q"></span> </p> <ul><li> <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=40362">How to create a robots.txt file</a> </li><li> <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=40364">Descriptions of each user-agent that Google uses</a> </li><li> <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=40367">How to use pattern matching</a> </li><li> <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=40368">How often we recrawl your robots.txt file</a> </li></ul> We've also done several posts in our <a href="http://googlewebmastercentral.blogspot.com/" title="webmaster blog">webmaster blog</a> about robots.txt that you may find useful, such as:<br /><ul><li> <a href="http://sitemaps.blogspot.com/2006/02/using-robotstxt-file.html" title="Using robots.txt files">Using robots.txt files</a> </li><li> <a href="http://googlewebmastercentral.blogspot.com/2006/08/all-about-googlebot.html" title="All about Googlebot">All about Googlebot</a> </li></ul> <span class="q"></span> <div style="direction: ltr;"> There is also a useful list of the bots used by the major search engines: <a href="http://www.robotstxt.org/wc/active/html/index.html" target="_blank">http://www.robotstxt.org/wc<wbr>/active/html/index.html</a> </div><br /><span style="font-weight: bold;">Next time...</span><br />Coming soon: a post detailing the use of robots and metatags, and another on specific examples for common cases.<br /><br /><span style="font-weight: bold; font-style: italic;">Update: </span>Added a sentence to paragraph 9 on access-control policies.

Controlling how search engines access and index your website

Posted by Dan Crow, Product Manager I'm often asked about how Google and search engines work. One key question is: how does Google know ...

Đọc thêm »

New life for network equipment

<span class="byline-author">Posted by Urs Hoelzle, Senior VP, Operations</span><br /><br />We take the Internet for granted throughout the U.S. and the westernized world; these days we expect to be "always on." But in large parts of the globe, this isn't yet so.<br /><br />The <a href="http://www.nsrc.org/" title="Network Startup Resource Center">Network Startup Resource Center</a> (NSRC), a non-profit organization based at the <a href="http://www.uoregon.edu/" title="University of Oregon">University of Oregon</a>, is working to change that, by bringing potential recipients of computing and networking hardware together with potential donors. By coordinating these donations, the NSRC undertakes projects to connect Internet-neglected areas in Asia, Africa, Latin America, the Middle East, the Caribbean and elsewhere--and they've done so since 1992. (Read about their <a href="http://www.nsrc.org/about.html" title="NSRC About">accomplishments</a>.)<br /><br />Over the years they've found that the most difficult--and most useful--equipment to obtain <a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0gLX2ueqEyS11QmrSMfNonkogId-gZnHCTik8JtTWgcEh7iIjR37fvzM8WIVq5qw1hdCmjq7Wiltth_1EV3x7NeANOt2ZbKlQTSVs-kPFBZiFsRxIln1DjU3Gb7U3Dfxcq4QPuAaXmFs/s1600-h/Picture1.jpg"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5020774059168588722" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0gLX2ueqEyS11QmrSMfNonkogId-gZnHCTik8JtTWgcEh7iIjR37fvzM8WIVq5qw1hdCmjq7Wiltth_1EV3x7NeANOt2ZbKlQTSVs-kPFBZiFsRxIln1DjU3Gb7U3Dfxcq4QPuAaXmFs/s200/Picture1.jpg" style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer;" /></a><span style="font-size:100%;">is routers, switches and wireless equipment. In many cases, a single, relatively inexpensive piece of hardware is the critical factor preventing a network from becoming operational, but in developing areas, networking hardware can be difficult or prohibitively expensive to obtain. We're pleased to <a href="http://www.nsrc.org/sponsors/" title="support">support</a> the NSRC by donating our decommissioned networking hardware. Thus far, we've heard that some of our retired switches have been sent to universities across Africa and in Guyana and Thailand. More will be shipping out in the coming months to universities and research institutes in, among others, the Democratic Republic of the Congo, Ecuador, Afghanistan and Vietnam.<br /><br />If your company would like to give financial assistance or hardware, or if you want to look into volunteering, please <a href="http://www.nsrc.org/#contact" title="get in touch">get in touch</a>. It will truly help connect the world. </span>

New life for network equipment

Posted by Urs Hoelzle, Senior VP, Operations We take the Internet for granted throughout the U.S. and the westernized world; these days we e...

Đọc thêm »

Show us your university campus in 3D

<span class="byline-author">Posted by Allyson McDuffie, Google SketchUp Education Program Coordinator</span><br /><br />Today the <a href="http://contest.sketchup.com/entry.php">Build Your Campus in 3D Competition</a> begins. This spring, you and your (presumably equally artistic) friends can honor your campus turf as you hone your 3D design skills just by modeling your school's campus buildings in <a href="http://www.sketchup.com/">Google SketchUp</a>, geo-reference them in <a href="http://earth.google.com/">Google Earth</a>, and submit them through the <a href="http://contest.sketchup.com/entry.php">competition website</a> to earn lasting online glory. And the winners get a visit to Google, all expenses paid.<br /><br />You're eligible if you're a higher education student in the U. S. or Canada. You can team up with other students, or take the project on yourself. (To do the best work possible, we suggest you have a faculty advisor.) The deadline for entries is June 1, and the winning entries will be posted to the <a href="http://sketchup.google.com/3dwarehouse/">3D Warehouse</a> by July 10.<br /><br />We’re pretty jazzed that our panel of judges includes Bobby Brooks from <a href="http://corporate.disney.go.com/careers/who_imagineering.html">Walt Disney Imagineering</a>, Ken Harsha from <a href="http://www.ea.com/">Electronic Arts</a>, Janet Martin from <a href="http://www.commartsdesign.com/">Communication Arts Inc.</a> Paul Seletsky from <a href="http://www.som.com/opener.cfm">Skidmore, Owings & Merrill</a>, Gary Smith from <a href="http://www.gis-help.com/">Green Mountain Geographics LTD</a>, and Ken M Tse from <a href="http://www.hksinc.com/">HKS Architects, Inc.</a><br /><br />We hope to see your stomping grounds soon.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmSW74GcpUdOLWQLifVqV5sVFefplMZy9Ht8OyFSFTMTpIiV46QYRQpk8iaC8BUb4Hm0NKVO7UkWHdVbjHpydjpZNIAA1rulpMBA9PnlnT36o5nzyx1W-MBSZAm1apCsix90bUn37naq2O/s1600-h/Sketchup_campus.jpg"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5023713150513155474" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmSW74GcpUdOLWQLifVqV5sVFefplMZy9Ht8OyFSFTMTpIiV46QYRQpk8iaC8BUb4Hm0NKVO7UkWHdVbjHpydjpZNIAA1rulpMBA9PnlnT36o5nzyx1W-MBSZAm1apCsix90bUn37naq2O/s320/Sketchup_campus.jpg" style="cursor: pointer;" /></a>

Show us your university campus in 3D

Posted by Allyson McDuffie, Google SketchUp Education Program Coordinator Today the Build Your Campus in 3D Competition begins. This spring...

Đọc thêm »

A look ahead at Google Video and YouTube

<span class="byline-author">Posted by Salar Kamangar, Vice President, Product Management</span><br /><br />In November, we <a href="http://www.google.com/intl/en/press/pressrel/youtube.html">officially closed our acquisition</a> of YouTube, and since then we've received a number of questions about what will happen next. The summary is that <a href="http://video.google.com">Google Video</a> and <a href="http://www.youtube.com">YouTube</a> will continue to play to their respective strengths. But here's a bit more detail: <br /><br />Google's strength -- and its history -- is grounded in search and in innovating technologies to make more information more available and accessible. YouTube, meanwhile, excels at being a leading content destination with a dynamic community of users who create, watch and share videos worldwide.<br /><br />Google search results already include links to content that's hosted on YouTube. Starting today, YouTube video results will appear in the Google Video search index: when you click on YouTube thumbnails, you will be taken to YouTube.com to experience the videos. Over time, Google Video will become even more comprehensive as it evolves into a service where you can search for the world's online video content, irrespective of where it may be hosted.<br /><br />This is part of Google's overall goal to give you the highest quality search results possible. For example, some users who do a Google search for Martin Luther King, Jr. may want to find websites about him. Others may want to see images of him. And others may want to watch video footage...<br /><br />YouTube, as we've stated previously, will remain an independent subsidiary of Google, and will continue to operate separately. Google will support YouTube by providing access to search and monetization platforms and, when/where YouTube launches internationally, to international resources. YouTube co-founders Chad Hurley and Steve Chen and the rest of the YouTube team will continue to innovate exciting new ways for people to "broadcast themselves."<br /><br />Earlier this week, we announced one example of innovation in monetization and distribution with a new AdSense video test. We'll be working with a wide set of content providers, grouping together high quality video content from providers with high quality ads and offering them as playlists which publishers can select from and display on their AdSense sites. (There's more about the test <a href="http://adsense.blogspot.com/2007/01/this-is-test-video-content-on-adsense.html">on the AdSense blog</a>.)<br /><br />Today represents just the first step in our plan to bring you a comprehensive video search and content platform. We'll provide ongoing updates as they unfold.

A look ahead at Google Video and YouTube

Posted by Salar Kamangar, Vice President, Product Management In November, we officially closed our acquisition of YouTube, and since then w...

Đọc thêm »

Let's get together

<span class="byline-author">Posted by Griff Hazen, Software Engineer</span><br /><br />Online groups have become a great tool for managing and sharing activities and ideas for a defined group of people. Back in October we introduced a new version of <a href="http://groups.google.com/?utm_source=en-hpp-gd4beta&utm_medium=hpp&utm_campaign=en">Google Groups</a> (in beta) in the hopes that it would help people work together better. We received tons of feedback, and today we're dropping the "beta" and unveiling a number of new features in more than a dozen languages. If you're already in a Google Group, you'll be transitioned automatically, and if you're new to a group, welcome!<br /><br />Now you can customize the look of your group, create and edit web pages, upload and share files (including photos), and view member profiles. And for your discussions, there's no need to struggle to follow interrupted conversations, as Google Groups now includes the same style of organization that Gmail users love. <a href="http://groups.google.com/intl/en/googlegroups/tour3/index.html?utm_source=en-blog-groups3&utm_medium=blog&utm_campaign=en" title="Take a tour to learn more.">Take a tour to learn more.</a><br /><p>And if you're wondering how you might use a group, how about:<br /></p><ul><li>coordinating your <a href="http://groups-beta.google.com/group/nixa1997">class reunion</a> (or make it private for a family reunion)</li><li>learning about <a href="http://groups-beta.google.com/group/Greenhouse-Hydroponics">Greenhouse Hydroponics</a> and participate in discussions</li><li>finding the group you're looking for, or <a href="http://groups.google.com/?utm_source=en-hpp-gd4beta&utm_medium=hpp&utm_campaign=en">create one of your own</a><br /></li></ul><p>We hope you enjoy Google Groups; we really enjoy working on it, and it's been very rewarding to see all the useful things you and so many others have done with it. </p>

Let's get together

Posted by Griff Hazen, Software Engineer Online groups have become a great tool for managing and sharing activities and ideas for a defined ...

Đọc thêm »

Our New York speaker series

<span class="byline-author">Posted by Marcus Mitchell, Engineering Director, Google New York</span><br /><br /><a href="http://maps.google.com/maps?f=q&hl=en&q=76+Ninth+Avenue,+New+York,+NY,+10011+%28Google+New+York%29&ie=UTF8&om=1&ll=40.744331,-74.004121&spn=0.011965,0.043259&t=h">Google New York</a> is sponsoring a speaker series that will bring technology industry leaders in to our offices to share their unique perspectives. Speakers will come from Google and beyond to cover topics ranging from the history of software development to the future of the Internet. With this series, we hope to create a collegial atmosphere where members of the technical community can learn from and get to know one another.<br /><br />To start the year off, you're invited to a talk called "Physics, Speed, and Imprecision: What Works and What Doesn't in Software, and Why," given by our very own <a href="http://www.google.com/intl/en/corporate/execs.html#bosworth">Adam Bosworth</a> on Monday, January 29th.<br /><br />Over beer and wine, Adam will discuss why machine learning and natural language seem to work today when they didn't in the 90s, and why Ajax is successful now when it wasn't earlier. The reasons lie in the realm of physics and customer psychology, particularly in our need for speed and limited tolerance for imprecision.<br /><br />Please <a href="http://services.google.com/events/nyspeaker_series07">join us</a>, but registration is limited, so hurry.

Our New York speaker series

Posted by Marcus Mitchell, Engineering Director, Google New York Google New York is sponsoring a speaker series that will bring technology ...

Đọc thêm »

Don't miss your opportunity to be part of Google's first Code Jam Latin America!

<span class="byline-author">Posted by </span><span chatdir="2"><span chatindex="E5B429F5C89AE07E1">Berthier Ribeiro-Neto, Engineering Site Director</span></span><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinohs-7ZjUd_xZ8qykWgq1SgNpcb1yrjQojCTPpEOPdDrjxefrjTysFzYpuYxbLOf-fT9SKKcJOpwtv9WguE7Kjan28t2O2E51ePeP1GciAiytfBSEM3fg8w0Idc15IU-wa8yeYEmLfXCA/s1600-h/CodeJam_LatAm.jpg"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5021191261618278242" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinohs-7ZjUd_xZ8qykWgq1SgNpcb1yrjQojCTPpEOPdDrjxefrjTysFzYpuYxbLOf-fT9SKKcJOpwtv9WguE7Kjan28t2O2E51ePeP1GciAiytfBSEM3fg8w0Idc15IU-wa8yeYEmLfXCA/s320/CodeJam_LatAm.jpg" style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer;" /></a>There are only 5 days left until registration closes for <span name="st"> Google </span> Code Jam Latin America 2007. So far over 4,000 competitors have signed up for a chance to showcase their programming skills and win an all expenses paid trip to <span name="st">Google</span>'s Brazil engineering office, where they will compete for R$75,000 (Brazilian reales) in cash and prizes. The finals will be held on March 1. Latin Americans have registered in large numbers from Brazil, Argentina, Mexico, Peru, Colombia, Chile, Venezuela and Ecuador as well as many other countries.<br /><br />The top 50 finalists will be flown to Belo Horizonte to show us what they've got. Think you've got what it takes to Code Jam? Click <a href="http://www.google.com/codejamlatinamerica/">here</a> <span style="text-decoration: underline;"><span style="text-decoration: underline;"></span></span> to register!

Don't miss your opportunity to be part of Google's first Code Jam Latin America!

Posted by Berthier Ribeiro-Neto, Engineering Site Director There are only 5 days left until registration closes for Google Code Jam Latin ...

Đọc thêm »