Search engine indexation and crawling made easy

Another great article on SEOmoz, well worth a read to understand spiders' behaviour.
http://www.seomoz.org/blog/an-illustrated-guide-to-matt-cutts-comments-on-crawling-indexation