I’ve always been a big fan of quick page load times on sites from overall User Experience standpoint. But, ever since Google announced that they were using site speed in web search ranking I have become a big time stickler about this. After watching this for the last couple months, I’ve realized that my peskiness about the whole thing warranted just cause.

Proving that Site Speed is directly correlated Pages Crawled

I tend to keep an eye on Google Webmaster tools to see any blips in crawl rate, pages crawled, etc. I don’t want any big issues to show up that catch me by surprise. When Google made the announcement I made sure to keep an eye on site speed. Here are a couple examples of the DIRECT CORRELATION

Website 1: Page Load times shoot up, Pages Crawled drops:
First look at graph three and notice the massive upswing in the page load times or as google titles it “time spent downloading a page.” Then, from there, look at the top two graphics and notice the massive drop off in the number of pages crawled per day and the amount of content downloaded per day.

Page Load time increase

Website 2: Page Load times go down, Pages Crawled dramatically increases:
On the flip side of the above chart, looking at another site, check out graph three again first. Notice that the time spent downloading a page (thus page load time) has decreased dramatically. Now, as you look at graphs one and two, notice the massive uptick in the number of pages crawled and amount of content downloaded daily.

Pages Crawled Increases

I can tell you that this was a big positive for website 2, to notice a HUGE positive in the number of pages crawled daily.

What this means for SEO and User Experience

Understanding how to diagnose and address these issues is extremely important to any SEO and Webmaster now at days. I was a huge advocate of this at PayPal and lent a helping hand in diagnosing this and addressing it in hopes of increasing the user experience. Again though, we now know what a big part this obviously plays, as I’ve shown the direct correlation above with the number of pages crawled increasing or decreasing based on the page load times.

I was very happy to see this as it does increase the overall user and searcher experience by coming to sites that are loading pages faster. But, it is extremely important for an SEO to note and understand the importance of site speed, especially with sites that have extremely large indexes.

Diagnosing Site Speed in Google Webmaster Tools

Start by logging into webmaster tools and talking a look at the Diagnostics -> Crawl Stats section and make note of any increases or decreases in the third graph about page download times. If there are any big rises or drops, check out the above two graphs to see if it had an effect on your pages crawled. If you are seeing relatively steady movement, move on to the Labs -> Site Performance section.

On the Site Performance page you’ll notice a few things. To start with the graph that represents the avg. number of seconds it takes to load your pages. Something to take note of that they’ve highlighted the optimal time to download a page as 1.5 seconds or faster. Depending on your site, they’ll list about 10 URLs and their respective page load time. Finally, they provide some very useful info (which I’ll get to in a minute) on how to address these concerns.

There are a couple tools that you can use to diagnose site speed and page load times. My favorite is yslow and Google recently released their Page Speed tool, both of these plug directly into my favorite Firefox Plugin: Firebug.

Addressing Site Speed and Page Load time Concerns

Google Webmaster Tools and their Page Speed tool, along with YSlow provide you ample amounts of information to start addressing the site speed concerns one step at a time. Some of the biggest mistakes that people make run into is the size of images and image optimization, number of HTTP request and number of external JavaScript and CSS files. Here is a list of issues and how to address them:

Making fewer HTTP Requests
The number of HTTP requests you make can put a massive strain on the page load times. As mentioned in the screenshot below, try to combine external files to a very minimal amount. I’ve seen sites that have as many as 20 external JavaScript files and some of those files with only 1-4 lines of code. That is the type of thing that can be combined to reduce the number of http requests. The same goes with CSS files and optimizing the number of CSS calls into one main CSS document.

http requests

JavaScript Minification and Compression
Yahoo!’s Developer Network blog has a post dedicated to how to minify javascript. The main reason for doing this is to shrink the overall size of the JavaScript file by getting rid of comments, new lines, etc. that aren’t necessary for the JavaScript to process. There are a couple tools out there to do this, JSMin and YUI Compressor.

Using Gzip to compress components
You can drastically reduce the HTTP Requests and Responses using Gzip compression, and, it is the most popular and effective form right now. In Apache you can use mod_gzip and in IIS you can do this through IIS Manager and adding a Web Service extension using these IIS compression tips.

CSS Sprites
CSS Sprites are extremely useful for reducing the number of image requests (thus lowering http requests). Having your background images in one large image and using CSS for the background image and positions you can use a single image for many purposes. A List Apart has a great resource that goes into the details of the HTML Markup and CSS that is necessary to make this dream a reality.

Much much more!
There are a ton more ways of improving site performance and reducing page load time to not only help your page crawl rate and pages crawled on your site, but also, increase the user experience. Some of the things I didn’t even get to touch on like adding expires headers, reducing the number of DNS lookups, using a CDN (Content Delivery Network), etc. are all very important to reducing page load times.

Have you noticed similar things with page load times and pages crawled? How are you dealing with Site Speed concerns for both SEO and User Experience?

Published On: July 29th, 2010 / Categories: SEO /

About the Author: Tony Adam

Tony Adam is a serial technology entrepreneur, investor, and Fractional CMO. He is currently the Founder & CEO of Visible Factors a Digital Marketing Agency providing Direct-To-Consumer (DTC) brands, startups and large organizations services around growth and online marketing principles like SEO, Google Ads, Meta Ads, and Email/Lifecycle Marketing. Prior to Visible Factors, Tony founded Eventup, an Event Venue Marketplace and grew to 12 cities and over $1MM top line revenue in under one year. Throughout his career, he has worked with early stage startups, SMBs, Fortune 500 companies and high-profile brands including Yahoo!, PayPal and Myspace.
Share This Story Via Social Media!


  1. Shane July 29, 2010 at 6:17 pm - Reply

    Site speed is probably a big factor, but it's not a direct correlation. Check out the stats on these two different sites: http://www.askshane.org/off-topic/crawl-stats.php

    • tonyadam July 29, 2010 at 6:29 pm - Reply

      Interesting Shane, I guess each of our graphs tells a different story. I wonder if it is only limited to certain types of sites or certain sized sites. Let's keep testing, it's the only way to know!

      • Shane July 29, 2010 at 7:54 pm - Reply

        Exactly! I love testing :)

  2. […] This post was mentioned on Twitter by Tony Adam, Jaamit Durrani, SEO Portland, Joe Mangum, Kristy Bolsinger and others. Kristy Bolsinger said: Site Speed: Reducing Page Load Times and Increasing Pages Crawled http://bit.ly/9hTatI by @tonyadam (cc @evilbeet – you'll like this!) […]

  3. Cyrrus July 29, 2010 at 11:34 pm - Reply

    Indexing is just part of the game. Once you see increased SEO traffic, then you can celebrate ;)

    • tonyadam July 30, 2010 at 6:07 am - Reply

      Totally agree Cyrrus, but, it's the first step!

  4. Gary Chiro July 30, 2010 at 5:55 am - Reply

    Tony do you think 100/100 page speed is possible? I see that even Google.com can't mange it they get 92/100

    • tonyadam July 30, 2010 at 6:08 am - Reply

      Gary, I don't think 100/100 is realistic, my goal is to get pages to under 1.5 second page load times….when i'm there, I'm happy :)

    • Matthijs August 2, 2010 at 1:30 am - Reply

      Why wouldnt it? The question is, is 100 out of 100 important or is the actually loadtime important? Well from what I can say its the loadtime ;-)
      The score is just an indication and a guidance. It shows what you did and can do to improve the pagespeed on-site. Beside you can have a 100/100 score but if your server (due what so ever reason) is slow your pagespeed will still be slow.

      • tonyadam August 2, 2010 at 4:31 am - Reply

        That's actually a great point, focus more on the page load time vs. the score…nice call out Matthijs! :)

  5. Gareth July 31, 2010 at 9:03 pm - Reply

    Those graphs a pretty interesting, I'd like to see this info over a large data set.

    • tonyadam August 2, 2010 at 4:32 am - Reply

      Gareth, some of these are over VERY large data sets…unless you mean different sites…i've seen identical results with 5 different sites, email me and let me know how i can help! :)

  6. […] Tony Adam: Site Speed: Reducing Page Load Times and Increasing Pages Crawled […]

  7. Pieter August 1, 2010 at 7:40 am - Reply

    We've done great efforts in bringing down page load speeds and recently moved up from no 2 to no1 on the most important keyword in our market. Our website is way faster than our nearest competitor and former no 1, so it seems like Google really rewards good performance!

    • tonyadam August 2, 2010 at 4:34 am - Reply

      Congrats Pieter on taking the #1 spot! I'm seeing the same thing in rankings as I am indexing/crawling…and it looks like you were the recipient of the same! :)

  8. Sean - SEO August 1, 2010 at 11:53 pm - Reply

    Google is telling exactly what it has been following. We all know how good is Google in providing results with fast and reliable information. Also, Google many times stated that its objective is to provide the information with high relevancy and speed. Though it is a bit hard to dynamic websites, I think it is now the time to re-design/ alter at least their main product pages.

    • tonyadam August 2, 2010 at 4:35 am - Reply

      I couldn't agree more Sean, I've been a big fan of quick page loads for years because of the user experience aspect…having it effect traffic will only make people look at it much closer, which makes me very happy!

  9. […] Tony Adam: Site Speed: Reducing Page Load Times and Increasing Pages Crawled […]

  10. […] Site Speed for SEO: The correlation between site speed and pages crawled by Google […]

  11. Pierce August 3, 2010 at 3:30 pm - Reply

    I don't know if this is a true correlation. From experience all I have seen is the more pages google downloads the lower the time spent downloading.
    I see the upper limit of those graphs have been blanked out, but if its similar to mine with 31,000 pages downloaded per day with an average of 343ms per page load.. in all honest thats not going to make much difference if its 600ms or 343ms but you still get those lovely up and down graphs.
    What I am trying to say is the more pages google downloads the better the average it seems to get.
    If you want to make a truly significant difference enable gzip on your pages. That way google wont mind downloading many more pages for the same amount of bandwidth ;)

  12. rolfbroer August 3, 2010 at 7:56 pm - Reply

    Nice test! Always interesting to see the movements of the Googlebot.

  13. evdenevenakliyat December 8, 2010 at 4:25 pm - Reply

    interesting analysis, visitor-friendly sites, do we understand this point of..Thankssssss

  14. Vlad January 20, 2011 at 7:57 am - Reply

    Everyone has been saying that increasing the site's speed would increase the number of pages crawled, but I haven't seen much proof of that until I saw this article. Thanks for the demonstration Tony

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe To Receive The Latest News

Add notice about your Privacy Policy here.