Site Speed: Reducing Page Load Times and Increasing Pages Crawled

I’ve always been a big fan of quick page load times on sites from overall User Experience standpoint. But, ever since Google announced that they were using site speed in web search ranking I have become a big time stickler about this. After watching this for the last couple months, I’ve realized that my peskiness about the whole thing warranted just cause.

Proving that Site Speed is directly correlated Pages Crawled

I tend to keep an eye on Google Webmaster tools to see any blips in crawl rate, pages crawled, etc. I don’t want any big issues to show up that catch me by surprise. When Google made the announcement I made sure to keep an eye on site speed. Here are a couple examples of the DIRECT CORRELATION

Website 1: Page Load times shoot up, Pages Crawled drops:
First look at graph three and notice the massive upswing in the page load times or as google titles it “time spent downloading a page.” Then, from there, look at the top two graphics and notice the massive drop off in the number of pages crawled per day and the amount of content downloaded per day.

Page Load time increase

Website 2: Page Load times go down, Pages Crawled dramatically increases:
On the flip side of the above chart, looking at another site, check out graph three again first. Notice that the time spent downloading a page (thus page load time) has decreased dramatically. Now, as you look at graphs one and two, notice the massive uptick in the number of pages crawled and amount of content downloaded daily.

Pages Crawled Increases

I can tell you that this was a big positive for website 2, to notice a HUGE positive in the number of pages crawled daily.

What this means for SEO and User Experience

Understanding how to diagnose and address these issues is extremely important to any SEO and Webmaster now at days. I was a huge advocate of this at PayPal and lent a helping hand in diagnosing this and addressing it in hopes of increasing the user experience. Again though, we now know what a big part this obviously plays, as I’ve shown the direct correlation above with the number of pages crawled increasing or decreasing based on the page load times.

I was very happy to see this as it does increase the overall user and searcher experience by coming to sites that are loading pages faster. But, it is extremely important for an SEO to note and understand the importance of site speed, especially with sites that have extremely large indexes.

Diagnosing Site Speed in Google Webmaster Tools

Start by logging into webmaster tools and talking a look at the Diagnostics -> Crawl Stats section and make note of any increases or decreases in the third graph about page download times. If there are any big rises or drops, check out the above two graphs to see if it had an effect on your pages crawled. If you are seeing relatively steady movement, move on to the Labs -> Site Performance section.

On the Site Performance page you’ll notice a few things. To start with the graph that represents the avg. number of seconds it takes to load your pages. Something to take note of that they’ve highlighted the optimal time to download a page as 1.5 seconds or faster. Depending on your site, they’ll list about 10 URLs and their respective page load time. Finally, they provide some very useful info (which I’ll get to in a minute) on how to address these concerns.

There are a couple tools that you can use to diagnose site speed and page load times. My favorite is yslow and Google recently released their Page Speed tool, both of these plug directly into my favorite Firefox Plugin: Firebug.

Addressing Site Speed and Page Load time Concerns

Google Webmaster Tools and their Page Speed tool, along with YSlow provide you ample amounts of information to start addressing the site speed concerns one step at a time. Some of the biggest mistakes that people make run into is the size of images and image optimization, number of HTTP request and number of external JavaScript and CSS files. Here is a list of issues and how to address them:

Making fewer HTTP Requests
The number of HTTP requests you make can put a massive strain on the page load times. As mentioned in the screenshot below, try to combine external files to a very minimal amount. I’ve seen sites that have as many as 20 external JavaScript files and some of those files with only 1-4 lines of code. That is the type of thing that can be combined to reduce the number of http requests. The same goes with CSS files and optimizing the number of CSS calls into one main CSS document.

http requests

JavaScript Minification and Compression
Yahoo!’s Developer Network blog has a post dedicated to how to minify javascript. The main reason for doing this is to shrink the overall size of the JavaScript file by getting rid of comments, new lines, etc. that aren’t necessary for the JavaScript to process. There are a couple tools out there to do this, JSMin and YUI Compressor.

Using Gzip to compress components
You can drastically reduce the HTTP Requests and Responses using Gzip compression, and, it is the most popular and effective form right now. In Apache you can use mod_gzip and in IIS you can do this through IIS Manager and adding a Web Service extension using these IIS compression tips.

CSS Sprites
CSS Sprites are extremely useful for reducing the number of image requests (thus lowering http requests). Having your background images in one large image and using CSS for the background image and positions you can use a single image for many purposes. A List Apart has a great resource that goes into the details of the HTML Markup and CSS that is necessary to make this dream a reality.

Much much more!
There are a ton more ways of improving site performance and reducing page load time to not only help your page crawl rate and pages crawled on your site, but also, increase the user experience. Some of the things I didn’t even get to touch on like adding expires headers, reducing the number of DNS lookups, using a CDN (Content Delivery Network), etc. are all very important to reducing page load times.

Have you noticed similar things with page load times and pages crawled? How are you dealing with Site Speed concerns for both SEO and User Experience?