Dealing with the limitations of AJAX and Flash can be an SEOs worst nightmare. There are so many issues that come into play & usually you can deal with many AJAX SEO issues by developing Progressively Enhanced code. While, I’ve written about how to address crawling and indexing with AJAX and SEO. I wanted to take a little bit of a deep dive into this again, along with the bigger issue, ranking.

The URL Problem:

As the web becomes more and more dynamic, especially with the introduction of the stream, real-time updates, etc., the ability to crawl and index that content becomes a problem. AJAX introduced a problem to search engines by dynamically changing page or website content with URL Fragments, or hash marks, “#”. this becomes a problem because traditionally (esp. in web 1.0 days) hash marks were used as an anchors to content on a single page. Since search engines would already crawl the page, they would ignore the URL fragments (“#”) because they were already crawling the page.

For example, these two URLs would be seen as the same and would have no reason to crawl the url fragment because it was already being done.:

• http://www.tonyadam.com/faq
• http://www.tonyadam.com/faq#question-10

Fixing Crawling and Indexing through Progressive Enhancement

While AJAX is not new to the web, there is a problem that it introduces to search engine crawlability and indexing that wasn’t around way back when. We now have websites that dynamically generate content within it’s current page architecture. So, sites started implementing AJAX as a way to dynamically change the page content without page refreshes and therefore create much nicer (and sexier) user experiences, as they wouldn’t have to wait for a new page to load, etc.

That said, you are left with this URL fragment that a search engine was not built to crawl and index. So, it becomes increasingly important to use web standard techniques like Progressive Enhancement and leveraging the Hijax method. Most technology teams adopt these methods and should be standard practice across technology and engineering functions.

Relevancy and Ranking Problems that arise

As more and more of the web becomes dynamic, being strategic in the uses of the dynamic content is important. The way we rewrite URLs and change page content is extremely important to having relevant content show up in the Search Engine Results Pages (SERPs). Just like building an entire site in Flash can be detrimental to your organic search traffic, building an entire site in AJAX can do the same thing. The reason being is that you end up with URL structures that contain #’s (URL Fragments) all over the place.

The drawbacks with #’s and rankings
This becomes a problem when people like bloggers, writers, site owners, etc. start linking to your content with #’s in the URLs. Search engines would ignore everything after the # and assume all links go to the homepage. For example,If I decide to build an entire site on basketball, so, assuming I want to rank for terms like basketball news, basketball scores, etc., I would probably have them in my top nav.

If I was to have the content refresh using AJAX, here is what my URL structure would look like:

• http://www.basketball.com/
• http://www.basketball.com/#/news/
• http://www.basketball.com/#/scores/
• http://www.basketball.com/#/players/
• http://www.basketball.com/#/stats/

Or deeper sections of the site like

• http://www.basketball.com/#/players/ray-allen/stats/
• http://www.basketball.com/#/players/phil-jackson/awards/

Do you see the problem with the URL Structure? If this site was purely built with AJAX, as mentioned above, anybody that links to my site would link to http://www.basketball.com/ and I would get no deep content links. If I get links to my homepage with obscure keywords like “Ray Allen player stats” or “Phil Jackson coaching awards” and since everything after the # is ignored, those keywords would then be pointed at the homepage, which is not search engine relevant.

Using #’s and AJAX to your advantage
Alternatively, there are situations where it makes more sense not to have unique URLs on content to conserve link equity and target at one main page. For example, I would not want a URL for various filters on a users stream in their profile because I would dilute the link equity to those stream items.

If I was to have a unique URL and link to the following, I would dilute link equity:

• http://www.myspace.com/tonyadam/stream/all
• http://www.myspace.com/tonyadam/stream/photos
• http://www.myspace.com/tonyadam/stream/videos

In this situation, I would rather have URLs that conserve the link equity to the main profile url:

• http://www.myspace.com/tonyadam/#/stream/all
• http://www.myspace.com/tonyadam/#/stream/photos
• http://www.myspace.com/tonyadam/#/stream/videos

By doing so, links to the # URLs will then all be targeted at the main URL http://www.myspace.com/tonyadam/ which will create more link equity to a stronger, more relevant page.

Now, we could sit here all day and talk about all the ways to manipulate results by sending links to versions of the URL that are rewritten, but, that is not a scalable tactic, is temporary, and could lead to search penalties. At the end of the day, the goal should be to create a dynamic website that is relevant search engines and users, that creates the most holistic experience.

Think of it this way, after all your hardwork and innovation, if you can’t get users to your website, what does the shiny new object matter anyway?

With the “Web 2.0″ world that we live in, companies are inclined to create “snazzy” new AJAX experiences for users. I have been dubbing this the AJAX Dilemma lately because organizations/companies/website owners are creating sites that are not accessible and SEO friendly. They are risking crippling their businesses in the areas of search traffic by not planning SEO into the lifecycle of the product.  Ensuring there is a Non-Javascript experience for search engines and accessibility is not a new practice. Creating sites using “Progressive Enhancement” and using “Unobtrusive JavaScript” has been around for quite some time.By creating web standard code and using the methods above, you are not sacrificing SEO for User Experience.

This is important to creating search friendly websites because ensurng search engines are able to crawl and index your content is necessary to acquiring search traffic and market share. By not embracing these approaches, you are leaving yourself in a handicap for SEO efforts and leaves you at a competitive disadvantage. In fact, it should not even just be designed on a project per project basis, but should be built into code that takes advantage of web standards and progressive enhancement.

The AJAX Dilemma for SEO

Show and Hide type interactions that are used in content areas throughout a site, like tabs or accordions, can create a crippling effect on the amount of content crawled by search engines, if they are not built correct. This content is typically built with poorly written JavaScript that does not take advantage of Web Standard code. The hidden content areas, when clicked on, will provide a string that follows the URL like, #somehiddencontent. Traditionally search engines ignore everything after the “#” (hash) tag.

Here are some examples of these interactions that are poorly written and don’t provide a graceful fail when JavaScript is turned off.

Here is an example of tabs used on iFoods.tv with JavaScript on:

Tabs using AJAX on ifoodstv

Here is that same interaction, with JavaScript turned off:

Notice something different? Of course you do, it is VERY blatently obvious that the entire content within the tabs is completely missing. This is a large opportunity that is missed by ifoodstv.com at getting content not only indexed, but great internal links to deep content.

Now, taking a look at the way that iMedix does tabs is a little different, here it is with JavaScript on:

Here it is with JavaScript Off, on a different tab:

Do you notice a difference here at all? No?! Well, thats right, because there isn’t one.

Also, to add to that, iMedix is rewriting URLs so that there are no hash tags in the URLs:

This is a fantastic job of ensuring that there is not only crawlable content, but crawlable URLs. This is very important, search engines tend to ignore things after hash tags (#) in URLs. This is especially impotant with AJAX since that is typically how urls are created. Jeremy Keith (JavaScript expert extraordinaire) has described some ways to get around the AJAX issues, including has tags, and has described it as Hijax.

The Solution

The solution to creating AJAX that is SEO friendly is to ensure that you are building your site using Progressive Enhancement and Unobtrusive JavaScript. Along with that, url issues that could lead to canonicalization and duplicate content issues can be avoided using methods like Hijax. With the extremely competitive markets out there on the web, this is VERY important. It is possible to create rich user experiences with Ajax that are great for SEO.

As an example of how to do this, you can build standard interactions on your site that are accessible and search friendly. (This is how we built any standard interaction at PayPal). Any standard show/hide type interaction can be built using standardized JavaScript APIs. Doing this will create Non-JavaScript versions of interactions that are accessible and search friendly, but along with that will drastically reduce the amount of code that you have to write and increase developer efficiency.

Resources to help with AJAX and SEO

Google Webmaster Central: A spiders view of Web 2.0
Dom Scripting: Hijax
Progressive Enhancement with AJAX

Follow me on Twitter for more info @tonyadam or subscribe to my feed to keep up to date!