VIDEO Q and A: Is our site being penalized by Google?

Dear Kalena...

The problems we are having are mainly with Google. We have 1,000 indexed pages showing in the 'site:' command from Google search. We have already done many things to get our rankings up but we appear to be penalized. Customers can not find our indexed pages by title OR content. We suspect sabotage by another company. We have sent emails and faxes to Google and asked them to investigate. They ignore us. We're out of options.

Darren

Kalena's Answer:

Dear Darren

Click here to see my video answer


Need more than advice? Take a Search Engine Marketing course online

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Q and A: Why have our most popular pages disappeared from Google?

Dear Kalena...

Thanks again for your help a couple of years ago. I need some more advice now and I don't even know where to start troubleshooting. About a week ago I noticed that the bulk of our most popular pages no longer seem to be on Google's radar. I'm talking about pages that used to show up in the top 10 results for typical searches in our industry. Currently, those pages don't show up AT ALL in Google's results, or Google offers a comparatively irrelevant page, like our home page or links page that might happen to have the keywords in question.

This problem is only with Google, not MSN or Yahoo. Probably began during the last 30 days. Other pages come up in Google SERPs just fine. Also, I checked our Google Webmaster Tools and everything looks OK - sitemap downloaded OK, all pages (including the problem ones) indexed. PR for the problem pages is unchanged (lackluster 2-3, but at least not lowered). The only thing I did different was to use a new sitemap a couple of weeks ago. Any ideas?

Rick

Kalena's Answer:

Dear Rick

I'm getting a lot of similar questions to yours at the moment and I'm convinced it is the result of a major tweak Google has made this month to their PageRank algorithm (not to be confused with the Google Toolbar PageRank green bar). Here's my reasoning:

• None of your pages show up in Google's Supplemental Index, indicating those pages haven't been removed from Google's main datacenter.
  
  
• Google is currently showing 179 pages from your site as being indexed, whereas Yahoo is actually showing over 300 pages indexed, indicating that Google may be suppressing the value of some of your pages.
  
  
• You didn't tell me the search query that returns the rankings you are talking about, but if you were previously ranking well for those terms and you've not changed the pages, then it's probably an external cause rather than something you did to cause the ranking drops.

Google makes small tweaks to their ranking algorithm on a regular basis. Some of these tweaks involve the addition of code filters to detect and suppress code it sees as artificially influencing your page's relevancy. It may be that the new algorithm includes a new filter that has picked up something on your pages Googlebot doesn't like, for example, excessive keyword repetition or duplicate content. Many of your pages have almost identical content to each other, which could have triggered a suppression filter.

Also, you have quite a large number of backward links showing in Yahoo (over 300) but only 8 showing in Google. It may be that Google has decided many of those backlinks are not relevant and has suppressed any influence they previously had on your rankings. The reciprocal link swapping concept you use on your site and the advice you give to potential link partners is quite flawed. It will likely only attract links from very low quality sites, diluting your own site's link popularity as a result. Many of the sites listed on your links page are completely irrelevant to your site. Read my link swapping rant for more info.

Finally, keep in mind that thousands of new pages get added to the Internet every day. Chances are that some of these might be targeting the same keywords and phrases that you are. If those pages are better optimized than yours, yours will naturally be pushed down in the results.

Need more than advice? Take a Search Engine Marketing course online

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Q and A: Has my domain glitch caused permanent de-ranking in Google?

Dear Kalena...

I let my domain www.visaplace.com expire, but I re-registered it within 24 hours. The site went black in the interim and it took about a 1/2 day to be repopulated online. However about a week to two weeks later I noticed that Google dropped my rankings for virtually all of my key words. I am totally invisible online now whereas I was well ranked before. I was told by my SEO person not to worry because once Google spiders my site again a few times I will get back up to my original positions.

My SEO person looked at my site and said nothing has changed and I was not blacklisted or anything. He said I should be back up within days to a few weeks. My question is: Is explanation credible? Is there another possible reason why I am de-ranked? I am really concerned.

Thanks so much
Michael

Kalena's Answer:

Dear Michael

Your SEO is right. What's probably happened is that Googlebot tried to index your site during the time the site was down and so dropped some/all of your previously cached pages. This can happen from time to time, especially with hosting outages etc. Obviously, if those pages were previously ranking well for certain search queries, but the pages have temporarily disappeared from Google's data store, those rankings will disappear too.

I see now that Google last cached your page on October 19 so all seems to be well again. I'm not sure how many pages were indexed before the domain problem, but Google shows 79 pages currently indexed.

To check if any site is listed in Google, you can use their Site Status Tool. If your SEO is worth his salt, he will have created a Google Webmaster Tools account for your site and uploaded an XML sitemap to Google Sitemaps on your behalf. This will tell Google how many pages your site has and what the URLs are so Googlebot can index it accurately. If you think Google has dropped some pages, be sure to have your SEO update the XML file and ping Google from within Webmaster Tools when it's uploaded.

If you want to keep close tabs on how/when Google indexes your site, ask your SEO to provide login access to your Webmaster Tools account or set one up for yourself.

Need more than advice? Take a Search Engine Marketing course online

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Q and A: Why hasn't Google indexed all the pages in my sitemap?

Dear Kalena...

Hi my website is www.pakinfobytes.com. Google has indexed my 13 pages then I add my sitemap to google webmaster tools which contain 17 pages but still Google only indexed 13 pages. Why? And how they indexed my all sitemap pages? Is there is any way to tell Google about backlinks or does Google automaticly detects the backlinks? Are the backlinks pages included in sitemaps? I mean if www.example.com have my link then do I add www.example.com to my sitemap or not? I am wondering about how Google knows the backlinks.

Regards
Bilal

Kalena's Answer:

Dear Bilal

Google doesn't automatically add all pages from your site to their index. Because the Internet is so large and millions of pages get added every day, it is not possible for Google to index every single one. They have to determine which pages are the most important and index those. The others either get excluded or included in their supplemental index. You can learn more about the supplemental index here.

According to a site search, Google has indexed 14 pages from your site. A check of the supplemental index on the main Google datacenters shows 8 pages from your site stuck in the supplementals. To get those out, you need to get links to them, preferably from domains that Google considers trustworthy, such as directories or popular sites.

There is no need to tell Google about your backlinks. You certainly don't add them to your sitemap. That is only for your own site pages. Provided your backlinks are from pages that are in Google's index already, the links will be found and registered towards your site's link popularity. Google never displays your true number of backlinks, only the ones they consider important. But rest assured that all links pointing to your site are taken into account when determining your site's true PageRank score.


Need more than advice? Take a Search Engine Marketing course online

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Q and A: Why has our Google PageRank dropped to zero?

Dear Kalena...

Can you please help us? I just came across your site and you seem very knowledgeable.

Our problem is Google! Our site has been active for a few years now at www.theforeverrose.com We were once #1 for the search "the forever rose" (and ranked well for a few others as well). But we have been gradually slipping, now we are in position 90 for "the forever rose" and off the charts for others?

Our PageRank was once a three and gradually dropped to now 0. We cannot figure out why and things keep getting worse. We strictly follow all of Google's rules and ethics, we rank fine in Yahoo and MSN. I am tired of hearing the obvious; more links, more pages, better content, SEO.... etc, we have been doing that. I feel like we are just missing something really simple, something right in front of our eyes, something that is penalizing us!

Can you please help? Any of your help would be greatly appreciated.

Thank You,
Mike

Kalena's Answer:

Dear Mike

A quick check of your site with the Google Site Status tool shows that pages from your site are included in the index, but that Google may not know about all your site pages. The site was last indexed by Googlebot on 25 April and you have one backward link displayed by Google but 81 backlinks shown on Yahoo.

Your home page has a Google Toolbar PageRank of zero and some pages have greyed out PageRank and no cache, suggesting they haven't been indexed. Curiously, Google is showing 46 pages from your site in their index, while Yahoo is only showing 25 pages indexed. The fact that the site has already aged and used to have a much higher PageRank may suggest a penalty of some kind.

But there could be a few explanations for your poor PageRank and lack of rankings:

1) You are using a black page background but you then have a table on it with a white background and black text. Some search engines will see this as black text on a black background. It's possible that this may be tripping spam filters.

2) Your site is built using old technology and contains a lot of code bloat. Tables are clunky and difficult for search spiders to index and Googlebot may have tripped up on your code and not indexed all your pages.

3) Your home page contains keyword repetition for the words "rose" and "roses". I don't think the repetition is excessive, but it may have triggered some type of suppression filter in Google.

4) Your site has poor link popularity and the sites that link to you tend to have a very low quality score and no PageRank e.g. cufflinksdepot.com/dir-gifts.htm and escapesportif.com/resources/gifts.html. You don't have enough incoming links pointing to your site from what Google calls "trusted sites" - popular directories, portals and authoritative sites in your industry. Your internal links could also use some work from an anchor text angle.

5) Most of your site pages might be stuck in Google's supplemental index, colloquially (but unfairly) known as Google Hell. Google's Matt Cutts explains why some sites have the bulk of the pages moved to the supplemental results:

"If you used to have pages in our main web index and now they’re in the supplemental results, a good hypothesis is that we might not be counting links to your pages with the same weight as we have in the past. The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit)."

Here's what you should do to address the problems:

1) For better indexing, consider upgrading the site design away from tables to clean HTML and use CSS for formatting. Until you do that, change the background of all pages to white to avoid any potential hidden text penalties from your table layout.

2) Run your site through a text-editor such as Lynx to see what search engines see when they index your site. Verify your site with Google Webmaster Tools and check the diagnostics for potential indexing problems.

3) Optimize your site from scratch. You should make sure your site is search engine compatible and optimized for a wider range of target search keywords and phrases rather than the obvious ones.

4) Create and upload an XML sitemap to Google Sitemaps or use the new Sitemaps Protocol in your robots.txt file to tell search engines where to find your XML sitemap. I like to use the free XML Sitemaps Generator to create my sitemaps.

5) Commence a link building campaign pronto. This campaign should include submitting your site to all the major and minor directories and search engines where the site doesn't currently feature, as well as niche directories and portal sites in your specific industry. Where possible, anchor text incorporating your target keywords should be used within the links. My consulting company can take care of link building for you if you like.


Once changes to your site code have been made and you have achieved some good quality links, most of your problems should disappear. If the problems persist, file a re-inclusion request with Google, explaining what might have triggered penalties and what changes you've made to address the issues. Although technically your site hasn't been excluded from the Google index, this should prompt a review of your site by Google's anti-spam team and hopefully result in any suppression penalties being lifted.

Good luck and let us know how you get on!

Add to: Digg | Del.icio.us | Ma.gnolia | RawSugar | Reddit

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Q and A: Is my site in the Google Sandbox?

Dear Kalena...

I have owned a site of more than 70 pages for 1 month. Even Google indexed a part of them and I submitted to other directories, but I have no visitors (health.com.md). Where is my mistake? Does this mean that I am in Google sandbox? Thank you all for your time.

Oleg

Kalena's Answer:

Dear Oleg

Google last indexed your site on 18 April so at least some pages of the site are being indexed. If the site is only a month old, then it is highly likely that you are experiencing Google's "sandbox", more accurately known as the Aging Delay, which can effectively make new sites invisible in Google for up to 9 months.

I see that you have a Google verification meta tag in the site, so you have likely uploaded an XML sitemap and verified your site in Google's Webmaster Tools. You should check your site's statistics here regularly to ensure it is being fully indexed and also track the Google search queries the site is being found for.

Once your site is released from the aging delay, you will start to see the number of search queries increase and your traffic will soon follow.

Add to: Digg | Del.icio.us | Ma.gnolia | RawSugar | Reddit

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Q and A: Does Google Sitemaps eliminate the need to optimize a site built with frames?

Dear Kalena...

I have two sites; One www.alpine-property.com is top of Google rankings for all my chosen keywords, it has no frames, daily updated content and lots of inbound links, so not surprising.

My other site www.alptitude.com is the dunce. Frames, static content and just a few inbound links. I have keyword friendly text in the no frames tag and a simple site map. I also have a few pages of no frames content, these are the pages indexed by Google. I obviously know how to fix this, but the time required to rewrite the site in a no frames way is very large, it has a lot of transactional code on a linked secure server which I use to run the business.

In the meantime I am interested in the effect of using the Google Sitemap tool. Will this eliminate the frames problem? The Google FAQs about this ask me to submit both the frameset url and the url of each target page. This sort of suggests that they will be able to index all the content.

Steve

Kalena's Answer:

Dear Steve

Thanks for the caffeine. The answer to your question is yes and no. Sure, Google might be able to index your frames pages outside of the frames-context and eventually match up your content and work out what your site's about. But why make it so difficult? And why take the risk when you can spend a little time bringing your site into the 21st century and ditching the frames?

It's not only search engines that the frames might offend. Searchers tend not to like frames-based sites either and unless you start taking your site seriously, you can't expect visitors to. It's not that difficult to replace your frames pages with more modern code.

If you want to continue pretending it's 1997, at least check Danny Sullivan's Search Engines and Frames Tutorial to make sure your frames pages are optimized as much as possible. Follow the instructions on Google's Webmaster Tools for frames-based sites, create an XML sitemap and track the indexing of all your pages to see if the framed ones get indexed. My bet is they don't.

Add to: Digg | Del.icio.us | Ma.gnolia | RawSugar | Reddit

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Google confirms 301s are better than 302s to move a domain

Yes, I am meant to be on holidays and no, I'm not meant to be posting here, but I felt this news was too important to wait another two weeks.

At the Search Summit Conference this week, I had the opportunity to ask Adam Lasnik from Google a question that I get asked a lot: Is it better to use 301 Permanently Moved or 302 Temporarily Moved redirects if you need to move a site to a new domain?

Adam replied that provided you are using the same page file names, you should absolutely use 301s rather than 302s on your old pages if you want Googlebot to re-index your site quickly. He also recommended keeping the old site live until the new site was cached and transferring the site over in different stages, depending on the number of pages it has. Google Support Engineer Maile Ohye added that you should also make sure you verify the new domain and upload your XML sitemap for it via Google Webmaster Tools to aid faster indexing.

I asked if using 301s to the new domain would be more likely to trigger the aging delay to kick in for the new site, but Adam reassured me that using 301s in combination with Google Sitemaps should make the domain relocation process fast and painless. He used an example of a 500,000 page e-commerce site he watched moving domains recently via 301s in three stages and claimed that Googlebot had entirely indexed the new site in just over five weeks.

Add to: Digg | Del.icio.us | Ma.gnolia | RawSugar | Reddit

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Q and A: Why has our site suddenly dropped out of Google?

Dear Kalena...

Hi there, I have done a lot of work optimizing my wife's web site and succeeded in getting the site to number one on Google, Yahoo and MSN for the search term - "childminder milton keynes" - It took me 3 months to do it and her business has boomed. She is now completely booked up. I do regular checks to ensure the site is still no 1 for the search term and on Yahoo and MSN the site is still at number 1, however, on Google the site isn't even listed in the first 15 pages whereas 2 weeks ago it was indeed listed number 1. I am completely baffled, can you help please.

Thank you very much.
Mark

Kalena's Answer:

Hi Mark

First up, thanks for the caffeine contribution, it really helps! Now, about that site.

I've run the site through Google's Site Status Tool and according to the results, it is still being indexed, with the last visit by Googlebot on 14 January. However: the current Google cache of the site is completely blank and the Google Toolbar PageRank for the site is zero out of 10. Both these things indicate a major problem.

Now, I know the site is over a year old and that you last made changes over a month ago, so my guess is that rather than the aging delay, an algorithm penalty or other such manual suppression, Googlebot encountered problems when indexing the site last, which resulted in zero pages being indexed and stored. Naturally, the site has dropped off the charts because there is zero information stored in Google's datacenter as a result of the indexing and caching issue.

However, I'm not surprised Googlebot had trouble indexing the site. It breaks all the rules for search engine compatibility by using outdated Frames technology. Honestly! Frames are sooooo 1996. Search engines have always had trouble indexing frames-based sites and haven't gotten much better at it over the years. Search engine spiders generally only see the master frame-set (the page pulling all the frames together), not the individual frames. Consequently, there is no content for the search engine to index, apart from the content of the NoFrames element.

Because search engine spiders index sites by following links and because there are usually no links within the frame-set HTML code, search engines are usually unable to index frames-based sites beyond the home page. If you insist on using such dated design technology, you absolutely need to give the search engines a juicy No Frames tag to suck on. Yours currently states:

"Sorry, the Little Steppers website is only veiwable (sic) through a frame compatible browser. Please upgrade to a frame compatible browser."

What does that tell a search engine about your business? Zero, zilch, zip. The only reason your site was ranking on Google for "childminder milton keynes", was because you used that phrase within your Title Tag.

Ideally, a short keyword-filled description of the site should be included in the NoFrames element, as well as a link to the site map or main links page, which acts like a signpost for search engines so they know where to find and index further site content. Danny Sullivan wrote a terrific tutorial about how to optimize frames-based sites. Make sure you read it. But if you are really serious about optimizing your wife's site for search engines, you'll update the technology to a design that is search engine friendly.

To fix your immediate problem, here's what I suggest:

• Verify your site with Google's Webmaster Tools, check for site errors and study Googlebot's indexing patterns.
  
  
• Create and upload an XML sitemap to Google Sitemaps and study the results via Webmaster Tools. See the free sitemap creator that I recommend.
  
  
• Use Danny's tutorial to reestablish the Frames Context for each page on your site so search engines can jump from one page to the next when indexing.
  
  
• Give Frames the flick!
  

Oh and one last thing, you are using keyword-stuffed ALT IMG tags on your home page. That is a real Google no no. Better nip that in the bud before you DO get penalized.

Add to: Digg | Del.icio.us | Ma.gnolia | RawSugar | Reddit

Subscribe via: Yahoo Feeds | Feedburner | Technorati | Bloglines

Q and A: I've cleaned up the site spam so why isn't our home page being indexed by Google?

Dear Kalena...

Hope you're happy and healthy. Thanks so much for your help last time. I cleaned the hidden text off the page and we got re-indexed quite quickly.

But now we seem to have dropped out again. Although some of our subsidiary pages are indexed, the homepage does not seem to be, unless I'm searching incorrectly.

Any ideas?

Best wishes
Robin
www.breatheonline.com/

Kalena's Answer:

Dear Robin

Ok, a couple of things:

1) You obviously have two domains pointing to the same content (www.breatheonline.com and www.breatheyoga.co.uk). If you view the cache of your .com home page, you'll see that Gooogle produces the cache of the .co.uk home page. It seems that Google has decided that your UK domain is the main one and seems to be caching only that site.

2) A "site:URL" search for each domain shows that 22 pages from your .co.uk domain have been indexed while 4 pages from your .com domain have been indexed. To have Google index two domains with identical content is a dangerous thing because one will usually be suppressed and you rarely have the control over which one. I'm not sure about your DNS and IP setup, but you need to decide which domain you wish to promote in search engines and park the other domain to the same IP as the main one. You can also inform Google which site is your main domain via the Sitemaps Protocol. Check out the free XML sitemaps creator that I recommend.

3) Google is indexing and caching your home page at www.breatheyoga.co.uk just fine, from what I'm seeing. You simply need to do a search for your full URL. The last cache of the page was taken as recently as 26 November.

4) Once you've sorted out your domain issue, it might be a good idea to prepare an XML sitemap and submit it to Google Sitemaps as explained in Google's Webmaster Tools area.

5) Some of your incoming link partners are pointing to the .com site while others point to the .co.uk site. This is diluting your link popularity. Decide which domain is more important to promote via search engines and ask all your link partners to change their links to point to that site. Make sure you do the same thing with all your internal site links.

---------------------------

[If you found this post helpful, you might benefit from downloading our free Search Engine Optimization lesson]

Google Sitemaps Gains Support from Yahoo and MSN

The uptake of Google Sitemaps by webmasters has resulted in an unexpected alliance between Google, Yahoo and MSN. The three search giants have joined forces to launch Sitemaps.org, a new webmaster protocol to encourage a web standard for sitemap creation and submission.

Included in the protocol is:

• an XML schema format recommended for sitemaps
• definitions of the various XML tags
• detailed instructions for sitemap creation
  
• sitemap validation tools
  
• instructions for alerting search engine crawlers about your sitemap

Looks like paid trusted feed services are dead and buried! Good riddance. And if you're looking for a free XML sitemap creator, check out this one.

Q and A: Has our site been penalized by Google?

Dear Kalena...

An excellent website! have found many useful articles in the "ask kalena" section.

I was curious as to whether our website www.simondelasalle.com has been penalized for some reason. We have recently changed our navigation from image based to text links to increase the amount of keywords on the page. Is it possible this website has been "sandboxed"?

It looks like Google's indexing of the site is weak compared to MSN. Any feedback you could provide would be greatly appreciated.

Thanks for your time and have an excellent day.

Simon

Kalena's Answer:

Dear Simon

Thank you for the coffee! The caffeine is greatly appreciated.

I've had a look at your site and the first thing that strikes me is that there is no Google cache for many of the pages. Your home page has a healthy Toolbar PageRank of 5 out of 10 so there is no penalty. Next, I entered your URL into Google's new Site Status Tool. Bingo! I found two problems:

1) Googlebot last successfully accessed your home page on Jul 29, 2006 (which explains the caching issue).

2) Google does not know about all the pages of your site.

I'm not sure when your major re-design occurred, but if it was around the end of July, then it's highly likely the site has since been sandboxed, as you guessed. Best course of action is to prepare and submit an XML sitemap to Google Sitemaps using a free XML sitemap creator and wait till Googlebot travels your way again. After verifying your site with Google, you can check if Googlebot struck errors during his last crawl.

Good Luck!

---------------------------

[If you found this post helpful, you might benefit from downloading our free Search Engine Optimization lesson]

Q and A: If dynamically generated pages don't technically exist, how do they get indexed?

Dear Kalena...

I am puzzled about dynamically generated pages. Since they don't actually 'exist', how do they get indexed? (I know you can mod_rewrite their urls, but they still don't 'exist' anywhere, right?)

AP

Kalena's Answer:

Dear AP

Well technically, no they don't exist until the page is "called". But from what I understand, a search robot, when indexing a site, can create them on the fly, just like a browser can. That's how they get indexed and links between them are followed.

Most search spiders/robots won't index more than a few pages of a dynamic site because of the unknown number of pages they represent. That's why services such as Google Sitemaps and Yahoo Site Explorer have developed - so you can "tell" a search engine how many URLs your site has - dynamic or otherwise. You should create an XML sitemap and submit it to both services.

Mod_rewrite is good, but prob not necessary now that engines have better technical ability to process dynamically generated content.

---------------------------

[If you found this post helpful, you might benefit from downloading our free Search Engine Optimization lesson]

Q and A: How do you use xml-sitemaps?

Dear Kalena...

You have a testimonial listed on xml-sitemaps.com. Do you use it to produce sitemaps for your clients? Do you load the PHP s/w on each of them, or run it remotely? Did it require a special license to run it on non-personal sites?

Chris

Kalena's Answer:

Dear Chris

Yes, I use xml-sitemaps.com to produce sitemaps for my clients. I use the site's own generator to produce the XML files for each client and then copy the files to my PC, before uploading to Google Sitemaps.

As far as I know, you don't require a special license to use the software for business sites, but if you've got a site exceeding 500 pages, you'll need to buy their stand-alone software. It's only USD14 and well worth the investment.

---------------------------

[If you found this post helpful, you might benefit from downloading our free Search Engine Optimization lesson]

Q and A: Why aren't all my site pages being cached in Google?

Dear Kalena...

Our website is in the top 100,000 websites (Alexa Rank) and i find that Googlebot has crawled almost all our webpages but when i click on "show cache" in Firefox, it is not showing for a lot many pages.

I hope that you would have seen such incidences earlier and would like to know the reason why it happens and what can be done to better it?

Waiting on you!
Rishi

Kalena's Answer:

Dear Rishi

I've checked a few pages of your site and most are showing up as recently cached in Google. There were a couple uncached and quite a few showing a very old cache.

Usually caching errors are caused by:

1) Googlebot abandoning the indexing of your site due to a problem it struck in your code.
2) Googlebot abandoning the indexing of your site due to reaching the maximum site data quota set by Google.
3) A no-cache tag appearing in the code on your page.
4) Googlebot avoiding certain areas of your site by obeying the contents of your Robots.txt file
5) A lack of internal / external links pointing to a particular page on your site (Googlebot being unable to find it).
6) Failure to include all site pages in your navigation structure and/or your Google Sitemap.
7) Incorrect formatting or uploading of your Google Sitemap XML file. Try creating an XML sitemap from scratch.

You should check all these possibilities and monitor your site's indexing via Google Sitemaps.

Hope this helps!

---------------------------

[If you found this post helpful, you might benefit from downloading our free Search Engine Optimization lesson]

Q and A: Why isn't Google indexing my site?

Dear Kalena...

My site has not been indexed in Google for the last one month. I submitted articles, blogs to different sites and submitted links to 100 directories. Could you please tell me what are the strategies I should adopt to get listed in Google? My site got indexed in Yahoo and MSN.

J

Kalena's Answer:

Dear J

It would have helped if you'd included your site URL in your question! Without that, I can only guess at the problem. Here are my best guesses:

1) If your site has only recently been launched, you are probably experiencing Google's aging delay for new sites, which can last up to 9 months.

2) You say you've submitted "links to 100 directories". If this has been done as part of some dodgy link scheme, then Google may have penalized you for it. Brush up on why here.

3) If only a few pages on your site have been indexed, your site's navigation may be preventing or discouraging Googlebot from finding all your content. Create a search engine friendly navigation structure and prepare and upload an XML sitemap to Google Sitemaps.

---------------------------

[If you found this post helpful, you might benefit from downloading our free Search Engine Optimization lesson]

Q and A: Why is Google only indexing my home page?

Dear Kalena...

I published a site a few months ago. The site is http://www.maxwell3.com. I used the Add URL page to add my site to the Google index. Google picked it up a few weeks later.

Unfortunately, it appears that only the homepage is being indexed and I can't seem to determine why. I even created a SiteMap that holds information about the pages in the system and registered it at Google w/o any problems. It's like Google just decided to only use the default page.

The site map is located at http://www.maxwell3com/maxwell3_sitemap.xml

I would love 2 cents of advice. I was last scanned by the googlebot last month and it comes thru about once a week.

Sam

Kalena's Answer:

Dear Sam

I've had a look at the site and the first thing that struck me was that it hasn't been cached by Google. This indicates a problem of some kind, or perhaps somewhere in the code or Robots.txt file you have specifically blocked Googlebot from indexing the site?

I noticed you used MS Visual Basic to build the site - have you made sure the code validates with W3.org? My Google Toolbar is showing a PageRank of zero out of 10, which is not unusual for a new site, but the non-caching issue bothers me.

Also, I can't get your XML sitemap to load at the address you provided above. Make sure the URL is correct and then submit your sitemap to Google Sitemaps.

---------------------------

[If you found this post helpful, you might benefit from downloading our free Search Engine Optimization lesson]

Q and A: Which is the best Google Sitemap creator to use?

Dear Kalena...

I have been searching for an online tool that will create an XML sitemap for Google automatically and I've found quite a few. But they all seem quite complicated and some of them charge a fee.

Do you have a favorite or can you recommend a good one to use, preferably a free one?

thanks
Akita

Kalena's Answer:

Dear Akita

Funny you should mention this. To create my Google Sitemaps, I had been using SiteMaps Pal, but I haven't been completely happy with it because it seemed to skip sub-directories and pages quite often.

But just this week I came across the free XML Sitemaps Generator and I'm now a big fan. It not only trawls through ALL levels of your site, but it gives you a running count of pages, provides a text-based URL list and a HTML sitemap you can import straight into your site, plus it generates the XML file for you in both compressed and un-compressed versions. Sweet! If only the free version gave you the ability to manually edit page priority and index dates and re-generate the XML, I'd be an even bigger fan (hint hint!)

There is also a low-cost script-based version for sites with thousands of pages. It can be set up to index your site on a regular basis and produce an updated XML file for automatic upload to your site via FTP. The XML Sitemaps Generator is also included in Google's List of 3rd Party Plugins for Sitemaps.

A: Will other search engines index the XML sitemap I create for Google Sitemaps?

Hi John

My first response based on what I've recently learnt about the XML Sitemap Protocol, was that if you link to your XML sitemap from a page on your site already indexed by a search engine crawler, then yes, it should get indexed.

HOWEVER, Yahoo actually sells an XML indexing service for large and dynamically generated sites called Search Submit. This service has an annual fee and a cost-per-click pricing structure so it is highly doubtful that they are going to index and include your XML sitemap for free when they are already charging for this service.

But I wouldn't worry too much - I believe that paid inclusion services like Search Submit will be phased out as XML becomes the web standard for content and we move closer to a true Semantic Web. In the meantime, I wouldn't bother creating separate site maps for separate engines, just like I wouldn't bother creating unique pages for unique engines. Make a single, accurate XML sitemap and stick with it. Do what makes sense from a user's perspective and you can't go wrong!

Q: Will other search engines index the XML sitemap I create for Google Sitemaps?

Dear Kalena...

I'm interested in the new beta feature from google, the xml site map.

Do other search engines also use this new site map structure? I mean, I set up my sitemap.xml that google will crawl. Will yahoo.com/inktomi also crawl the same page or do I have to create a separate site map for each search engine?

Thank you.

John