How often does google crawl sites now days?

[Staples]

Many years ago I had a website that got a few hundred hits a day. 90% of the visitors who were not returning visitors were referred by Google. I have also had another website however I have never done anything with it until recently.

I look in google's cache of my undeveloped but sitting for years site and the date of the crawl was a year ago. This was the undeveloped site that said coming soon on the front page and had no links. I remember google's crawl being every two month back in 2000 but is the date I see really indicative of how unoften google crawls sites now days? Maybe it is that if the site seems to be undeveloped, google only crawls it annually or less?

Does anyone look at how often google crawls their site?

 

[devourment77]

I get crawled almost every day, I use php counter 7 to log everything, but google does prefer to crawl certian things more than others.. or should I say prefers to index certain things more than others..

google hates url get variables like something.php?id=2

I got around this on my site with a simple .htaccess file to rewrite the url to look like something/2/

here is a good example on my site:

http://www.brandon-san.com/archiveNews.php?id=39 and http://www.brandon-san.com/article/39/ are the same page (both dynamic), but I just use htacess to rewrite the url and link to the /article/39 link on my site and not the url with get variables.

also page rank will help determine how much you get crawled. Right now my site (brandon-san.com) has a page rank of 3 and I get crawled usually once a day, but my friend who owns www.badice.com has a page rank of 5, and he gets crawled almost every other hour.

These are just some things to look into. I will PM you some more info.

 

[Staples]

That is nice to know. My old site didn't use query strings except in one place and I was able to get very high on the google rankings if certain words were searched. This new site does use query strings. Maybe I will fix that to use /fdsaf/dasf/ instead. Here's hoping that google crawls the site again sometime in the near future. Any ideas of how I can force it to crawl sooner than its timer dictates? (which could be a year away for all I know).

http://64.233.167.104/search?q=cache:2LLjbGoWUGIJ:rocketbulletin.com/+&hl=en

My cached page from May 20, 2004 looks nothing like the one now.

 

[devourment77]

get linked on a high profile site/high page rank site... or just get links out their.. for instance link up with people that have similar sites or if you have friends with sites, just have them add you to their links page.. all it really takes is for one googlebot to find the link, then you will be index/crawled.

hope this helps.

Also.. google likes h1/h2/h3 tags.. it will look for these more than meta tags.. so use h2 (styled with CSS) or h3 (styled with CSS) tags for titles/page titles on your site, google will look for these first.

 

[HorsePunchKid]

My old site didn't use query strings except in one place and I was able to get very high on the google rankings if certain words were searched. This new site does use query strings. Maybe I will fix that to use /fdsaf/dasf/ instead.

I really seriously doubt that Google cares that much whether you pass information in a query string or not. I run a couple of relatively high-traffic, high-PageRank sites (narrowly focused in content) that rely heavily on query strings for essentially all of their content. I also personally use a PHP gallery system that previously used query strings, and Google had no trouble finding the content and ranking it reasonably.

I think it would still be preferable to avoid lots of query strings when possible. My new gallery setup uses rewrites; it's on my list of things to do for the sites I run at work. The only reason I prefer it is that it generally makes URLs easier to type and to communicate to people who aren't terribly familiar with complex URL formats. I also think there's some value in being able to hide the server environment to some extent. (For example, if I switch from CF to PHP, my URLs can stay the same.)

Anyway, if somebody has some reasonably conclusive evidence that Google disfavors URLs with query strings, I'd definitely be interested! I've looked a few times over the past couple of years and never found anything that wasn't more easily explained by other factors.

 

[devourment77]

"However the mere presence of a question mark in the URL will introduce a delay in google's indexing of the page. This from Mike Grehan's Interview with Daniel Dulitz of Google:

“So the mere presence of a question mark in the URL doesn't throw us for a loop at all. We will crawl those pages more slowly than we do with pages without question marks in the URL. Purely because they've identified themselves as being dynamic and we certainly don't want to bring anyone's site down because we're hitting the database too hard.”

Small sites will not need to worry much about this delay as it means your server is hit every few minutes, not a few times a second, but for larger sites this can slow down your site's inclusion into the index.

------------------------------------------------------

Your right.. google will crawl them, just not as quickly as they would a static looking site. I got this information from http://www.storebuilder.co.uk/article90.html

 

[specterious]

I've made several sites that use query strings and noticed that Google does index the information within pages generated by variables but does not link directly to them.

For example: Google will return results that are contained on a page from a URL like page.php?variable=value but if you look at the link it's actually just page.php. I notice this a lot when searching too. I get to a site from google results and have to dig a little more to find what google actually showed.

While changing URLs to work like /category/item/ probably won't increase traffic (at least from google), I think it should help direct the traffic.

 

[HorsePunchKid]

"However the mere presence of a question mark in the URL will introduce a delay in google's indexing of the page. This from Mike Grehan's Interview with Daniel Dulitz of Google...

Good information; thanks! You can find the original interview here. Interestingly, he goes on in that same statement to confirm my suspicions about how they treat CMS-backed sites. You'll want to read his comments yourself, but he basically says that query strings correlate with blogs, and blogs tend to have many views of the same information, which tends to inflate the PageRank undeservedly.

Going off-topic here, but that's my biggest frustration with Google. There needs to be some way to filter out blogs. Throwing "-blog" into the query helps a bit, but I'd prefer something more reliable.

I've made several sites that use query strings and noticed that Google does index the information within pages generated by variables but does not link directly to them.

I've definitely never had that problem. I've seen Google end up without a query string on a page that needs one, but in every case in my own work, it has been because I've accidentally let the broken link slip into the website somewhere.