Difference between revisions of "Category:ErrorLog"

(Google caching issue: Two possible solutions)
(Google caching issue)
Line 154: Line 154:
  
 
If you want the current versions of adult-content domain pages to be indexed, then treat Googlebot and other robots as if they were already logged in and had clicked the "view adult content" button.  --[[User:Iapetus|Iapetus]] 10:47, 4 November 2007 (PST)
 
If you want the current versions of adult-content domain pages to be indexed, then treat Googlebot and other robots as if they were already logged in and had clicked the "view adult content" button.  --[[User:Iapetus|Iapetus]] 10:47, 4 November 2007 (PST)
 +
 +
Iapetus - who is your comment directed at?
 +
 +
I, as owner of the adult sites, have no way of changing aboutus.org's http response codes to google's crawlers. It is up to the admin of aboutus.org to allow google to crawl and index the latest versions of the page, given that they happily allowed them to crawl the old versions of the page in the first place, causing google to cache details which we really didn't want exposed in this way.
 +
 +
There are only two solutions. Either:
 +
 +
a) allow known crawlers to index all aboutus.org pages, whether they are about adult sites or not (thus allowing the crawler's cache to be updated to remove any old information), or
 +
 +
b) delete the pages, which would cause the original cached page to be dropped from the index.
 +
 +
The current behaviour of returning a 302 temporary redirect basically tells the crawler that the pages have not changed since they last saw them (which means they indefinitely cache the old data), and to basically ignore the "login form", since it is "temporary" redirect (http 302). Returning a 401 provides no guarantee that the old page would be dropped from a crawler's cache. Only a 404 would be close to guaranteeing that.
 +
 +
I would appreciate it if someone from aboutus.org would respond to this. We have over 400 domains affected by this issue, and it is causing significant stress for a number of people in our team. Please, please please do something about it.
  
 
== spam filtering ==
 
== spam filtering ==

Revision as of 00:27, 5 November 2007

Also see BugSquashing and FireFighting

This category currently contains no pages or media.