Difference between revisions of "Rewrite PageScrapeBot"
Arif Iqbal (talk | contribs) |
Arif Iqbal (talk | contribs) |
||
| Line 28: | Line 28: | ||
== Steps to get to [[DoneDone]] == | == Steps to get to [[DoneDone]] == | ||
| − | + | === Create a web server that responds to http requests === | |
| − | + | ||
| − | + | === Find dmoz categories of the website === | |
| − | + | ||
| − | + | === Site crawler that finds logo, and meta tags on the front page === | |
| − | + | * Metatags extraction | |
| + | * language of the page | ||
| + | * logo | ||
| + | * Is this page forwarded to some other domain, hosted, or parked | ||
| + | * Location of home page for the site. | ||
| + | |||
| + | === Capture alexa information === | ||
| + | like thumbnail, whois, related domains, site rank, status, adult content etc | ||
| + | |||
| + | == Gather Information == | ||
Ruby Libraries | Ruby Libraries | ||
* mechanise mechanize | * mechanise mechanize | ||
Revision as of 06:34, 3 September 2007
What (summary)
The current PageScrapeBot is part of a large Java tomcat process running on our database master. We want to rewrite it in ruby and move it off of the database master. Rewrite PageScrapeBot is one Task in the larger WhoisRefresh Project.
The bot currently goes and gets the whois record from alexa, crawls the site looking for a logo, meta information for category guessing, looks for "aboutus" information on the site and several other things, then dumps all of that information into the database from where it is collected and reassembled by the PageCreationBot.
Why this is important
- Gives us mastery over our technology. We can change it easily (e.g., add amazon books that mention the domain), or adapt it for a different problem.
- Allows us to turn off Apache and the large Java tomcat process on our database master.
DoneDone
Given a domain, the bot collects the following information and adds it to the database
- A snippet about the domain, gathered from the site itself
- A thumbnail portrait of the front page
- A best guess at the logo for the site
- A list of related domains
- The address (or perhaps addresses) for the site along with citation from where the address came
- Meta information about the owner, email, alexa rank, language of the site, whether it is online or not, and whether it is classified as adult by alexa or not
- dmoz categories that contain the site
- metainformation about the domain (for example is it an idn domain)
- location of the homepage for the site (www.example.com vs example.com)
- incoming links to the frontpage
- keywords from the meta information on the frontpage
- the logo for the website
- all the meta information ... for example, is it parked? does it forward? is it online? is the registration protected?
- sites that it links out to
Steps to get to DoneDone
Create a web server that responds to http requests
Find dmoz categories of the website
Site crawler that finds logo, and meta tags on the front page
- Metatags extraction
- language of the page
- logo
- Is this page forwarded to some other domain, hosted, or parked
- Location of home page for the site.
Capture alexa information
like thumbnail, whois, related domains, site rank, status, adult content etc
Gather Information
Ruby Libraries
- mechanise mechanize
- hpricot
- topsoil web spider
