Difference between revisions of "WhoisRefreshRunRefresh"

OurWork

Run over all pages pertaining to website information that have 0 human edits and get and insert fresh whois information. FOr example, www.aboutus.org/facebook.com.

Steps to DoneDone

• Find out how many pages this would hit - approx 7,659,827
• modify one page
  • Contact information:
    • Contact name
    • Contact email (protected)
    • Street Address (protected)
    • City, State/Province, Postal Code
    • Geocode for maps location
    • Contact Phone Number
    • Contact Fax Number
    • Wiki comment for as-of date of whois info
• design a process to modify all pages

Possible Scenario

• One of our valued clients enters the following url : http://www.aboutus.org/i_am_not_on_aboutus_yet.com
• Unfortunately, this page currently does not exist in our db.
• The default wiki behavior is to return a newly created empty page to the client.
• Surely, we can do better.
• So we try to make a best-effort autogenerated page
• Our top-level glue will first call PageScrapeBot's process method with this new domain as its argument. This will result in domain-specific information being dumped into database.
• It will then do the same to fetch whois information by calling WhoIsParsings' parse method.
• At the end of this process, the db is populated with relevant details regarding this domain.
• Once loaded with all this amunition...it will fire a request to pagecreationbot to create this new page using relevant data from db.
• And voila, we have a newly created page for our valued client.