Difference between revisions of "WhoisRefreshRunRefresh"

OurWork

This is part of the WhoisRefresh project. Run over all pages pertaining to website information that have 0 human edits and get and insert fresh whois information. FOr example, www.aboutus.org/facebook.com.

Steps to DoneDone

• Find out how many pages this would hit - approx 7,659,827
• modify one page
  • Contact information:
    • Contact name
    • Contact email (protected)
    • Street Address (protected)
    • City, State/Province, Postal Code
    • Geocode for maps location
    • Contact Phone Number
    • Contact Fax Number
    • Wiki comment for as-of date of whois info

• design a process to modify all pages
  • Find the page to update (slow page walk)
  • Get and Parse fresh Whois information
    • Since we can only query for a given domain once, we should probably keep the full response around in case we parse it wrong
  • Change the address on the page
    • WikiTransformer can be gutted and used for this
    • When save the new page revision, make sue to suppress creation of the linksUpdate job

Anything having to do with page creation bot is out of scope for this particular project and should be refactored away to a more apporpriate place.

Status

Friday Nov 23, 2007

• WhoisInserter is now also able to do the following:
  • Alternate between enom and ni to fetch whois records if nothing found locally.
  • Only touch domain pages.
  • Only hits those pages with zero human edits in the contact section.
• Brainstorming the test cases (work in progress)

Things to do Monday

• Decide on the wiki text if contact info is not parsed. Currently obfuscates all of it in an address tag.
• Do major work on test suite.

Looks good. Do you think this will be running the first 100 by the end of today or possibly tomorrow? --Brandon 21:23, 25 November 2007 (PST)

Thurs Nov 22, 2007

• Started refactoring whois_inserter script to incorporate the following:
  • Fetch whois records from domain registrars if not found locally. This also includes alternating between ni and enom. As a side effect, the fetched record is stored in our local system. (done for ni, enom refactoring pending)
  • Hit only those pages with zero human edits. As an improvement, might want to relax this condition and update all pages with zero human edits in contact section. Performance Implications need to be taken into account though.
• Improve Test Suite for whois-inserter.

Things to do tomorrow:

• Finish enom refactoring
• Handle human edits issue.
• Spend some time on test suite
• If new feedback comes out of staging server demo, try to understand and possibly incorporate them.

Excellent! Last time we chatted it was next week for first 100 records will be run on live. Is that still what you guys are thinking? --Brandon 22:16, 22 November 2007 (PST)

Wed Nov 21, 2007

WhoisRefreshRunRefresh is able to do the following:

• Insert new Contact Section if no such section exists.
• Replace Contact Section contents with new whois data.
• Street Address and Email is obfuscated.
• City, State, Country, Postal Code and phone number are plain text. (tweaked today)
• Proxy Registration is detected and appropriate message is inserted into contact section rather than the proxy address, etc. (tweaked today)
• Google Maps is enabled.
• Geo-Coordinates are embedded in the page_html and are no longer show up in edit textbox. (done today)
• Some scripts written for importing/exporting table records in chunks from prod. database (work in progress)

Things to do:

Work on the import/export scripts.

Get feedback of the current status of WhoisRefreshRunRefresh from Ray and others. Tweak functionality if required.

• I'd like you guys to run the bot on 100 pages on the live site today. --Brandon 21:27, 21 November 2007 (PST)

Old Stuff

Currently, PageCreationBot, PageScrapeBot and WhoisParsing together generate a page as follows:

• PageCreationBot creates a domainbox template and uses the thumbnail tag to imbed the thumbnail into itself. PageScrapeBot fetches thumbnail from alexa for a given domain name and stores it locally in a pre-defined directory. We need to figure out the mechanism behind thumbnail tag. i.e. how does it locate a particular thumbnail image. Corresponding to this, we need to provide a mechanism to put the fetched thumbnail image so that the mediawiki can locate it.
• PageCreationBot creates a section named 'Logo' where it puts the logo that PageScrapeBot fetched from the site itself. The logo is inserted into the page using the wiki Image tag. Need to find a better way of doing this ala thumbnails.
• Next, the PageCreationBot creates a description section which is filled with description fetched from alexa followed by any about us text extracted from the site. (The aboutus text is contained in a sub-section)
• Related and Inlinking Domains sections are populated. Related Domains are fetched from google, whereas sites linking in are fetched from alexa.
• Keywords fetched from meta tags in the home page are placed in a seperate section 'Keyword'
• Categories fetched from alexa are used to create categories that the page belongs to using the categories tag.
• Contact info is to be fetched from contact table that is populated by WhoisParsing and put in it's own section.

Things to do:

• Need to embed logo and thumbnails in same manner.
• Understand mechanism behind thumbnail tag.
• Devise a mechanism to detect registration by proxy. Decide on plan of action if proxy registration encountered.
• Decide on course of action based on the status of domain. i.e. parked, locked.

Possible Scenario

• One of our valued clients enters the following url : http://www.aboutus.org/i_am_not_on_aboutus_yet.com
• Unfortunately, this page currently does not exist in our db.
• The default wiki behavior is to return a newly created empty page to the client.
• Surely, we can do better.
• So we try to make a best-effort autogenerated page
• Our top-level glue will first call PageScrapeBot's process method with this new domain as its argument. This will result in domain-specific information being dumped into database.
• It will then do the same to fetch whois information by calling WhoIsParsings' parse method.
• At the end of this process, the db is populated with relevant details regarding this domain.
• Once loaded with all this amunition...it will fire a request to pagecreationbot to create this new page using relevant data from db.
• And voila, we have a newly created page for our valued client.