Difference between revisions of "WhoisRefresh"

Create Domain Page Specification

• Create adds the domain to the electric sheep queue
• The electric sheep pull the domain from the queue and begin nibbling information about the domain

Refresh Specification

When a domain page is viewed:

• Do we have a contact info section that has been human edited? If so do nothing
• Else
  • Pull new contact information and dump into contact info section
  • Overwrite if contact info section already exists
  • Protect with the "address" tag
  • Lookup lat/longitude via google api
  • Mashup lat/longitude with google maps

We're done when

• Clicking through random pages ...

Next

• Get the tests failing
• Check the tests in
• Research our address extraction strategy: (HMMs, MDL)
• Update Brandon's test environment to include the current test database

Problems

• The formats across registrars are inconsistent
• Different countries use different conventions for their addresses
• .org registry limits requests to 4/minute on port 43. (Web-based queries limited to 50/minute)

Synopsis

When a page is viewed we check to see if we have the contact info for it. If not, we get the whois record for it and use that to populate the page. In the process we obfuscate the email and postal addresses to make it harder for spammers. We also grab the lat/long for the postal address so that we can display a map.

AcceptanceTest

Look at 50 pages that we know don't have contact info and verify that the contact info is coming in.

Background

• rfc3912 the current protocol specification
• Domain Statistics by TLD
• 100 oldest dot com domains
• Registrar Stats

The ranking and percentages come from http://populicio.us/toptlds.html and are at least as stale as November 2006.

Information We Need

• Date of lookup
• Registrant Address
• Admin Address
• Phone
• Email

Question: do we need both the registrant and admin addresses or is one enough? In the past we've only used one. - Ray | talk

Next

• Get gpMakeImage working so that tests pass
• Send email
• Given an address get a lattitude/longitude
• Obfuscate address

<address>fe565342385cbcce7cb35b486876b8d5</address>
 .
 .
 .
<address asgraphic="HASH" latitude="" longitude="" error="This is where an error message goes">
...This address is displayed as a graphic to make 
it more difficult for web robots to harvest it.  
If you would like to change the address that is 
displayed, simply replace these instructions 
with the new address and then save the page...
</address>

Latitude/Longitude

Need a table so that we can associate one or more lat/long pairs with a page.

Attemp to Fix Freetype

http://lists.freedesktop.org/archives/fontconfig/2006-October/002501.html

When you get the followin error

dyld: lazy symbol binding failed: Symbol not found: _FSPathMakeRef
  Referenced from: /usr/local/lib/libfreetype.6.dylib
  Expected in: flat namespace

Fix your freetype to work with MacOSX fonts

  freetype-config:
  - libs="-lfreetype -lz"
  + libs="-lfreetype -lz -Xlinker -framework -Xlinker CoreServices -Xlinker -framework -Xlinker ApplicationServices"

  freetype2.pc:
  - Libs: -L${libdir} -lfreetype -lz
  + Libs: -L${libdir} -lfreetype -lz -Xlinker -framework -Xlinker CoreServices -Xlinker -framework -Xlinker ApplicationServices

Then rebuild php from scratch (make clean; make; make install)

Turns out this was a false start ... didn't work