Difference between revisions of "Rewrite PageScrapeBot"

(Removing rake test failure trace. The page was crying out for some beautification.)
Line 33: Line 33:
 
* robots.txt mock
 
* robots.txt mock
 
* Better [[AboutUs]] Information Extractor.
 
* Better [[AboutUs]] Information Extractor.
−
 
−
== Rake Test Failure: Trace ==
 
−
(in /home/dev/nimbus/compost-whois-refresh_backup)
 
−
/www/bin/ruby -Ilib:test "/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb" "test/unit/pagecreationbot_test.rb" "test/unit/slow_page_walker_test.rb" "test/unit/exporter_test.rb" "test/unit/geo_cache_test.rb" "test/unit/wikitransformer_test.rb" "test/unit/related_link_test.rb" "test/unit/outbound_link_test.rb" "test/unit/casespace_test.rb" "test/unit/revision_test.rb" "test/unit/page_test.rb" "test/unit/stats_mailer_test.rb" "test/unit/pagescraperbot/alexadataparser_test.rb" "test/unit/pagescraperbot/logo_finder_test.rb" "test/unit/pagescraperbot/page_scrape_bot_test.rb" "test/unit/pagescraperbot/website_crawler_test.rb" "test/unit/pagescraperbot/google_fetch_test.rb" "test/unit/alexa_test.rb" "test/unit/user_test.rb" "test/unit/edit_test.rb" "test/unit/recentchange_test.rb" "test/unit/inbound_link_test.rb" "test/unit/au_page_lookup_test.rb" "test/unit/intersector_test.rb" "test/unit/scrape_test.rb" "test/unit/domain_test.rb" "test/unit/category_test.rb" "test/unit/category_page_test.rb" "test/unit/bender_test.rb" "test/unit/mediawiki_parser_test.rb" "test/unit/job_test.rb" "test/unit/keyword_test.rb" "test/unit/graphic_test.rb" "test/unit/au_patrollers_test.rb" "test/unit/log_analyzer_test.rb" "test/unit/categories_to_keep_test.rb"
 
−
/www/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require': no such file to load -- flexmock/test_unit (MissingSourceFile)
 
−
        from /www/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
 
−
        from /www/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require'
 
−
        from /www/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in `new_constants_in'
 
−
        from /www/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require'
 
−
        from ./test/unit/geo_cache_test.rb:3
 
−
        from /www/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load'
 
−
        from /www/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load'
 
−
        from /www/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in `new_constants_in'
 
−
        from /www/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load'
 
−
        from /www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb:5
 
−
        from /www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb:5:in `each'
 
−
        from /www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb:5
 
−
/www/bin/ruby -Ilib:test "/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb" "test/functional/bender_controller_test.rb" "test/functional/consensus_controller_test.rb" "test/functional/au_blackhole_domains_controller_test.rb" "test/functional/category_tool_controller_test.rb" "test/functional/user_registration_test.rb" "test/functional/geocode_controller_test.rb" "test/functional/patrol_controller_test.rb" "test/functional/web_services_controller_test.rb" "test/functional/page_processor_controller_test.rb" "test/functional/ru_bot_controller_test.rb" "test/functional/pagebuilder_controller_test.rb"
 
−
Loaded suite /www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader
 
−
Started
 
−
EEEEE.E......./home/dev/nimbus/compost-whois-refresh_backup/config/../lib/page_creation_bot.rb:111: warning: don't put space before argument parentheses
 
−
.F.F................EE........
 
−
Finished in 0.0 seconds.
 
−
 
−
  1) Error:
 
−
test_create(AuBlackholeDomainsControllerTest):
 
−
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/models/user.rb:12:in `find_from_session'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:29:in `get_user'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:25:in `set_user'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:435:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:619:in `perform_action_without_benchmark'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/1.8/benchmark.rb:293:in `measure'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:83:in `perform_action'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `process_without_filters'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:624:in `process_without_session_management_support'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_management.rb:114:in `process_without_test'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:15:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:382:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:353:in `post'
 
−
    ./test/functional/au_blackhole_domains_controller_test.rb:52:in `test_create'
 
−
 
−
  2) Error:
 
−
test_destroy(AuBlackholeDomainsControllerTest):
 
−
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/models/user.rb:12:in `find_from_session'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:29:in `get_user'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:25:in `set_user'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:435:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:619:in `perform_action_without_benchmark'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/1.8/benchmark.rb:293:in `measure'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:83:in `perform_action'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `process_without_filters'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:624:in `process_without_session_management_support'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_management.rb:114:in `process_without_test'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:15:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:382:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:353:in `post'
 
−
    ./test/functional/au_blackhole_domains_controller_test.rb:65:in `test_destroy'
 
−
 
−
  3) Error:
 
−
test_index(AuBlackholeDomainsControllerTest):
 
−
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/models/user.rb:12:in `find_from_session'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:29:in `get_user'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:25:in `set_user'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:435:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:619:in `perform_action_without_benchmark'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/1.8/benchmark.rb:293:in `measure'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:83:in `perform_action'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `process_without_filters'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:624:in `process_without_session_management_support'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_management.rb:114:in `process_without_test'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:15:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:382:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:353:in `get'
 
−
    ./test/functional/au_blackhole_domains_controller_test.rb:26:in `test_index'
 
−
 
−
  4) Error:
 
−
test_list(AuBlackholeDomainsControllerTest):
 
−
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/models/user.rb:12:in `find_from_session'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:29:in `get_user'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:25:in `set_user'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:435:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:619:in `perform_action_without_benchmark'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/1.8/benchmark.rb:293:in `measure'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:83:in `perform_action'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `process_without_filters'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:624:in `process_without_session_management_support'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_management.rb:114:in `process_without_test'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:15:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:382:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:353:in `get'
 
−
    ./test/functional/au_blackhole_domains_controller_test.rb:32:in `test_list'
 
−
 
−
  5) Error:
 
−
test_new(AuBlackholeDomainsControllerTest):
 
−
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/models/user.rb:12:in `find_from_session'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:29:in `get_user'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:25:in `set_user'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:435:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:619:in `perform_action_without_benchmark'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/1.8/benchmark.rb:293:in `measure'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:83:in `perform_action'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `process_without_filters'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:624:in `process_without_session_management_support'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_management.rb:114:in `process_without_test'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:15:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:382:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:353:in `get'
 
−
    ./test/functional/au_blackhole_domains_controller_test.rb:41:in `test_new'
 
−
 
−
  6) Error:
 
−
test_get_categories(CategoryToolControllerTest):
 
−
ActiveRecord::RecordNotFound: Couldn't find User with ID=1
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1031:in `find_one'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1014:in `find_from_ids'
 
−
    /www/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:419:in `find'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/models/user.rb:12:in `find_from_session'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:29:in `get_user'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/config/../app/controllers/application.rb:25:in `set_user'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:456:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:435:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:638:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:438:in `call'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:637:in `call_filter'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:619:in `perform_action_without_benchmark'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/1.8/benchmark.rb:293:in `measure'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:83:in `perform_action'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `send'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:430:in `process_without_filters'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.rb:624:in `process_without_session_management_support'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_management.rb:114:in `process_without_test'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:15:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:382:in `process'
 
−
    /www/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/test_process.rb:353:in `get'
 
−
    ./test/functional/category_tool_controller_test.rb:22:in `test_get_categories'
 
−
 
−
  7) Failure:
 
−
test_builds_page(PagebuilderControllerTest) [./test/functional/pagebuilder_controller_test.rb:28]:
 
−
<true> expected but was
 
−
<false>.
 
−
 
−
  8) Failure:
 
−
test_lowercase_works(PagebuilderControllerTest) [./test/functional/pagebuilder_controller_test.rb:40]:
 
−
<true> expected but was
 
−
<false>.
 
−
 
−
  9) Error:
 
−
test_builds_page(WebServicesBuildDomainPageTest):
 
−
NameError: uncaught throw `failure'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/test/functional/web_services_test_helper.rb:5:in `throw'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/test/functional/web_services_test_helper.rb:5:in `compost'
 
−
    ./test/functional/web_services_controller_test.rb:145:in `test_builds_page'
 
−
 
−
10) Error:
 
−
test_lowercase_works(WebServicesBuildDomainPageTest):
 
−
NameError: uncaught throw `failure'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/test/functional/web_services_test_helper.rb:5:in `throw'
 
−
    /home/dev/nimbus/compost-whois-refresh_backup/test/functional/web_services_test_helper.rb:5:in `compost'
 
−
    ./test/functional/web_services_controller_test.rb:158:in `test_lowercase_works'
 
−
 
−
44 tests, 108 assertions, 2 failures, 8 errors
 
−
/www/bin/ruby -Ilib:test "/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/rake_test_loader.rb" "test/integration/dummy_test.rb"
 
−
Command failed with status (1): [/www/bin/ruby -Ilib:test "/www/lib/ruby/ge...]
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:719:in `sh'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:726:in `call'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:726:in `sh'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:805:in `sh'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:740:in `ruby'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:805:in `ruby'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/testtask.rb:117:in `define'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:823:in `verbose'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake/testtask.rb:102:in `define'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:392:in `call'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:392:in `execute'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:392:in `each'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:392:in `execute'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:362:in `invoke'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:355:in `synchronize'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:355:in `invoke'
 
−
/www/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/tasks/testing.rake:45
 
−
/www/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/tasks/testing.rake:43:in `collect'
 
−
/www/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/tasks/testing.rake:43
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:392:in `call'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:392:in `execute'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:392:in `each'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:392:in `execute'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:362:in `invoke'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:355:in `synchronize'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:355:in `invoke'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1739:in `top_level'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1739:in `each'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1739:in `top_level'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1761:in `standard_exception_handling'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1733:in `top_level'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1711:in `run'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1761:in `standard_exception_handling'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/lib/rake.rb:1708:in `run'
 
−
/www/lib/ruby/gems/1.8/gems/rake-0.7.3/bin/rake:7
 
−
/www/bin/rake:16:in `load'
 
−
/www/bin/rake:16
 
−
rake aborted!
 
−
Test failures
 
−
 
−
 
  
 
== '''Steps to get to [[DoneDone]]''' ==
 
== '''Steps to get to [[DoneDone]]''' ==

Revision as of 12:45, 25 October 2007

Rating: 0 - 0 votes

Company Name

Company Contact

Page Type

This page is about a company.

OurWork Edit-chalk-10bo12.png

Contributors: Athar Hameed, Ali Aslam, Arif Iqbal, Hassan Javeed, Laiq Jafri, Mohammad Ghufran, Umar Sheikh

What (summary)

The current PageScrapeBot is part of a large Java tomcat process running on our database master. We want to rewrite it in ruby and move it off of the database master. Rewrite PageScrapeBot is one Task in the larger WhoisRefresh Project.

The bot currently goes and gets the whois record from alexa, crawls the site looking for a logo, meta information for category guessing, looks for "aboutus" information on the site and several other things, then dumps all of that information into the database from where it is collected and reassembled by the PageCreationBot.

Why this is important

  • Gives us mastery over our technology. We can change it easily (e.g., add amazon books that mention the domain), or adapt it for a different problem.
  • Allows us to turn off Apache and the large Java tomcat process on our database master.

DoneDone

Given a domain, the bot collects the following information and adds it to the database

  • A snippet about the domain, gathered from the site itself
  • A thumbnail portrait of the front page Done with alexa, should we consider to capture it from nameintel?
  • A best guess at the logo for the site
  • A list of related domains
  • The address (or perhaps addresses) for the site along with citation from where the address came
  • Meta information about the owner, email, alexa rank, language of the site, whether it is online or not, and whether it is classified as adult by alexa or not
  • dmoz categories that contain the site categories are fetched from alexa and not directly from dmoz
  • Metainformation about the domain (for example is it an idn domain)
  • location of the homepage for the site (www.example.com vs example.com)
  • incoming links to the frontpage
  • keywords from the meta information on the frontpage
  • all the meta information ... for example, is it parked? does it forward? is it online? is the registration protected?
  • sites that it links out to only for the home page
  • About Us Text extraction

Things that still need some minor effort

  • Rails multi threading issues kills db commands.
  • Facebook does not respond saying unrecognized browser.
  • robots.txt mock
  • Better AboutUs Information Extractor.

Steps to get to DoneDone

Improve AboutUs Information Extractor

Improve Logo Finder

  • Find Logo from site's AboutUs page
Currently, the bot tries to extract the site's logo from the main page. A possible pitfall with this approach is that the main page will contain lots of extra html text and parsing for logo image tag could be time consuming and not correct in some instances.
The idea is to first find the logo from the aboutus page if it exists. The rationale is that the html text that makes up the site's aboutus page will be significantly shorter and less complex than that of the main page.

Improve Test Cases

  • Update test cases for the following:
    • Google Data Parser
    • Alexa Data Parser
    • Website Crawler (including logo & aboutus text tests)
    • Integration Test Suite for PageScrapeBot

Schema

  • Redefine the schema.
  • Put it on the page creation bot.
  • Write migrations for the new schema
  • Verify the migration using console

Get compost up and running on all dev machines

  • Get compost up and running on everyone machines.
  • Make sure that the commits are going in the repository and database/models.


Fetch Domain Info from Google

  • Fetch Related links
    • Indexing issues because of threading.
    • proper handling of the case when more results demanded than available
    • Test Case: Domain name has no related links (2)
    • Test Case: Domain name has fewer related links than requested (2)
    • Test Case: Scrape all related links from the returned Google search page(s) (2)
    • Test Case: Scrape a subset of related links from the returned Google search page (2)
    • Performance tests (2)
  • Fetch Incoming links
    • Test Case: Domain name has no incoming links (2)
    • Test Case: Domain name has fewer incoming links than requested (2)
    • Test Case: Scrape a subset of incoming links from the returned Google search page (2)

Site crawler that finds logo, and meta tags on the front page

  • Pass Rake test
  • Write test cases for websites containing logos, aboutus
  • Parse aboutus pages of website
  • Fetch Logo
  • Metatags extraction.
    • Description (5)
    • Title
    • Keywords
    • language
  • Capture External links.
  • Capture Internal links.
  • Domain Information
    • forwarded/redirected
    • parked
  • Location of pages
    • home
  • Respect robots.txt and robots info in Metatags.

Capture alexa information

  • Rewrite code to push data in new database schema.
  • sites linking in
  • contact information
  • related domains
  • site rank
  • status
  • adult content
  • thumbnail
  • Multi threading support
  • Test cases
    • Connectivity problems(3)
    • Invalid domain(5)
    • Domain with some information
    • Domain with massive information(5)

Integrate with MediaWiki

  • Call the Scrap Bot from the Mediawiki and see if that works.

Concern/Issues

Tuesday Sep 11, 2007

  • Home page information. How is it different from domain information?

Tuesday Sep 11, 2007

  • Should we split Keywords?

Monday Sep 10, 2007

  • Alexa key for thumbnail information?
    • Same as for all alexa web-services ... account now has thumbnail enabled
  • what is ticket in alexa information?

Thursday Sep 6, 2007

  • where to put the thumbnail information. Image or url?
  • Schema Related
    • Why anchor text
    • primary key(id) for rails.
    • alexa info in domain table or separate table.
    • display, extension, isidn, rank in domain table?
    • owned domain ?
    • what is queue?
    • why schema info?
    • thumbnail information?
    • online in alexa

Wednesday Sep 5, 2007

  • position of related domain?
  • schema of the database?
  • number of links?
  • multiple versions of data?
  • whoisparsing?

Tuesday Sep 4, 2007

  • Code of the current crawler is in ?
  • Offline Crawling
    • versions of information
  • Testing of alexa information?
    • Monitoring rather testing. Result from alexa makes sense. Call for things that we know etc.
  • The way we are using google API to fetch pages etc.
  • Crawl the complete website to get meta tags etc.
    • Try to get all information in 5 pages. Heuristic approach
    • robots.txt dont crawl the disallowed website.
  • Changing the schema, if we feel like?

Monday Sep 3, 2007

  • Scalability concerns. ?
    • code of the current crawler is in ?
  • Name of people in the commit.
  • Amazon account
    • Alexa information is months old.
    • Incoming links through google.
    • Outgoing links? crawling the website.
  • Extracting information from web pages.
  • Small for small people.
  • web server.
  • web interface to see where we need to put in the calling.
    • app/web_services/web_services.rb in compost.
    • wiki/extensions/AboutUsWebServices in aboutus.
  • Schema ?

Schema

  • Domain
    • url
    • homepage

Development Repositories

  • /opt/git/compost-whois-refresh
  • /opt/git/aboutus-whois-refresh

Gather Information

Ruby Libraries

  • mechanise mechanize
  • hpricot
  • topsoil web spider




Rating: 0 - 0 votes

Company Name

Company Contact

Page Type

This page is about a company.

Retrieved from "http://aboutus.com/index.php?title=Rewrite_PageScrapeBot&oldid=12009937"