Difference between revisions of "StructuredDataFromWikiPages"
| Line 1: | Line 1: | ||
| − | <noinclude><big>[[OurWork]] < [[DevelopmentTeam]] < [[DevelopmentTeamPriorities|Priorities]] < </noinclude>[[BigBeautifulInfluenceUs]] {{JustTinyEditIcon| | + | <noinclude><big>[[OurWork]] < [[DevelopmentTeam]] < [[DevelopmentTeamPriorities|Priorities]] < </noinclude>[[BigBeautifulInfluenceUs]] {{JustTinyEditIcon|StructuredDataFromWikiPages}}<noinclude></big> |
__NOTOC__ | __NOTOC__ | ||
Revision as of 01:35, 25 March 2008
What (summary)
Extract data that users have entered onto Wiki pages and turn them into structured data for easier manipulation.
We need to extract
- Contact info
- Address
- Phone #
Why this is important
We are moving towards using more highly-structured data, but need to leverage the large quantity of data users have entered onto our site.
DoneDone
- Easily identify which data has been added or changed to a wiki page by human edits. (standard diff may work?)
- Apply heuristics (section, regular expression, machine learning, something else) to determine if a piece of data should belong in a structured field.
- If all human-added data can be extracted, indicate that the entire wiki page should be deleted.
- If there remain human-added data that can't be identified and extracted, return wikitext containing only the non-identified human data, with all bot-created data removed.
Steps to get to DoneDone
- Build many test cases--pick many random human-edited pages.
- Pull out revision histories (or at least diffs to compare to original bot scrape)
- Identify and extract human-edited data yourself. Great fun!
- Make test cases pass. (In order below?)
- First identify all human-edited data
- Then classify and extract said data
- Then determine if a page should be deleted, and, if not, which data should be left behind.
- Throw a wild and crazy party
