(S (NP (NNP John) ) (VP (VPZ loves) (NP (NNP Mary) ) ) (...) )
Wikipedia dumps: http://dumps.wikimedia.org/enwiki/20150304
Setup/finish you preprocessing pipeline. Design it in a modular way such that tasks that are independent are handled by separate calls (e.g. wget and subsequent preprocessing). Don’t cram them all into one script. It should be very ease to choose another source and rerun your preprocessing pipeline.
So far, no third-party applications/scripts are allowed.
It is most important to setup a working pipeline than to create a perfect clean-up procedure. You may destroy sentences to a certain extent, as long as the final result is free of noise and garbage.
Due: Thursday April 30, 2015, 16:00