Dementia Posted October 5, 2012 Report Share Posted October 5, 2012 Anyone ever use Wget or something similar to crawl an online journal database? What if I wanted to extract every single .pdf and have them placed neatly into sub-directories, organized by journal name? Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide Dementia's signature Hide all signatures Soundcloud Link to comment https://forum.watmm.com/topic/75968-wget/ Share on other sites More sharing options...
mcbpete Posted October 5, 2012 Report Share Posted October 5, 2012 wget -r -np -A.pdf whatever-the-url-is Erm possibly - haven't used it in years, don't blame me if you end up downloading the whole internet ... I seem to remember something about robots.txt preventing crawling (mass downloading) on sites as well Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide all signatures I haven't eaten a Wagon Wheel since 07/11/07... ilovecubus.co.uk - 25ml of mp3 taken twice daily. Link to comment https://forum.watmm.com/topic/75968-wget/#findComment-1887111 Share on other sites More sharing options...
Recommended Posts