Ivy Zemura yvI oo ii oo Posted February 19 Report Share Posted February 19 (edited) We need to scrape WATMM somehow, if even just segments of it, the admin guy said he's not going to archive or save anything and we have to do it ourselves. This thread is for posting discussion on figuring out how to do it. I'm no expert at web scraping or archiving, but I've tried a few things so far Methods: Browsing to a thread and printing it to PDF in chrome, each page individually Benefits: Preserves the page in a high quality way, with embedded links still manifesting as links inside the PDF so they can be followed Problems: Really slow and takes manual labor Can't be easily hosted online or shared with others afterwords Tips: Make sure you aren't logged in and have cleared all of your browser's caches to avoid personal metadata like your username being embedded within the printed PDF Progress: https://forum.watmm.com/topic/90748-autechre-production-methods-speculation/ Complete https://forum.watmm.com/topic/97375-world-music/ Complete https://forum.watmm.com/topic/76672-vaporwave/ In progress Scraping via wget Problems: Always gives me 403 forbidden errors, but I haven't tried every option like providing cookies strings, testing different user agent strings, etc Archive.org wayback machine https://archive.org/ Benefits: Archives to archive.org's servers which can be accessed by anyone later Problems: Tough to get the website to archive your page, they seem to have low availability lately due to ongoing DDOS attacks Tips: Can theoretically be ran through a browser plugin that gives you a quick way to archive any page you're currently on to the wayback machine, in practice I have no luck with this though and it just gets stuck loading forever Share your ideas and experiments Resources: https://wiki.archiveteam.org/index.php?title=Main_Page Edited February 19 by Ivy Zemura yvI oo ii oo decibal cooper, t yst r, Caged Element and 8 others 7 3 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/ Share on other sites More sharing options...
Ivy Zemura yvI oo ii oo Posted February 19 Author Report Share Posted February 19 The wayback machine can be good even with its existing content, see: https://web.archive.org/web/*/https://forum.watmm.com/topic/76672-vaporwave/page/* Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005134 Share on other sites More sharing options...
THE METZ TRACK Posted February 19 Report Share Posted February 19 you can easily use something like curl or wget, it's actually pretty straightforward. don't overthink it all our precious watmm memories will be backed up. we still have three months. Berk and psn 1 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide THE METZ TRACK's signature Hide all signatures faith <3 Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005135 Share on other sites More sharing options...
Ivy Zemura yvI oo ii oo Posted February 19 Author Report Share Posted February 19 https://web.archive.org/save The capture failed because Save Page Now does not have access rights for http://forum.watmm.com/topic/76672-vaporwave/page-82 (HTTP status=403). great... On 2/19/2025 at 6:28 AM, Dragon said: you can easily use something like curl or wget, it's actually pretty straightforward. don't overthink it all our precious watmm memories will be backed up. we still have three months. ok you are in charge of this, provide me the command nice: https://web.archive.org/web/*/https://forum.watmm.com/topic/* Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005136 Share on other sites More sharing options...
THE METZ TRACK Posted February 19 Report Share Posted February 19 yeah a couple of months ago i was backing up the entire DANK MEMES thread in the hope of creating a trippy timelapse video. i finished archiving it (90,000 memes lmao) but then i had trouble finding a way to create the video. @Ivy Zemura yvI oo ii oo dm me Uros, J3FF3R00, ignatius and 5 others 1 1 1 5 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide THE METZ TRACK's signature Hide all signatures faith <3 Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005137 Share on other sites More sharing options...
o00o Posted February 19 Report Share Posted February 19 Here is a tutorial: https://www.perplexity.ai/search/the-ipboard-forum-watmm-com-go-fzGBm_laQKqo41HFhGQaZg#0 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide o00o's signature Hide all signatures Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005138 Share on other sites More sharing options...
psn Posted February 19 Report Share Posted February 19 Hope someone does it. Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005139 Share on other sites More sharing options...
usagi Posted February 19 Report Share Posted February 19 I also think this is important. psn and Berk 1 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide usagi's signature Hide all signatures On 4/17/2013 at 2:45 PM, Alcofribas said: afaik i usually place all my cum drops on scientifically sterilized glass slides which are carefully frozen and placed in trash cans throughout the city labelled "for women alco" with my social security and phone numbers. Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005142 Share on other sites More sharing options...
cruising for burgers Posted February 19 Report Share Posted February 19 I hope it all burns to ashes... Rubin Farr, THE METZ TRACK, ManjuShri and 1 other 1 1 1 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide cruising for burgers's signature Hide all signatures ig discogs mixcloud soundcloud Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005152 Share on other sites More sharing options...
auxien Posted February 19 Report Share Posted February 19 i hope y'all catch the album mega threads and random into threads in the major artists forums. in theory there's still a few months so i'm sure going to find some time to try and catch some to help out Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide auxien's signature Hide all signatures / b c / m a s t o d o n / b l o t / Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005164 Share on other sites More sharing options...
usagi Posted February 19 Report Share Posted February 19 I'm mostly concerned about posts like this being lost. On 4/7/2010 at 11:31 PM, Alcofribas said: i met up with sean booth, one half of autechre (ae) and talked about everything from the duo's new album to horror films, even to what his mum's favorite sarny is innit. so, how does oversteps compare to your previous records? well, it's much better. we use a lot of melodies and wrote it on a kyma system MIDI'd up to some old roland drum machines we bought from moby. that bloke is a vegan, lol would you say it's more emotional than, say, Quaristice? nah what was different about your work methods? well, rob and i met back in 2001 and we've been working with pretty much the same equipment over in his dad's attic. so, this time around, we just did the same old shit. i usually make the beats and the melodies and he usually records it into his computer. then i master it and he plays about with his two kids. then i do the artwork. under your alias TDR? yeah, you've done yer research mate! thanks! what records inspired this album? ugh, i'd have to say "amber," "incunablua," "drukqs," all by autechre. and i guess mira calix, cuz she's my mate. do you buy a lot of new records? not really,...well, yeah, i by 'em off discogs a lot. i've got like every atom tm record. but mostly i buy mp3s off itunes. itunes is lush! do you think it's possible to make music after 9/11? nah how do you cope with the trauma of war? work on tracks, innit? are you going to tour in america for this album? no, just europe and mexico. mexico is fucking lush! are you working on any new tracks at the moment? yeah. are they any good? yeah. i've got one with this epic crescendo, like huge, fucking half hour crescendo. i'm calling it "bolero" by ravel. ever read watmm? yeah, i'm ET on there. lol what else is going on in your life these days? i'm making some kids with the missus, that's well lush. rob and i are doing yoga together too but with the distance we have to do it together on skype. nirvana is lush. when's the next album coming out? later this year, autumn prob. autumn is lush. thanks for talking with me! lush mate. Expand (jk ofc there's treasure troves in all the genre threads that need saving also) auxien, jules, baph and 2 others 2 3 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide usagi's signature Hide all signatures On 4/17/2013 at 2:45 PM, Alcofribas said: afaik i usually place all my cum drops on scientifically sterilized glass slides which are carefully frozen and placed in trash cans throughout the city labelled "for women alco" with my social security and phone numbers. Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005165 Share on other sites More sharing options...
auxien Posted February 19 Report Share Posted February 19 On 2/19/2025 at 1:42 PM, usagi said: I'm mostly concerned about posts like this being lost. just download the browser extension and save page every time you see one...(i believe this works? someone correct me if i'm mistaken) Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide auxien's signature Hide all signatures / b c / m a s t o d o n / b l o t / Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005168 Share on other sites More sharing options...
zazen Posted February 19 Report Share Posted February 19 (edited) Havent tried it but https://github.com/TUVIMEN/invision-scraper and its successor https://github.com/TUVIMEN/forumscraper Edited February 19 by zazen th555 and colunga 1 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005169 Share on other sites More sharing options...
Ego Posted February 19 Report Share Edit Hide Delete Posted February 19 Most scrapers wont work because WATMM is using Cloudflare which blocks the usual scraping methods. I have currently archived all threads from May 2013 to Feb 2025. Should be around 25% of all content. Aim to finish this month. Uros, dr lopez, t yst r and 13 others 5 11 Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005170 Share on other sites More sharing options...
usagi Posted February 19 Report Share Posted February 19 On 2/19/2025 at 2:58 PM, Ego said: Most scrapers wont work because WATMM is using Cloudflare which blocks the usual scraping methods. I have currently archived all threads from May 2013 to Feb 2025. Should be around 25% of all content. Aim to finish this month. legend. colunga and t yst r 2 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide usagi's signature Hide all signatures On 4/17/2013 at 2:45 PM, Alcofribas said: afaik i usually place all my cum drops on scientifically sterilized glass slides which are carefully frozen and placed in trash cans throughout the city labelled "for women alco" with my social security and phone numbers. Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005172 Share on other sites More sharing options...
zazen Posted February 19 Report Share Posted February 19 On 2/19/2025 at 2:58 PM, Ego said: Most scrapers wont work because WATMM is using Cloudflare which blocks the usual scraping methods. I have currently archived all threads from May 2013 to Feb 2025. Should be around 25% of all content. Aim to finish this month. Thats great, what format are they archived in? t yst r 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005173 Share on other sites More sharing options...
decibal cooper Posted February 19 Report Share Posted February 19 On 2/19/2025 at 6:32 AM, Dragon said: yeah a couple of months ago i was backing up the entire DANK MEMES thread in the hope of creating a trippy timelapse video. i finished archiving it (90,000 memes lmao) but then i had trouble finding a way to create the video. @Ivy Zemura yvI oo ii oo You are doing God’s work, danke t yst r 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005174 Share on other sites More sharing options...
oscillik Posted February 19 Report Share Posted February 19 perhaps https://github.com/projectdiscovery/katana o00o 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide oscillik's signature Hide all signatures Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005176 Share on other sites More sharing options...
o00o Posted February 19 Report Share Posted February 19 On 2/19/2025 at 3:26 PM, oscillik said: perhaps https://github.com/projectdiscovery/katana That looks very nice Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide o00o's signature Hide all signatures Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005180 Share on other sites More sharing options...
Ego Posted February 19 Report Share Edit Hide Delete Posted February 19 (edited) On 2/19/2025 at 3:16 PM, zazen said: Thats great, what format are they archived in? I'm just collecting the full page source for each page into an SQLite database. A bit wasteful but want to make sure we don't miss anything. 22GB and counting... Later on we can consider parsing out all the actual posts and metadata. I also need to do a run to fetch all the images that were uploaded to the board. Edited February 19 by Ego Alcofribas, psn, usagi and 3 others 6 Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005183 Share on other sites More sharing options...
psn Posted February 19 Report Share Posted February 19 Somebody buy this guy a beer. t yst r 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005188 Share on other sites More sharing options...
cruising for burgers Posted February 19 Report Share Posted February 19 (edited) I don't authorize the use of my content in the new forum... don't really wish to have any connection whatsoever to watmm and its owner... but anyway good job Edited February 19 by cruising for burgers colunga 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide cruising for burgers's signature Hide all signatures ig discogs mixcloud soundcloud Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005190 Share on other sites More sharing options...
Freak of the week Posted February 19 Report Share Posted February 19 On 2/19/2025 at 5:18 PM, psn said: Somebody buy this guy a beer. colunga 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005191 Share on other sites More sharing options...
dr lopez Posted February 19 Report Share Posted February 19 hilarious we have to do this because of one person's obdurate hubris Uros, Ivy Zemura yvI oo ii oo, Alcofribas and 2 others 2 2 1 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Hide dr lopez's signature Hide all signatures On 11/24/2015 at 12:29 PM, Salvatorin said: I feel there is a baobab tree growing out of my head, its leaves stretch up to the heavens Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005192 Share on other sites More sharing options...
snue apnu Posted February 19 Report Share Posted February 19 (edited) I've spent the last 2 weeks transcribing the contents of the LTM forum and Rephlex BBQ forum by hand, with a biro, to a series of hardbound foolscap notepads. About 7% done. Edited February 19 by perunamuusi Freak of the week, colunga, zazen and 6 others 3 3 1 2 Thanks Haha Confused Sad Facepalm Burger Farnsworth Big Brain Like × Quote Link to comment https://forum.watmm.com/topic/105501-~scraping-watmm-project~/#findComment-3005198 Share on other sites More sharing options...
Recommended Posts