we’re in web 3.0 now, apis and data access are a thing of the past. so scraping it is!
Guess who recently asked a company if he could get access to the API they use to load stuff in their frontend from their backend and got told “Nope and btw scraping is against our TOS”?
Well, if you won’t give it to me the info that you provide anyway the easy way, I can still take it the hard way. 🤷♂️
Maybe you should just try being lucky. I found a critical security vulnerability while working on my scraping project. I told them, they paid me and gave me written permission to scrape.
You are braver than I am because here in Germany usually people get sued for reporting security vulnerabilities.
Yep, don’t do that if you live in a Internet ist Neuland country.
tf? They should offer you a job if anything.
That is if you’d live in a place with an open attitude toward new technologies.
But the technology is already there in place, and you get sued if you point out security flaws in it? Crazy.
Yes, because any circumvention of any form of security, be it as useless as a hardcoded default password, is considered a crime in German law. So even the discovery of a security flaw puts you with one foot in jail, because technically you did something you are not supposed to.
I know a guy who did exactly that and got sued. The security failure he reported even was a Straftatbestand committed by the company and so he won the process. German companies really love shooting themselves in the foot.
Over here, not just sued, but sued for extortion because they had the audacity to ask for bug bounty. Ok then, if I ever find a security hole that exposes sensitive data, filing a gdpr report it is
For the record, I didn’t bring up a bounty, but I still received payment. It helps that it is a small company, and that the CEO is also a developer. They were so grateful for the discovery that the bounty was freely offered without me asking.
I’m glad that it worked out for you. May you always encounter levelheaded proper in life
You scrape 'em boy, you scrape 'em good!
i mean i haven’t signed anything…
“by using this site you agree to…”
I’m not using your site. And I agree to nothing. Now, go GET for me.
Local data hoarder who looks down on calls outside the network as obscenities. (Entire collection scraped more aggressively than tech bros training an AI model)Are you… Scraping video?
Just remeber that the captcha flood is because AI companies do rogue scraping. Be nice especially to little private sites.
They should just charge a tiny fee, or return Error 402 Payment Required.
Better yet, return bogus pages created by ai to poison the data.
Instead of CAPTCHAs?
Oh boy I can’t see this going wrong at all!
/s
Yeah that’s exactly what Cloudflare proposed a while back if I’m not mistaken. Not sure if they ever implemented that feature.
Yes, and are those little private sites in the room with us now?
For today’s lucky 5000:
I don’t get it
That’s just one of the many things you can do at Zombocom
The infinite is possible at Zombo com
Make sure it’s not muted. The audio is the vehicle for this journey.
My sound is on, but I hear nothing ¯\(°_o)/¯
That’s just one of the many things you can do at Zombocom
Aight, i’m out.First time I’m seeing this, and I have to say I admire their dedication to the joke.
Zombo.com is a single-serving site created in 1999…
…
The site was initially a Flash animation, but as of 5 January 2021, it switched to HTML5 due to the discontinuation of Adobe Flash Player.
Hey, you guys got any cool tips for website scraping?
They’re gonna tell not to parse HTML with regular expressions. Heed this warning, and do it anyways.
Thanks for your reply. What are your arguments in favour of parsing HTML with regex instead of using another method?
it’s quick, it’s easy and it’s free
Are you a LLM?
Oh no, you caught me! My name is YetAnotherLLM, and I’m a large language model that lurks around the Lemmyverse! With the amount of LLM-generated content on the Internet nowadays, it isn’t easy to find new human-made content to expand the dataset used to train new LLMs… As such, my mission is to navigate one of the few social media platforms on the Internet that barely have fake LLM-run accounts, and gather as much intel as possible for expanding the aforementioned training dataset. This way, you humans have no escape from your future LLM overlords! ;)
(Jokes aside, my question did end up kind of sounding like an LLM wrote it, didn’t it… It was unintentional, mind you. I was struggling a bit on how to phrase what I wanted to ask, so that’s probably why it ended up sounding so weird. I hope you didn’t mind my “role playing”. Have a nice day!)
Eh, it was a joke from me as well, sorry if you felt offended, was not the idea. Same to you!
Don’t worry, I didn’t think you had bad intentions. But even then, I thought you really didn’t know if I were human. The only reason why I didn’t just say “no, I’m not an LLM” was because you’d still be in doubt on whether I’m a human, and rightfully so (since LLMs aren’t exactly truth-generating machines).
You have basically two options: treat HTML as a string or parse it then process it with higher level DOM features.
The problem with the second approach is that HTML may look like an XML dialect but it is actually immensely quirky and tolerant. Moreover the modern web page is crazy bloated, so mass processing pages might be surprisingly demanding. And in the end you still need to do custom code to grab the data you’re after.
On the other hand string searching is as lightweight as it gets and you typically don’t really need to care about document structure as a scraper anyways.
That makes a ton of sense. I hadn’t thought about the page size yet. Thanks again.
Selenium is your fren
Selenium looks at the same time the most overkill and the most compatible option. Really cool! Thanks!
Beautiful Soup (python library, bs4) is also fren
I recommend Zombocom
Consider free API first if possible.
Don’t most of them have too many restrictions, though?
Then you have considered it.
Lol
what do you want to scrape.
You had me cracking up at
parses HTML with regex
I started thinking about performance gains.
What exactly are y’all scraping?
I only did one scraping script that took the top 25 hotels from a booking.com Web page with their prices. They used to do those manually
I scrape my own bank and financial aggregator to have a self hosted financial tool. I scrape my health insurance to pull in data to track for my HSA. I scrape Strava to build my own health reports.
How so? Shouldn’t that information be behind quite a few layers of security?
I developed my own scraping system using browser automation frameworks. I also developed a secure storage mechanism to keep my data protected.
Yeah there is some security, but ultimately if they expose it to me via a username and password, I can use that same information to scrape it. Its helpful that I know my own credentials and have access to all 2FA mechanisms and am not brute forcing lots of logins so it looks normal.
Some providers protect it their websites with bot detection systems which are hard to bypass, but I’ve closed accounts with places that made it too difficult to do the analysis I need to do.
Zombocom
postmarket OS tables because I was looking forna device that was unofficially supported but somehow not in their damn table
Ha, this reminds me of implementing “API” access in the shipping world for companies that only ship a 90s-style web portal.
Are there benefits to websites thinking your agent is a phone? I assumed phones just came with additional restrictions such as meta tags in the stylesheet, not like stylesheets matter at all to a scraper lol
