Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters Git Popular version control system, primarily for code VPS Virtual Private Server (opposed to shared hosting) nginx Popular HTTP server
[Thread #165 for this comm, first seen 10th Nov 2026, 15:50] [FAQ] [Full list] [Contact] [Source code]
I have a simple static site. It’s been up for less than a month. On the cheapest possible VPS, someone gave me. Robots.txt:
User-Agent: * Allow: /How long do yall think it will take for it to blow up?
Personally I proxy everything through cloudflare tunnels using a wildcard cert, and have Authelia in front of everything. No open ports.
Haven’t had bot problems since swapping to this setup.
I found that using a wildcard domain cert for HTTPS and an uncommon higher port number works just fine to avoid crawlers and scrapers. I also hide my ssh port behind WireGuard because I got annoyed with the bots filling up my logs with failed attempts. I’ve been doing that for nearly a year now and I have had no more issues with bots
I’m going off the assumption that the people who run crawlers and scrapers are going for known common ports for quick ease of access
I don’t feel the need to run anything extra such as Anubis either. The inconvenience of using an uncommon higher port number has been minimal and I still get to enjoy my tiny corner of the internet
Higher port? I suppose you just run things for yourself and never share them on the public internet then.
I share my site with only with people I know personally and they don’t seem to have any issue accessing my site
They don’t seem too bothered adding
:8000to the address. Also they are usually connecting at home or on their phone and are not frequently visiting it so the inconveniences are minimalMost people getting hammered by bots are running public facing websites. There a custom port is suspicious as heck.
A limited access service like yours is better served by a VPS or a simple login before everything.
I don’t understand how it’s suspicious if I am sharing my address with a custom port to people I know. I even take time to explain why I do it and in a way a nontechnical person would understand
The connection is still encrypted because I have a proper domain cert. There’s no warning or any other issue. I have spent more time securing my hardware/operating system/software than anything else, often taking priority over hosting more services
Do I really need to take all these extra steps like a VPS or a login for a basic blog generated from markdown files and a basic file server that has copies of my git repositories? Seems excessive when I already found a solution for myself that takes less effort and works to my satisfaction
I think there’s value in experimenting outside the current set standards and norms. Especially since corporations and states are obviously abusing the trust of these standards and norms for their own personal benefit
I personally feel like a small web/indie web can find their own their place again on the internet with a little creativity and a will to break away from the current norms. Trying to do more of the same will eventually lead to issues like burnout or ever growing frustration
I’m doing this for fun and for myself above all else. Experimenting with different, low effort approaches to the crawler/scraper issue is fun for me. Involving more outside services or adding more internal software does not sound like fun to me. It doesn’t sound like much fun for other people in this thread or by the original blog post either
It’s hard to post original work to share but also know it can just get scraped but I can’t give up personal websites. Social media looks so bland and seeing completely personalized blogs are the most enjoyable part to the internet now. Spending more time in niche spaces on the internet more than ever.
I host tons of open source projects on https://forge.sciactive.com/
Recently, it was getting absolutely hounded with tons of meaningless requests. I’m assuming they are just results of AI.
So, I put Anubis in front of it. Now images hosted there won’t load on other sites, people have to wait forever to see the code, and the experience on phones is even worse. Just because AI doesn’t respect anything, and the people who run it are too stupid to realize that.
This is why we can’t have nice things.
It’s odd, I see so many people complain about Anubis taking a significant amount of time, but I’ve only ever had it 1-2 seconds on my devices
Then let’s ship your devices.
You can set a trap that isn’t in robots.txt or is forbidden there. Anyone who ignores your robots.txt is not friendly.
Just because AI doesn’t respect anything, and the people who run it are too stupid to realize that.
They’re perfectly capable of realizing it. How do you think it got built that way? They don’t respect anything, either.
True.
This makes a difference on Forgejo: https://github.com/go-gitea/gitea/pull/34024
Also, obviously, a caching reverse proxy. Forges and wikis are hard because the bots drill down into the thousands of version control pages that get generated on demand and cant be cached.
If I didn’t enjoy the cat & mouse of it, I’d definitely walk away from running a site. Just out of curiosity, I created an hourly graph today showing malicious bot, legitimate bot and human ratio graphs for one of my sites. The malicious hits make up around 80% hourly and human hits are usually well under 10%.
What sorts of measure you have taken to prevent bots from exhausting your compute and bandwidth
Another example expanding on my previous comment.
This IP from Hong Kong hit my site over 5,000 times in a couple minutes. The first 12 page loads were legitimate URLs and got served. On their first exploit attempt, they were blocked and the remaining 4,98x page loads were no-effort 403s. This type of thing has made it possible for me to have a cheap public-facing site as I wouldn’t be able to afford an endless array of malicious IPs doing this day in and day out. Before I went this route, I would have to shut the site down during high impact periods of these visitors as they’d corrupt and crash the database.
As soon as an IP is determined to be malicious, subsequent pageloads result in a single query to match the IP and they get dumped to a 403 static error.
With the bots that hammer a site looking for the top 1,000 or so exploitable URLs or partake in speed scraping, this has worked wonders for my server load. I catch them in their 1st-5th page load and the rest just result in basically no effort on the part of the script.
The AI scrapers and all the PITA they cause is one of the primary reasons I don’t host a blog site.
Indieweb suffers heavily due to this.
I host a few sites and I haven’t had a problem yet. Sure, logs suggest a majority of traffic is bots, but the cheap VPS I’m hosting on hasn’t melted so I haven’t been motivated to do anything about it.
Lemmy is particularly susceptible. Hosting a static site with a limited number of pages is fine. But Lemmy needs to check all posts, find the votes, sort them, and compile the page. It does that for practically every page load, and so loading a page once is quite “expensive”.
Then add on to that the almost unlimited pages of content for bots to crawl through, bots just get caught endlessly scraping pages. Even powerful servers struggle, which is why most Lemmy instances have implemented some sort of protection against bots.
All the logged-out views in Lemmy should be cached, otherwise you’re going to have scalability problems. edit: you can probably do this without even touching the Lemmy code. There’s already nginx serving lemmy.ca, which can be configured to cache it and respect the Vary header. (I suspect the admins are already way ahead of me here, but I just wanted to share the general approach, which is pretty universal.)
Why not just use more static pages?
4chan has been ‘static’ forever, the backend builds each page every ~15s, pages served to users are always static. It’s ‘justwerked’ for 20 years now on a shoestring.
Lemmy is built as a separate web app and back end that communicate via APIs. It makes it easy to support apps and alternate web interfaces, unlike Piefed which only started adding an API reasonably recently and doesn’t have a fraction of the eco system that Lemmy has.
But I think an important part is that Lemmy is explicitly customised for users. While bots are getting the same logged out version, the intent of Lemmy is not to be scraped by bots but it’s to be custom for each user. You’re supposed to log in, subscribe to communities you like, block people you don’t like, and have posts you’ve already seen hidden (if you set this in your settings).
Serving a custom page to each user based on their settings, blocks, and subscriptions isn’t really suited to static pages.
Sounds like 4chan rolled their own server-side page cache. It probably made more sense for 4chan to do that then, back when the internet was still pretty new.
Nowadays there are a few packages that perform a similar function, like Varnish. Varnish picks up requests from the internet and where possible, it returns the response as static HTML from its own cache.
There are other types of caching at different stages of the request lifecycle that can reduce the gap between dynamic and static website performance.
There were attempts to rewrite imageboard software with modern dynamic frameworks and caching infrastructure, but it scaled worse than oldschool index and thread html regeneration on post event, and was cancelled. The effort was called something like Infinity, by 8chan.
Back in the day people used Squid for that if they needed it.
I believe you, but the linked article is not about hosting Lemmy.
No it’s not, it was more of a tangential comment.
I think this is the photography site he’s talking about.
He mentioned “so many scrapers hit my site that it saturated my upload with generated content feeding the them.” which seemed unusual to me. I’m not sure why such a site would have a public upload. It doesn’t feel like a site that would have a technical challenge with the scrapers, but I also get the impression the main issue is the thought of hosting all his photos just for AI scrapers to train their image generation models.
saturated my upload
I interpreted that as LP having the site running in their basement, and responding to scrapers saturates their outgoing bandwidth.
Ah that would make more sense!
It doesn’t need to. Like any similar site, it can cache a lot of that in databases like redis, or cache the rendered page in part or in whole.
Parts will be cached but it’s per instance. For a 100 user instance, what are the chances that a recent cache of All page 74 is available? Bots are crawling every page, and most pages are content from last month, last year, or 3 years ago.
I would expect Lemmy instances to have some limiters set for guest connections, especially for ones coming from datacenters. Like, it should be easy and senseful to only link and cache up to All page 10, or Hot page 5, and refuse guest loads of anything beyond that.
Ultimately this has been a recent-ish problem, and Lemmy simply doesn’t have this sort of function yet. It’s a goo thought, though.
As the other commenter said, bad bots are not easy to tell. OpenAI, Anthropic, etc identify themselves so are easier to block. They also tend to respect a robots.txt. The issue is the b=bots that don’t limit themselves, and route through hundreds of residential IPs. You literally see dozens of IPs all with approximately the same number of requests, like 80,000 per IP. If you block them they route through others. They seem to have an endless number of residential IPs to route through, presumably due to the things you see on Lemmy about the TV apps that proxy traffic even when the TV is off. I notice a lot via South American or South East Asian countries, which lead me to suspect at least some of these residential IPs are proxying through compromised devices that might not have up to date security patches available.
I dont believe it is that easy to tell if an IP is coming from a data center, unfortunately. To some extent you can tell its location with the geolocation database, but that is not authoritative and can be wrong or even contain intentional falsehoods. You can also see what is adjacent to it and make guesses that way, but that is often inconclusive due to common tunneling protocols like MPLS.
I believe the only remotely accurate way to tell if a connection is a bot is to use modules that detect bot behavior and redirect what it detects. Some users have even made infinitely recursive “traps” for bots that do not respect their robots.text file, which make their web page seem infinite so it scrapes forever over these fake pages that are less work to host than they are for the bot to parse through.
It is a very tough problem with few easy solutions. It will likely result in an arms race much like ads and ad blockers, hackers and firewalls, etc. It has so far, anyway.
I dont believe it is that easy to tell if an IP is coming from a data center
The bad scraping bots are proxying their traffic via resiential IPs, so it doesn’t even matter. Presumably this is through the free TV apps that you see mentioned on Lemmy occasionally, or through compromised devices.
Yeah, internet is no longer the interesting place it used to be. Interesting, intriguing, exciting. Now it’s worse than going to a tax office… the day before taxes… On Christmas… 😂
We should all go back to HTML and web rings
And Usenet but with spam filters somehow
I am actually trying geminispace. It has nothing to do with google btw, but it’s quirky, it’s weird and I am actually getting some old internet vibes
I run a DaaS product Trace Jobs that serves job posting data.
My expectations are pretty low and am never disappointed. Self-hosting can be pretty affordable.
