I’m wondering if I’m starting to outgrow Tailscale… my wife keeps having networking issues on Android due to Tailscale, the Nvidia Shield kills the Tailscale app randomly, and my parents’ TV doesn’t have a Tailscale app…
I feel like the time is approaching to publicly expose some of my services to the internet…
Any other tips?
You must log in or register to comment.
Tried setting this up, caught a few already
I find it hilarious you censored the banned IPs.
Exposing stuff to the internet shouldn’t be that scary… I haven’t had any incident so far in 8 years. Yes, you see plenty of illegitimate access attempts in the logs, but if everything is properly patched, it should be OK.
If you’re looking to actually do Fail2ban, look into crowdsec first. It’s a similar concept but instead of creating your own block lists by people hammering against your system until they’re banned, it uses community-populated lists to pre-ban known bad actors.
I know a lot of people shit on it from a decentralization perspective, but I use Cloudflare to expose all my services. Then anyone who hits my sites has to go through Cloudflare’s detections first. I have all my services behind a reverse proxy (nginx proxy manager) running locally, and that’s the only though exposed to the Internet through my router, also that ONLY allows connections at all from Cloudflare IPs or my local network. My home IP is obfuscated, my services can only be accessed using the ports I define, and things are happy. I also block as much as possible on my router, and have automatic updates on all my server VMs/LXCs.
You could also set up a Cloudflare tunnel to go to the reverse proxy and avoid needing to expose anything to the direct Internet.
Just turn off caching for any media servers domains/subdomains if you go with Cloudflare, or else it will try to cache any media on their servers and it’s technically a ToS violation so people get their accounts banned. It’s a simple setup to disable cache though.
Crowdsec is absolutetly the way to go. Just be aware you need the engine and what they call a “bouncer” both running, but they have easy instructions about how to install both.
Theres also a plugin for opnsense which is pretty much plug’n’play.
But iz doesnt have access to service access logs so it will be a bit more handicapped.
I have all my services behind a reverse proxy and use Crowdsec to monitor and block automated attacks. I also have pocket-id for auth, I use SSO for apps that support it and others just require authentication to access them at all. The docs are pretty solid, it was easy to set up.
Make sure you know the services running on your server, the most likely way you could get attacked is by just leaving some vulnerable or misconfigured software running and accessible.
Also I’d probably set up account lockouts on any software you can, I know Jellyfin supports it.
Exposing services to the internet is a whole other game. Try wireguard first, i never had issues and use it mostly from my tablet.
I am kinda interested in WireGuard, but how does it work with multiple non-PC devices on different networks? Tailscale runs seamlessly on my Arch laptop, but Android, TVs, and streaming sticks have hiccups from time to time.
I have services that I want to share with my non-techie family. If a service stops working, they suck at debugging and fixing the issue on their own.
Tailscale is WireGuard under the hood, if you didn’t know. It’s an overlay network that uses WireGuard to make the actual connections, and has some very clever “stuff” to get the clients actually to connect, even if behind firewalls without needing port forwarding.
Using WireGuard directly basically just changes the app you use, which may or may not help with your issues. But the connecting technology is the exact same.
I’ve never used tailscale but use wireguard extensively.
There’s not much of a learning curve for you as the administrator. You have to discard some misconceptions you might bring from other VPNs but really after 30 minutes of looking at configs you’ll get it.
I use wireguard for my small team of 5 people to access self hosted services. You install wireguard, load the config, and then it just works.
The trick, if it can be called that, is using public dns for private services.
On your server, suppose you have service-a service-b and service-c in containers with ip addresses in the 10.0.2.0/24 range. Then you’d have a reverse proxy like traefik at 10.0.2.1. You’d also create a wireguard container with an IP in that same 10.0.2.0/24 range, and configure it’s wireguard adapter to be 10.0.12.1 or soomething so you have “2” for the containers and “12” for the wireguard clients.
Then in wireguard configurations you direct all traffic for 10.0.2.0/24 through the tunnel but everything else just uses their devices normal internet connection.
Finally create a public dns record pointing to the reverse proxy like *.mydomain.com > 10.0.12.1
now whatever.mydomain.com will resolve to your reverse proxy but is still only available to devices connected to the wireguard container on your server.
DuckDNS is great for this. Been using it for years.
My primary scare is wg being another potential single point of software failure, preventing remote access
You can run it on two servers at once. I have WireGuard running on my router (which can’t give full speed) and my server (which can run at full speed). Also running on another machine for fun.
Yes, I run many services and website on the public web from my homelab. Harden your server first. Like disabling root ssh login.
Also enable auto updates on your server. Use your router/server to block some counties using geoip (especially if those services are meant for only a couple of people within your county maybe?). You could also use block lists, there any many bad ip lists out there.
Configure rate limits in Nginx.
You also mentioned fail2ban. You can define many rules and actions. Like blocking ips that might go over your previously defined rate limits. Or 4xx action for ips that request a lot of non existing pages (404 errors) .
Also captcha won’t cut it anymore today. Try https://github.com/TecharoHQ/anubis
Of course expose only what you want to expose, so only open ports in your firewall you really want to open. Ideally put everything behind a reverse proxy like Nginx.
Let’s start with all of the things mentioned above. Ping me later if want to know more or have questions.
Harden your server first
Do you have any tutorials or guides on this handy?
Use your router/server to block some counties using geoip
Yeah, definitely all my users are in the same town/region/country as me. So this could be doable.
Configure rate limits in Nginx
Hm, currently using Caddy as my reverse proxy. I guess there’s some module for this.
only open ports in your firewall you really want to open
The only port I need open is 443 for accessing Jellyfin and Immich. I can definitely block 22 from the public internet. And fuck it no automatic redirects from 80 to 443. TLS or bust.
Caddy is also fine.
I wrote a blog about server hardining and you might find it useful: https://blog.melroy.org/2023/server-hardening/
Do you have any tutorials or guides on this handy?
Now that’s a deeeeep rabbit hole. I tend to go overboard on hardening and security, however, one good place to start is installing Lynis and run a scan. Lynis will spit out a rather extensive list of areas you need to harden or adjust and a score for your server. It will also give links where you can go and read up on the specific item in question. Now, not every one of the bullets in the list will apply, but you should give each careful consideration. Lynis is Free and Open Source Software (FOSS).
- Site: https://cisofy.com/lynis/
- Install: apt-get install lynis
- Run:
lynis audit system
I ran a scan just for demonstration purposes so you can see what the end results are. This is just a snippet:
spoiler
* Configure minimum password age in /etc/login.defs [AUTH-9286] https://cisofy.com/lynis/controls/AUTH-9286/ * Configure maximum password age in /etc/login.defs [AUTH-9286] https://cisofy.com/lynis/controls/AUTH-9286/ * Default umask in /etc/login.defs could be more strict like 027 [AUTH-9328] https://cisofy.com/lynis/controls/AUTH-9328/ * To decrease the impact of a full /home file system, place /home on a separate partition [FILE-6310] https://cisofy.com/lynis/controls/FILE-6310/ * To decrease the impact of a full /tmp file system, place /tmp on a separate partition [FILE-6310] https://cisofy.com/lynis/controls/FILE-6310/ * To decrease the impact of a full /var file system, place /var on a separate partition [FILE-6310] https://cisofy.com/lynis/controls/FILE-6310/Be mindful of where you get your hardening tutorials. There are hundreds of thousands out there. I would stick with authoritative sources.
ETA: I would also recommend reading up on Cloudflare Tunnels/ZeroTrust. I know some people are iffy about Cloudflare and I see their points. It’s worth a read in my opinion.
What kinds of things are you planning to expose? What I expose I hide behind a reverse proxy with IP whitelists. Whatever I don’t need access to on the go I don’t expose.
What kinds of things are you planning to expose?
Primarily Jellyfin and Immich.
What I expose I hide behind a reverse proxy with IP whitelists.
Do all your clients have fixed IPs? I have some clients that are phones or laptops, but I would imagine those change as people drive around to different cities or connect to different coffee shop WiFi.
It depends on what service - some, like Jellyfin, are accessed only from home IPs which are static (for music through Jellyfin I use offline mode to prevent too much mobile traffic), so I can add those specific IPs in the whitelist. Otger services I need to access from elsewhere, and I can add entire subnets (i.e. for my phone carrier network or VPN servers). Those change once in a while and that is annoying. Other services I want publically available.
Jellyfin especially still has some unsecured endpoints where it would be wise to take some.extra precautions. I think the risk some people seem to think this poses is a little overblown (i.e. rights holders finding your instance and reverse mapping your entire library and suing you to oblivion), but better not risk it.
I’ve had to stop using it on my Pixel. In the last few months I have more and more suddenly lost all connectivity outside of my tailscale network. I tried excluding apps but I still will randomly fail to receive SMS or calls, suddenly getting them delivered in a rush when I disconnect from tailscale.
If anyone has any tools to recommend troubleshooting the phones connection let me know. I have no idea how to learn more about the problem beyond the obvious “If tailscale isn’t on, it doesn’t happen.”
GAAH! OK! I’M NOT CRAZY!
The exact same thing is happening to my wife’s phone. We’re both on Pixel 8s, have the same VPN settings, but for some magic reason Tailscale breaks only her phone. She has to turn off Tailscale and reboot her phone to regain connectivity.
These shenanigans is why I’m considering just exposing things to the public internet. I’m using Tailscale on several device types and Tailscale adds friction to all of my devices (except Arch where everything always works).
I understand the friction is there for a good reason, but my family doesn’t. They just see that Jellyfin doesn’t work and that all of this is buggy and maybe they just should sign up for Netflix instead of dealing with all of these bugs.
Yeah, I had a overall bad experience with everything being buggy and then even devices that weren’t connected to tailscale would start trying to ping the tailnet address instead of the local (wasn’t using their funky bridge subnets feature or whatever it’s called, so I don’t know why it would happen).
Their magicDNS is cool in theory but caused me nothing but problems. Once I turned off their DNS and set up my own DNS server for it though, it’s gotten to basically be as seamless as they claim it’s supposed to be from the start. I’m no longer having any issues with it at all.
I’ve had a pretty good experience with it aside from this recent problem with my phone - Pixel 8 Pro. It’s a big deal right now - I have a number of self hosted services I use on my phone accessed through a shared subnet via tailscale. When I left it enabled, multiple times a day I’d lose connectivity entirely. It would get fixed if I just quickly disable-enable it… at least until it randomly happens again in an hour or two. I started using spit tunneling, which I think fixed the connectivity issue for internet-dependent apps but nothing I tried fixed calls and texts.
Unfortunately, my mother has been having a number of health issues so there is no fucking way I’m going to risk missing calls and texts…so I just deal with being disconnected from my servers for now. I really wish there was a solution or something I could do to figure out what’s going wrong. I can’t keep trying random things and risk it. Calls from my mother are virtually the only calls I get, other than spam.
fail to receive SMS or calls, suddenly getting them delivered in a rush when I disconnect from tailscale.
I think that’s a known issue, but the last time I looked there didn’t seem to be a known cause or a fix.
I wouldn’t
Is mTLS an option?
You also could setup a ingress node on the network to route certain IP addresses over Tailscale.
I personally prefer Netbird
I recommend looking into setting up site-to-site VPN configurations between routers at each location. I have this set up between my home network and my parents’ network.
Once you have it properly configured you can simply have the router itself handle routing of specific traffic over the VPN connection, instead of needing each device to connect to the VPN individually.
it’s a bit more complicated to set up and maintain but not anything outrageously complex, and absolutely worth it for your use case IMO
site-to-site VPN configurations between routers at each location … the router itself handle routing of specific traffic over the VPN connection, instead of needing each device to connect to the VPN individually.
Interesting. This could help with my home network and my parent’s, but I also need to handle cases where I don’t control the network. I have clients that are phones. My family would expect that the services keep working even if they’re in a different city on cellular or at a friend’s house.
Site-to-site and individual client setups are not mutually exclusive. They can co-exist.
I have Wireguard on both my phone and laptop - and tailscale should work the same way - which I only activate when I’m away from my home network.
You could even set it up where each “roaming” device is always connected to their “home” network by VPN, which uses site-to-site to further route the traffic where it needs to go.
“roaming” device is always connected to their “home” network by VPN
Ah, right. Well, currently I do have my wife’s and my phone on the Tailscale VPN. The issue I’m trying to solve is that the VPN app on Android (and other environments) isn’t 100% bug-free. For some unknown reason, my wife frequently has issues with Tailscale. It’ll break her entire networking on her phone. The only way to fix it is by rebooting her phone. I have no idea why because we have the same phone and the same settings and it works fine on my phone. I’ve tried turning off Tailscale, logging out, and back in, and the network won’t recover. Sometimes the Tailscale app won’t even trigger the SSO page to sign in. So it just stays permanently logged off.
The Nvidia Shield also has similar issues where I have to fuss around with the VPN.
So at this point, I feel like I’m done debugging VPN apps and maybe it would be easier for users if I expose stuff to the public internet. Obviously, it makes management for me harder, but that’s ok if everything Just Works for everyone without extra steps or without having to reboot your phone every week.
