Yes, although there have been a few CVE’s related to escalating privileges or breaking out of the sandbox. You’re going to want to keep those updating and keep on top of those.

There is one concern, in that the nix store (/nix/store) is world readable. It is not world writable, which is good, but there is a problem in that secrets can potentially be copied into the nix store. If you copy a file containing environment variables or the like into the nix store, it could theoretically be found it. This one is on the users of nix to be careful of.

With Nix flakes, the entire git repo that the flake originates from is copied into the nix store. Meaning if you have a nix flake in something that is supposed to be a private repo, or contains tracked sensitive data (untracked files are not copied into the nix store), then it could be found and inspected by other users.

My big concern with multi user in this case is not merely the Nix daemon though, but also Linux itself. Linux has been hit with a LOT of privilege escalation and container escape issues over the past few years, and many of them have been zero day’s. Given this, I no longer really have the same level of trust for Linux with regards to multi user isolation, for in cases like these.

Of course, in academic computing, I would probably just do multi user anyways, for simplicity. If you install tracking and monitoring, then you can attach malicious actions to identities. Because every user is registered and operates within the institution, if they break the terms of use for computing equipment, then you can punish them within the institution, or pursue greater legal action.

And then, you also would want to enforce 2FA to help minimize stolen accounts. While that can still be phished via fake login pages that ask for 2FA, it handles most of the things. Isolate the server via firewalls, and then it becomes a small enough target that doesn’t have enough value (assuming of course, your research isn’t too valuable and worth targeting), and you mostly don’t have to worry about it.

Sure, people will root it. But then they’ll send you a nicely worded email explaining how they rooted it and how to fix it. Which is what my friend did at my school, on our shared ssh server. Just keep it updated to handle the low hanging fruit.

However, if you are going to give it to untrusted users with unknown or temporary (not within the institution) identities, then things change, and you have to take it a lot more seriously. I no longer have confidence in just Linux’s user based isolation.

The first line solution I would go to, is to put users in containers and mount the nix store (and nix daemon) in and out. Something like a docker/podman container or Incus container. Of course, container escapes are still possible. If you are even more concerned about those, then you would want virtual machines. It is still possible to share the nix store between virtual machines, but it is more complicated.

In addition to that, virtual machines have a performance tradeoff (usually 95% or more of the host’s performance though), but there might be issues with sharing GPU compute resources among virtual machines, depending on the hardware you are using. Enterprise GPU’s usually support it though.

Debian has a policy of only cherry picking security updates, or critical bugfixes, in order to ensure maximum system stability. In general, they don’t do entire program updates or additional features. For the 4 year lifecycle of a Debian release, it will behave the same as it did yesterday.

As of today, current version of rsync in Debian’s stable’s packages is 3.4.1: https://packages.debian.org/stable/rsync (archive).

The versions with significant LLM assistance are 3.4.3 and later.

the physical display output is claimed by something else totally separate.

Yes, and the above solution I laid out is a way to get around that, that potentially dodges the complexity of configuring a truly headless session. It’s possible to run two KDE sessions at once, next to eachother, and then simply put one on the main display, with the second being “headless”, and not shown.

I started fiddling with the above setup I described, actually installing and testing sunshine. Unfortunately I hit some hiccups. I got sunshine to run, but with no input, and then then it attached to the wrong KDE, even though it was streaming the correct one initially. Now I’m running the same KDE session as a different user (since I was on the same user before), but having pairing issues, but I suspect these are because moonlight is seeing the wrong SSL cert, since the sunshine as a new user generated a new cert.

Do you actually need the headless backend? Or do you need a background session that doesn’t interfere with the main session?

Second comment:

https://docs.lizardbyte.dev/projects/sunshine/v0.21.0/about/guides/linux/headless_ssh.html

There is also this guide, but it is specific for Xorg and Nvidia gpus. But I suspect that if you replace the startx here with startplasma-wayland or dbus-run-session startplasma-wayland, then it will work. It would still be nvidia gpu specific though.

If you log in (preferably as another user) to another tty (Cntrl + Alt + F2/3/4/5), and then run dbus-run-session startplasma-wayland, does that work?

Sometimes just startplasma-wayland works, which used to work for me but didn’t work for me this time.

Then, you should be able to switch back to the original KDE session, which continues to work normally, at least it does for me. Both sessions should have shared access to the GPU, as well. You can then probably run sunshine in that second instance of KDE, while the monitor can be attached to tty1 or tty2, whichever one is the default where KDE is put.

Then, you would probably want to configure sunshine to work even when the screen is locked, that way you can lock the second session with Win + L, but it can still be interacted with remotely via sunshine.

EDIT: I only am somewhat confident that this setup will work, I haven’t tested it personally.

Debian has old software and doesn’t support some newer hardware. Even the latest version, Trixie, is still very far behind. I realize this now.

Debian has backports, which enable you to get select versions of newer software. Relevant to your case, you can get a newer version of the kernel, drivers, and other things relevant to hardware support.

Certbot?

It can automate provisioning of certificates using DNS-01 challenges, which have wildcard certificates.

https://eff-certbot.readthedocs.io/en/stable/using.html#dns-plugins

Example tutorial: https://www.digitalocean.com/community/tutorials/how-to-create-let-s-encrypt-wildcard-certificates-with-certbot

Once you do that it puts the cert in /etc/letsencrypt/live/ so you can then do whatever you want with it. You would have to handle distribution manually.

If you want to automate stuff across of a bunch of machines at once, I recommend Ansible: https://docs.ansible.com/projects/ansible/latest/index.html

One thing I read about is that Ubuntu provides unattended updates, so it can automatically update packages and restart the server (that seems a bit too far ;) ). It’s probably possible on Debian but not out of the box.

Ubuntu automatically has unattended upgrades, which makes I think makes it a popular choice for VPS providers to push (beyond being popular in corporate/institutions overall), since they don’t have to worry about users forgetting security updates. However, it doesn’t enable automatic reboots. But, it does look like automatic restarts of services updated via unattended upgrades is done, but only as of Ubuntu 24.

Another thing to note is that Ubuntu has updates that are explicitly for security, and then everything else, including more general bugfixes and program updates with additional features. By default, unattended upgrades on Ubuntu only do security updates.

Of course, both a unattended upgrades and automatic reboots are possible on Debian (same software, Ubuntu just preconfigures it), although default configurations can vary. I wouldn’t be suprised if a VPS provider was shipping a default Debian configuration that enabled automatic upgrades.

On Debian, for the most part, ALL updates are only for security issues or severe bugfixes (program crashes or the like). Debian, for the most part, doesn’t do minor bugfixes at all, or do program feature updates. I prefer this model, since it’s easier to manage than having separate types of package updates. It ensures absolute stability, a guarantee that the system tomorrow will behave the same as it did yesterday, while still enabling automatic security updates. This model is ideal for a server I don’t want to babysit, or for your grandpa who loses his mind when the button he is supposed to click was moved one spot over.

In addition to that, I like the policy of automatic reboots. With the stable, slow moving nature of both distros, it’s safe to automatically reboot to ensure that kernel vulnerabilities, or vulnerabilities in other critical systems are fixed. Automatic updates and reboots can be the difference between someone being able to escape a docker container or someone not being able to.

There is one thing to note, is that adding additional repos (or PPA’s which technically you aren’t supposed to work on Debian), can be dangerous, and you have to be careful: https://wiki.debian.org/DontBreakDebian/#Don.27t_make_a_FrankenDebian

The main problem is that if a third party repo and Debian provide the same package, and the system is configured to prefer the third party repo, then you can be installing a potential dependency to the rest of your system that isn’t actually tested against your system, or compatible.

You have to be really careful to ensure that the programs in the repo are actually built and designed for your system, and also that the your system does not default to installing them.

Third party repos also break the guarantee of stability that automatic updates depend on. Third party repos don’t have a separate security channel, so Ubuntu will probably avoid touching them, even if critical security fixes are needed. Debian will update them, but unlike Debian’s packages, the overall program updates won’t be guaranteed to be behave the same due to potential major/minor version changes.

For example, if you get Docker from docker’s repo instead of Debian’s/Ubuntu’s, now your version of docker can no longer auto update and receive potential security fixes safely.

Overall, I prefer Debian but Ubuntu is fine. If I was in an environment that used Ubuntu already, I would also just use Ubuntu. If you are already using Debian, then Debian is probably preferable.

What made moodle frustrating?

I have also heard the opposite take, that canvas sucks and moodle is great.

Did you use it as a student/user, or as an administrator?

There is also moodle.

Canvas and moodle are really popular in schools and universities.

It looks like KDE and linux mint have the ability to auto unlock and mount drives, according to this bug:

https://bugs.kde.org/show_bug.cgi?id=466731

Probably a feature of the desktop environment.

Overhewire bandit is a tutorial and introduction to CTF’s. It covers some of the lesser known commands like strings or uniq, by making you use them in challenges. Of course, it later goes on into basic exploits. But it’s a good intro to some of that stuff, and I like that it’s hands on.

Depending on how desperate you are, and if you are willing to delve into the dark arts, there are ways to get new KDE on stable distros. It’s probably way more effort than it’s worth, however, and I wouldn’t really recommend them to most users, due to the steep learning curve and various limitations. However, if I were to ever build a Linux distro, then I might consider them, as a way to mix and match older and newer packages.

I will also preface that KDE Plasma is one the more unreliable and prone to breakages part of my Linux system (Arch). People like to joke about krashes, but it’s kinda true. The bottom bar likes to crash sometimes, and I then I have to run systemctl --user restart plasma-plasmashell (the fact that I have that memorized scares me). KDE never really stops working overall, but various papercuts appear which I find workarounds for. Gnome is better though, and is much more reliable, IF you aren’t using extensions that aren’t officially supported, which often break on major Gnome updates. It’s probably because of that that Arch Linux sat on the Gnome update for a bit, to wait for the extensions to update and become compatible, whereas Fedora just said YOLO and pulled the trigger lmao. However, Gnome’s harder dependency on systemd might make it tough to drag a newer version onto another distro using the methods mentioned below.

The 3 ways I can think of (in order of how much I prefer them):

1. The Nix package manager can install recent packages from Nixos’s rolling release or more up to date stable distro on other distros.

Nixos is both a distro and a package manager, and their repos include things like desktop environments, that wouldn’t normally be included in solutions like brew. In fact [Hyprland’s wiki has a guide on using Nix to install Hyperland on non Nixos distros](https://wiki.hypr.land/Nix/Hyprland-on-other-distros/. This will not work for KDE however. By default, Nix packages on other distros don’t have GPU hardware acceleration, which is kinda needed for a DE. You would have to wrap these packages via Nixgl. The best way is to use home manager and use that to wrap the packages.

Wait, it looks like the docs have been updated: https://home-manager.dev/manual/unstable/usage/gpu-non-nixos.html#sec-usage-gpu-non-nixos . If you are using home manager, it looks like GPU wrapping is automatically enabled if you set targets.genericLinux.enable to true, although this requires root.

Then you would have to do some stuff to setup the login manager so that it can detect the Nix/home-manger version of KDE.

2. Bedrock Linux let’s you install multiple Linux distros at once, next to eachother, mixing and matching packages and utilities.

With bedrock, you could use a Debian base system, using the Debian bootloader, kernel, and systemd, but you could be getting your DE components from Arch. Unfortunately, it doesn’t support many things or distros (see pre installation checks), and some of the things like grub + btrfs would suck to lose.

3. Distroxbox can run an entire desktop environment from within another docker container.

The main limitations with this setup, is that you are now inside the container by default. If you execute bash, or a terminal, or whatever, it’s actually the container’s version, and not the hosts. You can configure it to automatically find and execute the host’s version of the program if it’s not found in the container. But, I would prefer just cherrypicking the DE from another distro, in order to use only that component, which is why this one is third and my least favored. But it does technically work.

Containers are commonly used to distribute programs that depend on different library versions, including different libc versions.

And yes, you could compile software for specific hardware and then deploy it via containers.

What is the advantage in using a container to say run gromacs, rather than pointing the user to the path of the compiled binary

Different libc versions. There are also some sandboxing and security restrictions that are applied.

I definitely want to have some persistent services running: databases, a couple web applications and maybe a Jupiter webpage to visualize results.

Maybe docker swarm mode, with the nvidia container runtime is easier? It lets you spin up services on multiple nodes, which you could then load balance with a conventional reverse proxy. Although if you are trying to do a single instance of Jupyter, then I don’t know about it’s ability to serve multiple users, or the security of that setup. Usually, people go for things like JupyterHub + the Dockerspawner. Jupyterhub handles authentication or

There is also a Kubernetes spawner, or you can use an alternative web application that can dynamically spin up jupyterlab/notebooks, like Kubeflow, or coder, but I don’t think any of that is what you want in regards to that specific usecase for Jupyter.

If you just want to present rendered stuff, check out: https://quarto.org/ , which is a static site generator capable of executing jupyter notebooks, and rendering them to websites. It can also render I use it for my blog, which is ironic because I don’t actually use any of the data science features quarto has. I really like quarto because it’s the only static site generator which I found has fulltext search built in/easily enabled. It runs some javascript over a generated index, and you can search the website without any form of backend needed.

Paperless ngx seems to support fulltext search of PDF’s, in addition to other document types.

Two years later, and I have an answer, after reading this: https://archive.kernel.org/oldwiki/btrfs.wiki.kernel.org/index.php/SysadminGuide.html#Subvolumes

which was linked from the Arch Wiki, but it is the old wiki, which is obsolete and no longer updated. Basically, in a nested subvolume layout, the nested subvolumes inherit mounting options of their parent subvolume. This might be changeable, but it’s the default.

On the other hand, separate subvolumes have their own mount options. Although not really important for a swap file, since the file itself can have copy on write features removed, you might want this if you want a subvolume containing folders of data where you want copy on write disabled for performance purposes. This matters for things like postgres, which has it’s own alternatives to journaling/CoW:

Because WAL restores database file contents after a crash, journaled file systems are not necessary for reliable storage of the data files or WAL files. In fact, journaling overhead can reduce performance, especially if journaling causes file system data to be flushed to disk. Fortunately, data flushing during journaling can often be disabled with a file system mount option, e.g., data=writeback on a Linux ext3 file system. Journaled file systems do improve boot speed after a crash.

From here: https://www.postgresql.org/docs/18/wal-intro.html

And then the other benefit of a top level subvolume is management. You can mount it anywhere you want. You can mount it independently of other btrfs subvolumes, meaning you could share a swap file between two installed distros to save space, although this breaks hibernation, so you probably wouldn’t want that. It’s just that top level subvolumes give you flexibility nested subvolumes don’t.

Firstly, you should check out what the organization you are building for uses. If they use Red Hat, or Ubuntu, then you should probably just build your solution on top of those operating systems (or a Red Hat clone like Rocky or Alma). Both of those are popular in many organizations, and it would probably be better to use what people are familiar with and know how to troubleshoot or work their way around. Potentially, they even have support contracts, giving them the option of calling the parent company for help.

Ansible is good. The learning curve is definitely less steep than Nixos, and it’s easier to teach people. One benefit is that you can reuse existing public roles and playbooks. For example this one: https://github.com/galaxyproject/ansible-slurm , which installs slurm.

You would probably have to write additional playbooks or roles to install nvidia drivers or configure the system, but then they can stick.

openhpc

Firstly, is there anything specific from here you need? Secondly, is there anything in here that’s not available in existing distro repositories, like Ubuntu or Red Hat’s?

It certainly looks like an interesting project, but a 3 node cluster is pretty small, and I find it hard to justify things like OpenMP/MPI, which is basically a special compiler that compiles programs to run across multiple machines at once. For that runtime to work, you do actually have to compile the programs, which are written for it, using it, which can require work on the side of the people who want to run applications or simulations.

The more likely setup, to me, is that Slurm is going to to run docker containers via Apptainer. Slurm would handle assigning containers to nodes based on free resources, but they wouldn’t actually share resources like memory or CPU. This setup is still plenty useful, and very common.

Original comment, from before I read that you already selected slurm below. This comment isn’t relevant, as I realized I was targeting the wrong things but I’ll just leave it here regardless.

For the platform itself, you should use either Kubernetes or Slurm. Slurm is popular in academia, and Kubernetes is popular in corporate, but they are used interchangably depending on specific needs.

Slurm is better for scheduled tasks. Like let’s say you want researchers at a school to be able to run a long running simulation. They can sign up, reserve time for that slurm cluster, and then send out a “job”, for that slurm cluster. The job will automatically be allocated to the node of the cluster with free resources, and then ran, and then stopped, and the researcher will receive the results back.

Kubernetes is better for persistent deployed services, like web services or the like. For example, AI inference. Kubernetes can also do batch jobs, but it doesn’t have the advanced time tracking or scheduling systems that slurm has (although sometimes people build that on top of Kubernetes in order to only deploy one clustered system).

Nixos is nice for configuration as code, but it lacks clustering or application/cluster orchestration features. You can use Nixos to deploy Kubernetes or Slurm, but I wouldn’t use it as the HPC platform itself. Nixos should work with Cuda in addition to that.

Firstly, please don’t use Debian testing. It doesn’t receive security updates in a timely manner: https://programming.dev/post/20084040

Secondly, it looks like the versions of Nvidia drivers provided in Debian testing and Debian stable are the same:

https://packages.debian.org/trixie/nvidia-driver-full

https://packages.debian.org/forky/nvidia-driver-full

Please don’t use Debian Testing. It doesn’t receive security updates in a timely manner. I previously made a post about it:

PSA: You should know that Debian Testing does not receive security updates in a timely manner, and is not intended for production use.

Unlike Debian Unstable or Debian Stable, there is no fast track for vulnerability fixes needed into Debian testing. In my post above, I note that Debian stable and unstable received a fix for a very severe vulnerability. However, Debian testing was left to dry for almost a week, for a known critical vulnerability.

If you want more updated applications, you can use tools like flatpak, distrobox, appimage, or nix to layer newer applications onto a system of any Linux distro.

If you want newer base packages and underlying utilities, like a newer desktop environment, you can get stable distros that provide this, like Fedora, or Opensuse Leap.

Unfortunately it is still in Beta, but Opensuse Slowroll is OpenSuse’s solution, a rolling release that delays package updates to test and iron out bugs before pushing updated versions out. It’s somewhat of an in between a rolling release and a stable distro, and a very promising model.

You can also just directly use a rolling release distro. Debian Unstable isn’t as bad as it sounds. Unstable doesn’t mean unreliable or prone to breakages. Unstable just means that the bugs deal with will constantly be changing, instead of dealing with the same bugs all the time.

But please don’t use Debian Testing. It’s in the name, it’s only for testing purposes.