I’m looking for storage classes for a multi node cluster. I’m currently using Longhorn and NFS, but I’m not happy with the performance. My cluster doesn’t have beefy nodes, so Ceph/Rook is out of the question (for now).

Nodes:

  1. 8 GB RAM, 4 cores VM, control plane. 256 GB SSD
  2. 4 GB RAM, 2 cores, control plane, currently cordoned. 128 GB SSD
  3. 8 GB RAM, 4 cores, ARM, control plane. 512 GB SSD
  4. 8 GB RAM, 4 cores. 256 GB SSD
  5. 16 GB RAM, 6 cores. 256 GB SSD + 1 TB HD
  6. RPi 4, 4 GB RAM. 128 GB SSD
  • h3ron@lemmy.zipEnglish
    2·
    7 hours ago

    I have two storage nodes and one is much faster than the other.

    I’m currently evaluating a juicefs deployment based on two minio instances (one per node, replicated with async bucket replication) through a load balancer (sidekick) in failover. Because juicefs also needs a db for metadata, I went with valkey + sentinel.

    Juicefs provides a CSI driver that supports ReadWriteMany volumes and CSI snapshots and manages both read and write cache. Performance is much much better than Ceph. In theory it should be riskier (because of the async replication) but in practice I haven’t yet lost a bit.

    • eutampieri@feddit.itOPEnglish
      1·
      26 minutes ago

      Thanks! A bit more involved that I’d have thought but still worth considering! Could you update us after your evaluation?

  • Decronym@lemmy.decronym.xyzBEnglish
    3·
    16 minutes ago

    Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

    Fewer Letters More Letters
    NFS Network File System, a Unix-based file-sharing protocol known for performance and efficiency
    SSD Solid State Drive mass storage
    k8s Kubernetes container management package

    3 acronyms in this thread; the most compressed thread commented on today has 7 acronyms.

    [Thread #95 for this comm, first seen 15th Feb 2026, 11:20] [FAQ] [Full list] [Contact] [Source code]

  • custard_swollower@lemmy.worldEnglish
    3·
    11 hours ago

    I’m not sure you’ll get nice performance in local network with small appliances (consumer network hardware, mini PCs and rpi 4). I’ve never got sub-ms network disk access on 1Gbps switch and router. In the end I’ve done the opposite - I’ve added one k8s host with a lot of storage, and any storage services are deployed there. All the other k8s services rely on local SSDs.

    • eutampieri@feddit.itOPEnglish
      2·
      14 hours ago

      I used to use Ceph at work and I’m a bit reluctant to use it at home. Don’t get me wrong, it’s really cool, but those were beefy nodes, and I only have 1 Gbps between nodes

    • jonathan@piefed.socialEnglish
      2·
      12 hours ago

      I’d expect the performance to be awful but it still has relatively niche usecases, especially where performance isn’t a concern. I’m imagining legacy apps that don’t speak S3.