Ncdu takes ages to run on my system, its like 500GB+ of storage space, but it takes roughly an hour to finish scanning, probably a bit less, is there any alternative which either constantly monitors my files so it always knows the sizes for me to navigate or is significantly faster than Ncdu?

  • Emma_Gold_Man@lemmy.dbzer0.com
    41·
    11 days ago

    Advice from a long time sysadmin: You’re probably asking the wrong question. ncdu is an efficient tool, so the right question is why it’s taking so long to complete, which is probably an underlying issue with your setup. There are three likely answers:

    1. This drive is used on a server specifically to store very large numbers of very small files. This probably isn’t the case, as you’d already know that and be looking at it in smaller chunks.
    2. You have a network mount set up. Use the -x option to ncdu to restrict your search to a single filesystem, or --exclude to exclude the network mount and your problem will be solved (along with the traffic spike on your LAN).
    3. You have a single directory with a large number of small files that never get cleared, such as from an e-mail deadletter folder or a program creating temp files outside of the temp directories. Once a certain number of files is reached, accessing a directory slows down dramatically. The following command will find it for you (reminder - make sure you understand what a command does before copying it into a terminal, DOUBLY so if it is run as root or has a sudo in it). Note that this will probably take several times as long to run as ncdu because it’s doing several manipulations in series rather than in parallel.

    sudo find $(grep '^/' /etc/fstab | awk '{print $2}') -xdev -type f -exec dirname {} \; | sort | uniq -c | sort -nr | head

    explanation

    This command doesn’t give an exact file count, but it’s good enough for our purposes.

    sudo find # run find as root

    $( … ) # Run this in a subshell - it’s the list of mount points we want to search

    grep ‘^/’ /etc/fstab # Get the list of non-special local filesystems that the system knows how to mount (ignores many edge-cases)

    awk ‘{print $2}’ # We only want the second column - where those filesystems are mounted

    -xdev # tell find not to cross filesystem boundaries

    -type f # We want to count files

    -exec dirname {}; # Ignore the file name, just list the directory once for each file in it

    sort|uniq -c # Count how many times each directory is listed (how many files it has)

    sort -nr # Order by count descending

    head # Only list the top 10

    If they are temp files or otherwise not needed, delete them. If they’re important, figure out how to break it into subdirectories based on first letter, hash, or whatever other method the software creating them supports.