I don’t even know what they’re using the SSDs for.
Most businesses are too stupid to train their own models from scratch, and won’t use “foreign” ones so they won’t finetune them either.
On the inference side… SSDs aren’t used for much. Just storing Docker stuff/dependencies and model weights for the initial load, and that’s it. Maybe some data for bulk processing, but that’s no different than existing software. The one niche may be KV cache swapping for re-using prompt prefixes, but this is limited and being obsoleted by new attention mechanism.
So WTF do they even need SSDs and HDDs for? Honestly it feels like FOMO purchasing.
I don’t even know what they’re using the SSDs for.
Most businesses are too stupid to train their own models from scratch, and won’t use “foreign” ones so they won’t finetune them either.
On the inference side… SSDs aren’t used for much. Just storing Docker stuff/dependencies and model weights for the initial load, and that’s it. Maybe some data for bulk processing, but that’s no different than existing software. The one niche may be KV cache swapping for re-using prompt prefixes, but this is limited and being obsoleted by new attention mechanism.
So WTF do they even need SSDs and HDDs for? Honestly it feels like FOMO purchasing.