Also, what do you mean by synthetic data? If it’s made by AI, that’s how collapse happens.
But that’s exactly my point. Synthetic data is made by AI, but it doesn’t cause collapse. The people who keep repeating this “AI fed on AI inevitably dies!” Headline are ignorant of the way this is actually working, of the details that actually matter when it comes to what causes model collapse.
If people want to oppose AI and wish for its downfall, fine, that’s their opinion. But they should do so based on actual real data, not an imaginary story they pass around among themselves. Model collapse isn’t a real threat to the continuing development of AI. At worst, it’s just another checkbox that AI trainers need to check off on their “am I ready to start this training run?” Checklist, alongside “have I paid my electricity bill?”
The problem with curated data is that you have to, well, curate it, and that’s hard to do at scale.
It was, before we had AI. Turns out that that’s another aspect of synthetic data creation that can be greatly assisted by automation.
For example, the Nemotron-4 AI family that NVIDIA released a few months back is specifically intended for creating synthetic data for LLM training. It consists of two LLMs, Nemotron-4 Instruct (which generates the training data) and Nemotron-4 Reward (which curates it). It’s not a fully automated process yet but the requirement for human labor is drastically reduced.
the only way to guarantee training data isn’t from its own model is to make it yourself
But that guarantee isn’t needed. AI-generated data isn’t a magical poison pill that kills anything that tries to train on it. Bad data is bad, of course, but that’s true whether it’s AI-generated or not. The same process of filtering good training data from bad training data can work on either.
I’m Canadian. I would say that I don’t think much about it in terms of current events, I haven’t heard much in the news about it in recent years. And my assumption from that is that’s probably a good sign. There used to be a steady stream of bad news, and “no news” lies along the path in between “bad news” and “good news.”
I did see a video recently about Iraq’s plans for a giant new port facility on that little tidbit of Persian Gulf shoreline it has and road/rail link from it up through to Turkey, and thence onward into Europe. It sounded like a very optimistic development if it can be seen through to fruition, opening an alternative trade corridor to the Suez Canal. Anything that diversifies a country’s economy is a good thing, and anything that removes single points of failure in global shipping networks is also a good thing. I can’t imagine the Houthi obstruction of the Red Sea would still be a thing by the time that route opens up but at least it’ll be an option if something like it happens again.