Long story short Activity Pub only pulls the content it needs from remote servers when it needs it and can choose how to handle media (serve the original or cache and proxy). It already is similar-ish to a CDN.

AT-Proto is super complex, but my understanding is that a new server (app in AT-Proto parlance) needs to copy everything beforehand from all others, and needs to constantly replicate everything, wether it will be served or not, making the data transfers intractably massive.