I’m under the impression that Google deliberately hinders their YouTube platform just because you have ad-blockers. With videos taking time to buffer, seems to signal this, because it almost makes up nearly the amount of time for when ads take if you had to watch them. So since you’re ad-blocking and they don’t like that, they’ll make your experience miserable until you want to pay their service or not use ad-block.
that’s not a conspiracy theory, adblock devs are aware of this and I think it was even discussed at the yt-dlp project as they are affected too.