if others click the downvote button on my posts/comments, nothing happens at all as far as I know.

I think it shares the downvotes within that instance (possibly with other instances that allow downvoting) so anyone else viewing the post from the same instance as the user who downvoted will still see it and their browser will use that when sorting the posts.