I think enabling and disabling ufw actually fixed things on all servers and now I can ping all of them. I didn't test external clients but for some weird reason it works. Oh I hate debugging network stuff.