Is there someway to toggle if udp hole punching is used for a node without leaving and rejoining the mesh?

yes, you just need to edit the node in the UI and turn off "dynamic port"

Ah, well that's surprising for me. I've got a bunch of nodes that are talking with each other and they all have dynamic port turned off and no affordances are made in the network gateway for port forwarding... How is it working then if hole punching is disabled when the dynamic port option is off?

nodes in different networks can still talk without udp hole punching. It's meant to enhance connectivity with some forms of more restrictive NAT, but typically it still works in 80-90% of cases without udp hole punching enabled

How the port get negotiated then? E.g, at the gateway how does the packet get routed to the right host behind the firewall if there is no existing connection to use?

we do enable port forwarding on the individual machines. And typically if one of the machines is not behind NAT, then the connection can still be established

Sorry I mean port forwarding from the NAT gatway to the internal host. Both sets of machines are behind NATs so I'm puzzled how it's actually working.