So another thing I was missing was getting the updated mosquitto config file (where it explicitly says to use websockets). Now the malformed packet errors are gone, but all clients even those that are 17.1 are orange. Trying to join a new client gives me:

[netclient] 2023-01-06 23:48:02 error installing:  error creating node 401 Unauthorized {"Code":401,"Message":"You are unauthorized to access this endpoint."}