Hi Team my netclients giving warning and susequently after 5

Question

Hi Team my netclients giving warning and susequently after 5 10 mins going into error state

jolly-london-20127 · Answer

Can nodes reach each other despite showing in error state

cool-army-24422 · Answer

No

cool-army-24422 · Answer

Only nodes are able to connect to netmaker 1

jolly-london-20127 · Answer

any logs from clients sounds like MQ connection is not work

jolly-london-20127 · Answer

working

cool-army-24422 · Answer

Actually now I am away Can I share the logs tommorrw

cool-army-24422 · Answer

Hi < jolly london 20127> Good Morning I can see in logs Jul 15 04 07 58 Netclient AZ netclient 518397 netclient 2022 07 15 04 07 58 unable to connect to broker retrying is it the issue Thanks

jolly-london-20127 · Answer

yes that is definitely the problem

jolly-london-20127 · Answer

can you provide more logs

jolly-london-20127 · Answer

usually means mq is not configured properly

cool-army-24422 · Answer

sure < jolly london 20127>

cool-army-24422 · Answer

Please let me know if you know any furher details

jolly-london-20127 · Answer

please try these troubleshooting steps https gist github com mattkasun face2a7c1f32031a2126ff7243caad12

cool-army-24422 · Answer

Thanks < jolly london 20127> will try and reach out to you if any further issues come up

cool-army-24422 · Answer

so is this the issue with broker which is not allowing the peers to ping each other and eventually going into ERROR state

jolly-london-20127 · Answer

yes the broker is what sends updates to all machines

jolly-london-20127 · Answer

so if the broker is not functioning machines will not be alerted of any network changes

cool-army-24422 · Answer

Thanks < jolly london 20127> much appreciated for the exact info

cool-army-24422 · Answer

This may be a silly question these changes need to be applied on server side if I am not wrong

jolly-london-20127 · Answer

yes these changes are all server side

jolly-london-20127 · Answer

but the clients will likely need certificates so will need to run netclient pull

cool-army-24422 · Answer

mq container name mq image eclipse mosquitto 2 0 11 openssl depends on netmaker restart unless stopped volumes root mosquitto conf mosquitto config mosquitto conf mosquitto data mosquitto data mosquitto logs mosquitto log shared certs mosquitto certs expose 8883 labels traefik enable=true traefik tcp routers mqtts rule=HostSNI `broker hetznerhcm dima kmd dk` traefik tcp routers mqtts tls passthrough=true traefik tcp services mqtts svc loadbalancer server port=8883 traefik tcp routers mqtts service=mqtts svc traefik tcp routers mqtts entrypoints=websecure what I can see in my docker compose yml for mq block therer s no ports and its shared certificates so I need to update these changes and spin up the docker compose again

cool-army-24422 · Answer

< jolly london 20127> your comment here please

jolly-london-20127 · Answer

config looks correct

jolly-london-20127 · Answer

actually valid question to < bored island 21407> and < quiet continent 24852> I think we took 1883 out of the default docker compose but should we leave it in there

jolly-london-20127 · Answer

in any case it should still work I would test the other troubleshooting steps < cool army 24422>

cool-army-24422 · Answer

Ok < jolly london 20127> please let me know if you find any

bored-island-21407 · Answer

servermqport defaults to 1883 if not specified no need to put in docker compose

jolly-london-20127 · Answer

yeah but we no longer expose on MQ by default

jolly-london-20127 · Answer

https github com gravitl netmaker blob ec90dab43736a7fff780442557e5a1880e62845e compose docker compose yml L118

jolly-london-20127 · Answer

is that still reachable over the docker network we used to expose 127 0 0 1 1883

jolly-london-20127 · Answer

ehh nvm it looks like we haven t done that for a while

bored-island-21407 · Answer

you are correct MQ SERVER PORT should be set to 8883

bored-island-21407 · Answer

or need to add ports 127 0 0 1 1883 to mq section

cool-army-24422 · Answer

root Netmaker dev ~ docker logs mq 1657540466 mosquitto version 2 0 11 starting 1657540466 Config loaded from mosquitto config mosquitto conf 1657540466 Opening ipv4 listen socket on port 8883 1657540466 Opening ipv6 listen socket on port 8883 1657540466 Opening ipv4 listen socket on port 1883 1657540466 Opening ipv6 listen socket on port 1883 1657540466 mosquitto version 2 0 11 running 1657540467 New connection from 172 18 0 2 44778 on port 1883 1657540467 New client connected from 172 18 0 2 44778 as vO45Z7mt1hgzrqd1frc6dGj p2 c1 k60 1657693023 New connection from 172 18 0 2 46242 on port 1883 1657693023 New client connected from 172 18 0 2 46242 as b7axCSqQCzrosEE90zkWtoy p2 c1 k60 1657693023 Client b7axCSqQCzrosEE90zkWtoy disconnected 1657693025 New connection from 172 18 0 3 34764 on port 8883 1657693025 OpenSSL Error 0 error 14094412 SSL routines ssl3 read bytes sslv3 alert bad certificate 1657693025 Client disconnected Protocol error 1657693029 New connection from 172 18 0 3 34766 on port 8883 1657693029 OpenSSL Error 0 error 14094412 SSL routines ssl3 read bytes sslv3 alert bad certificate 1657693029 Client disconnected Protocol error 1657693033 New connection from 172 18 0 3 34768 on port 8883 1657693033 OpenSSL Error 0 error 14094412 SSL routines ssl3 read bytes sslv3 alert bad certificate 1657693033 Client disconnected Protocol error 1657693037 New connection from 172 18 0 3 34770 on port 8883 1657693037 OpenSSL Error 0 error 14094412 SSL routines ssl3 read bytes sslv3 alert bad certificate 1657693037 Client disconnected Protocol error

cool-army-24422 · Answer

if this may help you I see mq version is not the latest

jolly-london-20127 · Answer

I think I was wrong on this Even in 0 14 0 we were not setting 1883 https github com gravitl netmaker blob v0 14 0 compose docker compose contained yml

bored-island-21407 · Answer

not an issue

jolly-london-20127 · Answer

he probably just needs to re gen certs then

cool-army-24422 · Answer

how to do this

bored-island-21407 · Answer

the troubleshoot gist has steps

cool-army-24422 · Answer

aaaha yeah my bad

cool-army-24422 · Answer

root Netclient AKS DEV RG ~ netclient join t eyJhcGljb25uc3RyaW5nIjoiYXBpLmhldHpuZXJoY20uZGltYS5rbWQuZGs6NDQzIiwibmV0d29yayI6Im9yay1henVyZSIsImtleSI6IjFmMzYwODEwN2RhNDFhYzgiLCJsb2NhbHJhbmdlIjoiIn0= netclient 2022 07 15 11 47 47 joining ork azure at api hetznerhcm dima kmd dk 443 netclient 2022 07 15 11 47 47 UDP hole punching enabled for node Netclient AKS DEV RG netclient 2022 07 15 11 47 47 starting wireguard netclient 2022 07 15 11 48 19 unable to connect to broker retrying Ping tcp broker hetznerhcm dima kmd dk 443 49 12 241 214 443 Connected time=18 309649ms Ping tcp broker hetznerhcm dima kmd dk 443 49 12 241 214 443 Connected time=18 169549ms Ping tcp broker hetznerhcm dima kmd dk 443 49 12 241 214 443 Connected time=19 054748ms netclient 2022 07 15 11 48 23 could not connect to broker broker hetznerhcm dima kmd dk connect timeout netclient 2022 07 15 11 48 23 connection issue detected attempt connection with new certs and broker information netclient 2022 07 15 11 48 23 certificates key saved netclient 2022 07 15 11 48 55 could not connect to broker at broker hetznerhcm dima kmd dk 443 netclient 2022 07 15 11 48 55 failed to publish update for join connection timeout After re generating the certs and restarting the netmaker and mq image still not able to connect to the broker

jolly-london-20127 · Answer

in your netmaker config is SERVER NAME set to the dns entry for the broker

cool-army-24422 · Answer

yes SERVER NAME broker hetznerhcm dima kmd dk

cool-army-24422 · Answer

I think it s appropriate

jolly-london-20127 · Answer

Did you try the mosquito client connection test in the gist

cool-army-24422 · Answer

nslookup broker hetznerhcm dima kmd dk Server 127 0 0 53 Address 127 0 0 53 53 Non authoritative answer Name broker hetznerhcm dima kmd dk Address 49 12 241 214

cool-army-24422 · Answer

is it a positive sign

cool-army-24422 · Answer

Hi < 933510356842721340> Please suggest any fix Thanks

cool-army-24422 · Answer

Hi < jolly london 20127> need your comment here please

bored-island-21407 · Answer

can you connect to the broker from a node using the mosquitto client as outlined in the troubleshooting gist

cool-army-24422 · Answer

sure

cool-army-24422 · Answer

root Netmaker dev ~ sudo docker compose up d traefik is up to date netmaker is up to date coredns is up to date netmaker ui is up to date Starting mq error ERROR for mq Cannot start service mq mkdir run containerd io containerd runtime v2 task moby 86746f4f3878884fc77a9cbab6fb6f3ac641aa16a10e3d15529b5db95d6bee74 file exists unknown ERROR for mq Cannot start service mq mkdir run containerd io containerd runtime v2 task moby 86746f4f3878884fc77a9cbab6fb6f3ac641aa16a10e3d15529b5db95d6bee74 file exists unknown ERROR Encountered errors while bringing up the project

cool-army-24422 · Answer

unable to start mq itself now

bored-island-21407 · Answer

something is wrong with your docker setup I believe you need to purge everything as start over

cool-army-24422 · Answer

< bored island 21407> thanks for the suggestion will start from scratch again

cool-army-24422 · Answer

netclient 2022 07 19 10 50 42 joining kmd net at api hetznerhcm dima kmd dk 443 netclient 2022 07 19 10 50 42 UDP hole punching enabled for node Netclient AZ netclient 2022 07 19 10 50 42 starting wireguard netclient 2022 07 19 10 51 14 unable to connect to broker retrying Ping tcp broker hetznerhcm dima kmd dk 443 49 12 241 214 443 Connected time= 18 138254ms Ping tcp broker hetznerhcm dima kmd dk 443 49 12 241 214 443 Connected time= 19 364619ms Ping tcp broker hetznerhcm dima kmd dk 443 49 12 241 214 443 Connected time= 17 148702ms netclient 2022 07 19 10 51 18 could not connect to broker broker hetznerhcm dima kmd dk connect timeout netclient 2022 07 19 10 51 18 connection issue detected attempt connection with new certs and broker information netclient 2022 07 19 10 51 18 certificates key saved netclient 2022 07 19 10 51 50 could not connect to broker at broker hetznerhcm dima kmd dk 443 netclient 2022 07 19 10 51 50 failed to publish update for join connection timeout root Netclient AZ ~ I restarted and spinned up all docker conatiners but still I can see connection issue with the broker Is there any problem with the domain

cool-army-24422 · Answer

< jolly london 20127> < bored island 21407> would you like to suggest something here

bored-island-21407 · Answer

need more information log outpput from netclient node netmaker container and mq container did you follow all the steps in the mq troubleshooting gist

cool-army-24422 · Answer

yes

bored-island-21407 · Answer

can you connect from a mosquitto client

cool-army-24422 · Answer

can we connect over a call if you have any feasible time

bored-island-21407 · Answer

what were the results of your actions when following the mq troubleshooting gist

cool-army-24422 · Answer

root Netmaker dev ~ docker logs mq 1658226829 mosquitto version 2 0 11 starting 1658226829 Config loaded from mosquitto config mosquitto conf 1658226829 Opening ipv4 listen socket on port 8883 1658226829 Opening ipv6 listen socket on port 8883 1658226829 Opening ipv4 listen socket on port 1883 1658226829 Opening ipv6 listen socket on port 1883 1658226829 mosquitto version 2 0 11 running

bored-island-21407 · Answer

so no connections in the mq logs at all

cool-army-24422 · Answer

there are many

bored-island-21407 · Answer

and have you tried regenerating certificates for netmaker mosquitto

cool-army-24422 · Answer

yes

cool-army-24422 · Answer

and I spinned up everything from scratch again

bored-island-21407 · Answer

regenerated certs and restart mq

bored-island-21407 · Answer

then do netclient pull on clients

cool-army-24422 · Answer

Yeah I did all the process

cool-army-24422 · Answer

root Netclient AZ ~ netclient pull n kmd net netclient 2022 07 19 12 19 53 UDP hole punching enabled for node Netclient AZ netclient 2022 07 19 12 19 55 certificates key saved root Netclient AZ ~

bored-island-21407 · Answer

and you successfully connected to the broker with mosquitto client but netclient still has errors

cool-army-24422 · Answer

Yes

cool-army-24422 · Answer

Hey Guys A little update and deviation from the discussion topic It s observed that the machine where Netmaker server was configured has been compromised We are analysing into the potential threat After that we will be resuming the POC of creation of netmaker networking Thanks for your support