https://netmaker.org logo
#netmaker
Title
# netmaker
c

cool-army-24422

07/14/2022, 10:35 AM
Hi Team, my netclients giving warning and susequently after 5-10 mins going into error state
j

jolly-london-20127

07/14/2022, 11:33 AM
Can nodes reach each other despite showing in error state?
c

cool-army-24422

07/14/2022, 2:52 PM
No
Only nodes are able to connect to netmaker-1
j

jolly-london-20127

07/14/2022, 2:52 PM
any logs from clients? sounds like MQ connection is not work
*working
c

cool-army-24422

07/14/2022, 2:53 PM
Actually now I am away, Can I share the logs tommorrw
Hi @jolly-london-20127 , Good Morning 🙂 I can see in logs Jul 15 04:07:58 Netclient-AZ netclient[518397]: [netclient] 2022-07-15 04:07:58 unable to connect to broker, retrying ... is it the issue? Thanks.
j

jolly-london-20127

07/15/2022, 10:44 AM
yes that is definitely the problem
can you provide more logs?
usually means mq is not configured properly
c

cool-army-24422

07/15/2022, 10:52 AM
sure @jolly-london-20127
Please let me know if you know any furher details
j

jolly-london-20127

07/15/2022, 11:03 AM
c

cool-army-24422

07/15/2022, 11:05 AM
Thanks @jolly-london-20127 will try and reach out to you if any further issues come up
so is this the issue with broker, which is not allowing the peers to ping each other and eventually going into ERROR state
j

jolly-london-20127

07/15/2022, 11:06 AM
yes, the broker is what sends updates to all machines
so if the broker is not functioning, machines will not be alerted of any network changes
c

cool-army-24422

07/15/2022, 11:07 AM
Thanks @jolly-london-20127 , much appreciated for the exact info. 🙂
This may be a silly question, these changes need to be applied on server side. if I am not wrong?
j

jolly-london-20127

07/15/2022, 11:11 AM
yes, these changes are all server side
but the clients will likely need certificates so will need to run "netclient pull"
c

cool-army-24422

07/15/2022, 11:20 AM
mq: container_name: mq image: eclipse-mosquitto:2.0.11-openssl depends_on: - netmaker restart: unless-stopped volumes: - /root/mosquitto.conf:/mosquitto/config/mosquitto.conf - mosquitto_data:/mosquitto/data - mosquitto_logs:/mosquitto/log - shared_certs:/mosquitto/certs expose: - "8883" labels: - traefik.enable=true - traefik.tcp.routers.mqtts.rule=HostSNI(
broker.hetznerhcm.dima.kmd.dk
) - traefik.tcp.routers.mqtts.tls.passthrough=true - traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883 - traefik.tcp.routers.mqtts.service=mqtts-svc - traefik.tcp.routers.mqtts.entrypoints=websecure what I can see in my docker-compose.yml for mq block, therer's no ports and its shared certificates. so I need to update these changes and spin up the docker compose again?
@jolly-london-20127 your comment here please !! 🙂
j

jolly-london-20127

07/15/2022, 11:29 AM
config looks correct
actually...valid question to @bored-island-21407 and @quiet-continent-24852 , I think we took 1883 out of the default docker compose but should we leave it in there?
in any case it should still work, I would test the other troubleshooting steps @cool-army-24422
c

cool-army-24422

07/15/2022, 11:32 AM
Ok @jolly-london-20127 , please let me know. if you find any
b

bored-island-21407

07/15/2022, 11:33 AM
servermqport defaults to 1883 if not specified ... no need to put in docker compose
j

jolly-london-20127

07/15/2022, 11:34 AM
yeah but we no longer expose on MQ by default
is that still reachable over the docker network? we used to expose 127.0.0.1:1883
ehh nvm it looks like we haven't done that for a while
b

bored-island-21407

07/15/2022, 11:35 AM
you are correct; MQ_SERVER_PORT should be set to 8883
or need to add ports 127.0.0.1:1883 to mq section
c

cool-army-24422

07/15/2022, 11:37 AM
root@Netmaker-dev:~# docker logs mq 1657540466: mosquitto version 2.0.11 starting 1657540466: Config loaded from /mosquitto/config/mosquitto.conf. 1657540466: Opening ipv4 listen socket on port 8883. 1657540466: Opening ipv6 listen socket on port 8883. 1657540466: Opening ipv4 listen socket on port 1883. 1657540466: Opening ipv6 listen socket on port 1883. 1657540466: mosquitto version 2.0.11 running 1657540467: New connection from 172.18.0.2:44778 on port 1883. 1657540467: New client connected from 172.18.0.2:44778 as vO45Z7mt1hgzrqd1frc6dGj (p2, c1, k60). 1657693023: New connection from 172.18.0.2:46242 on port 1883. 1657693023: New client connected from 172.18.0.2:46242 as b7axCSqQCzrosEE90zkWtoy (p2, c1, k60). 1657693023: Client b7axCSqQCzrosEE90zkWtoy disconnected. 1657693025: New connection from 172.18.0.3:34764 on port 8883. 1657693025: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate 1657693025: Client disconnected: Protocol error. 1657693029: New connection from 172.18.0.3:34766 on port 8883. 1657693029: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate 1657693029: Client disconnected: Protocol error. 1657693033: New connection from 172.18.0.3:34768 on port 8883. 1657693033: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate 1657693033: Client disconnected: Protocol error. 1657693037: New connection from 172.18.0.3:34770 on port 8883. 1657693037: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate 1657693037: Client disconnected: Protocol error.
if this may help you, I see mq version is not the latest
j

jolly-london-20127

07/15/2022, 11:38 AM
I think I was wrong on this. Even in 0.14.0 we were not setting 1883: https://github.com/gravitl/netmaker/blob/v0.14.0/compose/docker-compose.contained.yml
b

bored-island-21407

07/15/2022, 11:38 AM
not an issue
j

jolly-london-20127

07/15/2022, 11:38 AM
he probably just needs to re-gen certs then
c

cool-army-24422

07/15/2022, 11:39 AM
how to do this?
b

bored-island-21407

07/15/2022, 11:40 AM
the troubleshoot gist has steps
c

cool-army-24422

07/15/2022, 11:41 AM
aaaha yeah my bad 😇
root@Netclient-AKS-DEV-RG:~# netclient join -t eyJhcGljb25uc3RyaW5nIjoiYXBpLmhldHpuZXJoY20uZGltYS5rbWQuZGs6NDQzIiwibmV0d29yayI6Im9yay1henVyZSIsImtleSI6IjFmMzYwODEwN2RhNDFhYzgiLCJsb2NhbHJhbmdlIjoiIn0= [netclient] 2022-07-15 11:47:47 joining ork-azure at api.hetznerhcm.dima.kmd.dk:443 [netclient] 2022-07-15 11:47:47 UDP hole punching enabled for node Netclient-AKS-DEV-RG [netclient] 2022-07-15 11:47:47 starting wireguard [netclient] 2022-07-15 11:48:19 unable to connect to broker, retrying ... Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time=18.309649ms Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time=18.169549ms Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time=19.054748ms [netclient] 2022-07-15 11:48:23 could not connect to broker broker.hetznerhcm.dima.kmd.dk connect timeout [netclient] 2022-07-15 11:48:23 connection issue detected.. attempt connection with new certs and broker information [netclient] 2022-07-15 11:48:23 certificates/key saved [netclient] 2022-07-15 11:48:55 could not connect to broker at broker.hetznerhcm.dima.kmd.dk:443 [netclient] 2022-07-15 11:48:55 failed to publish update for join connection timeout After re-generating the certs and restarting the netmaker and mq image still not able to connect to the broker
j

jolly-london-20127

07/15/2022, 11:50 AM
in your netmaker config, is SERVER_NAME set to the dns entry for the broker?
c

cool-army-24422

07/15/2022, 11:52 AM
yes SERVER_NAME: "broker.hetznerhcm.dima.kmd.dk"
I think it's appropriate
j

jolly-london-20127

07/15/2022, 11:54 AM
Did you try the mosquito client connection test in the gist?
c

cool-army-24422

07/15/2022, 11:58 AM
nslookup broker.hetznerhcm.dima.kmd.dk Server: 127.0.0.53 Address: 127.0.0.53#53 Non-authoritative answer: Name: broker.hetznerhcm.dima.kmd.dk Address: 49.12.241.214
is it a positive sign
Hi @User, Please suggest any fix. Thanks.
Hi @jolly-london-20127 need your comment here please 🙂
b

bored-island-21407

07/19/2022, 10:06 AM
can you connect to the broker from a node using the mosquitto client as outlined in the troubleshooting gist?
c

cool-army-24422

07/19/2022, 10:06 AM
sure
root@Netmaker-dev:~# sudo docker-compose up -d traefik is up-to-date netmaker is up-to-date coredns is up-to-date netmaker-ui is up-to-date Starting mq ... error ERROR: for mq Cannot start service mq: mkdir /run/containerd/io.containerd.runtime.v2.task/moby/86746f4f3878884fc77a9cbab6fb6f3ac641aa16a10e3d15529b5db95d6bee74: file exists: unknown ERROR: for mq Cannot start service mq: mkdir /run/containerd/io.containerd.runtime.v2.task/moby/86746f4f3878884fc77a9cbab6fb6f3ac641aa16a10e3d15529b5db95d6bee74: file exists: unknown ERROR: Encountered errors while bringing up the project.
unable to start mq itself now
b

bored-island-21407

07/19/2022, 10:21 AM
something is wrong with your docker setup. I believe you need to purge everything as start over
c

cool-army-24422

07/19/2022, 10:23 AM
@bored-island-21407 thanks for the suggestion, will start from scratch again
[netclient] 2022-07-19 10:50:42 joining kmd-net at api.hetznerhcm.dima.kmd.dk:443 [netclient] 2022-07-19 10:50:42 UDP hole punching enabled for node Netclient-AZ [netclient] 2022-07-19 10:50:42 starting wireguard [netclient] 2022-07-19 10:51:14 unable to connect to broker, retrying ... Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time= 18.138254ms Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time= 19.364619ms Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time= 17.148702ms [netclient] 2022-07-19 10:51:18 could not connect to broker broker.hetznerhcm.dima. kmd.dk connect timeout [netclient] 2022-07-19 10:51:18 connection issue detected.. attempt connection with new certs and broker information [netclient] 2022-07-19 10:51:18 certificates/key saved [netclient] 2022-07-19 10:51:50 could not connect to broker at broker.hetznerhcm.dima.kmd.dk:443 [netclient] 2022-07-19 10:51:50 failed to publish update for join connection timeout root@Netclient-AZ:~# -------------------------------------------------------- I restarted and spinned up all docker conatiners but still I can see connection issue with the broker Is there any problem with the domain?
@jolly-london-20127 @bored-island-21407 would you like to suggest something here ? 🙂
b

bored-island-21407

07/19/2022, 11:56 AM
need more information ... log outpput from netclient node, netmaker container and mq container. did you follow all the steps in the mq troubleshooting gist?
c

cool-army-24422

07/19/2022, 11:57 AM
yes
b

bored-island-21407

07/19/2022, 11:57 AM
can you connect from a mosquitto client?
c

cool-army-24422

07/19/2022, 11:57 AM
can we connect over a call if you have any feasible time ?
b

bored-island-21407

07/19/2022, 11:59 AM
what were the results of your actions when following the mq troubleshooting gist
c

cool-army-24422

07/19/2022, 12:04 PM
root@Netmaker-dev:~# docker logs mq 1658226829: mosquitto version 2.0.11 starting 1658226829: Config loaded from /mosquitto/config/mosquitto.conf. 1658226829: Opening ipv4 listen socket on port 8883. 1658226829: Opening ipv6 listen socket on port 8883. 1658226829: Opening ipv4 listen socket on port 1883. 1658226829: Opening ipv6 listen socket on port 1883. 1658226829: mosquitto version 2.0.11 running
b

bored-island-21407

07/19/2022, 12:05 PM
so no connections in the mq logs at all?
c

cool-army-24422

07/19/2022, 12:07 PM
there are many
b

bored-island-21407

07/19/2022, 12:08 PM
and have you tried regenerating certificates for netmaker/mosquitto?
c

cool-army-24422

07/19/2022, 12:11 PM
yes
and I spinned up everything from scratch again
b

bored-island-21407

07/19/2022, 12:11 PM
regenerated certs and restart mq
then do netclient pull on clients
c

cool-army-24422

07/19/2022, 12:20 PM
Yeah I did all the process
root@Netclient-AZ:~# netclient pull -n kmd-net [netclient] 2022-07-19 12:19:53 UDP hole punching enabled for node Netclient-AZ [netclient] 2022-07-19 12:19:55 certificates/key saved root@Netclient-AZ:~#
b

bored-island-21407

07/19/2022, 12:22 PM
and you successfully connected to the broker with mosquitto client but netclient still has errors?
c

cool-army-24422

07/19/2022, 12:32 PM
Yes
--------------------- Hey Guys, A little update and deviation from the discussion topic. It's observed that the machine where Netmaker server was configured has been compromised. We are analysing into the potential threat. After that we will be resuming the POC of creation of netmaker networking. Thanks for your support