https://netmaker.org logo
Join Discord
Powered by
Hi Team, my netclients giving warning and susequen...
# netmaker
c
Hi Team, my netclients giving warning and susequently after 5-10 mins going into error state
j
Can nodes reach each other despite showing in error state?
c
No
Only nodes are able to connect to netmaker-1
j
any logs from clients? sounds like MQ connection is not work
*working
c
Actually now I am away, Can I share the logs tommorrw
Hi @jolly-london-20127 , Good Morning 🙂 I can see in logs Jul 15 04:07:58 Netclient-AZ netclient[518397]: [netclient] 2022-07-15 04:07:58 unable to connect to broker, retrying ... is it the issue? Thanks.
j
yes that is definitely the problem
can you provide more logs?
usually means mq is not configured properly
c
sure @jolly-london-20127
Please let me know if you know any furher details
j
c
Thanks @jolly-london-20127 will try and reach out to you if any further issues come up
so is this the issue with broker, which is not allowing the peers to ping each other and eventually going into ERROR state
j
yes, the broker is what sends updates to all machines
so if the broker is not functioning, machines will not be alerted of any network changes
c
Thanks @jolly-london-20127 , much appreciated for the exact info. 🙂
This may be a silly question, these changes need to be applied on server side. if I am not wrong?
j
yes, these changes are all server side
but the clients will likely need certificates so will need to run "netclient pull"
c
mq: container_name: mq image: eclipse-mosquitto:2.0.11-openssl depends_on: - netmaker restart: unless-stopped volumes: - /root/mosquitto.conf:/mosquitto/config/mosquitto.conf - mosquitto_data:/mosquitto/data - mosquitto_logs:/mosquitto/log - shared_certs:/mosquitto/certs expose: - "8883" labels: - traefik.enable=true - traefik.tcp.routers.mqtts.rule=HostSNI(
broker.hetznerhcm.dima.kmd.dk
) - traefik.tcp.routers.mqtts.tls.passthrough=true - traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883 - traefik.tcp.routers.mqtts.service=mqtts-svc - traefik.tcp.routers.mqtts.entrypoints=websecure what I can see in my docker-compose.yml for mq block, therer's no ports and its shared certificates. so I need to update these changes and spin up the docker compose again?
@jolly-london-20127 your comment here please !! 🙂
j
config looks correct
actually...valid question to @bored-island-21407 and @quiet-continent-24852 , I think we took 1883 out of the default docker compose but should we leave it in there?
in any case it should still work, I would test the other troubleshooting steps @cool-army-24422
c
Ok @jolly-london-20127 , please let me know. if you find any
b
servermqport defaults to 1883 if not specified ... no need to put in docker compose
j
yeah but we no longer expose on MQ by default
is that still reachable over the docker network? we used to expose 127.0.0.1:1883
ehh nvm it looks like we haven't done that for a while
b
you are correct; MQ_SERVER_PORT should be set to 8883
or need to add ports 127.0.0.1:1883 to mq section
c
root@Netmaker-dev:~# docker logs mq 1657540466: mosquitto version 2.0.11 starting 1657540466: Config loaded from /mosquitto/config/mosquitto.conf. 1657540466: Opening ipv4 listen socket on port 8883. 1657540466: Opening ipv6 listen socket on port 8883. 1657540466: Opening ipv4 listen socket on port 1883. 1657540466: Opening ipv6 listen socket on port 1883. 1657540466: mosquitto version 2.0.11 running 1657540467: New connection from 172.18.0.2:44778 on port 1883. 1657540467: New client connected from 172.18.0.2:44778 as vO45Z7mt1hgzrqd1frc6dGj (p2, c1, k60). 1657693023: New connection from 172.18.0.2:46242 on port 1883. 1657693023: New client connected from 172.18.0.2:46242 as b7axCSqQCzrosEE90zkWtoy (p2, c1, k60). 1657693023: Client b7axCSqQCzrosEE90zkWtoy disconnected. 1657693025: New connection from 172.18.0.3:34764 on port 8883. 1657693025: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate 1657693025: Client disconnected: Protocol error. 1657693029: New connection from 172.18.0.3:34766 on port 8883. 1657693029: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate 1657693029: Client disconnected: Protocol error. 1657693033: New connection from 172.18.0.3:34768 on port 8883. 1657693033: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate 1657693033: Client disconnected: Protocol error. 1657693037: New connection from 172.18.0.3:34770 on port 8883. 1657693037: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate 1657693037: Client disconnected: Protocol error.
if this may help you, I see mq version is not the latest
j
I think I was wrong on this. Even in 0.14.0 we were not setting 1883: https://github.com/gravitl/netmaker/blob/v0.14.0/compose/docker-compose.contained.yml
b
not an issue
j
he probably just needs to re-gen certs then
c
how to do this?
b
the troubleshoot gist has steps
c
aaaha yeah my bad 😇
root@Netclient-AKS-DEV-RG:~# netclient join -t eyJhcGljb25uc3RyaW5nIjoiYXBpLmhldHpuZXJoY20uZGltYS5rbWQuZGs6NDQzIiwibmV0d29yayI6Im9yay1henVyZSIsImtleSI6IjFmMzYwODEwN2RhNDFhYzgiLCJsb2NhbHJhbmdlIjoiIn0= [netclient] 2022-07-15 11:47:47 joining ork-azure at api.hetznerhcm.dima.kmd.dk:443 [netclient] 2022-07-15 11:47:47 UDP hole punching enabled for node Netclient-AKS-DEV-RG [netclient] 2022-07-15 11:47:47 starting wireguard [netclient] 2022-07-15 11:48:19 unable to connect to broker, retrying ... Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time=18.309649ms Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time=18.169549ms Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time=19.054748ms [netclient] 2022-07-15 11:48:23 could not connect to broker broker.hetznerhcm.dima.kmd.dk connect timeout [netclient] 2022-07-15 11:48:23 connection issue detected.. attempt connection with new certs and broker information [netclient] 2022-07-15 11:48:23 certificates/key saved [netclient] 2022-07-15 11:48:55 could not connect to broker at broker.hetznerhcm.dima.kmd.dk:443 [netclient] 2022-07-15 11:48:55 failed to publish update for join connection timeout After re-generating the certs and restarting the netmaker and mq image still not able to connect to the broker
j
in your netmaker config, is SERVER_NAME set to the dns entry for the broker?
c
yes SERVER_NAME: "broker.hetznerhcm.dima.kmd.dk"
I think it's appropriate
j
Did you try the mosquito client connection test in the gist?
c
nslookup broker.hetznerhcm.dima.kmd.dk Server: 127.0.0.53 Address: 127.0.0.53#53 Non-authoritative answer: Name: broker.hetznerhcm.dima.kmd.dk Address: 49.12.241.214
is it a positive sign
Hi @User, Please suggest any fix. Thanks.
Hi @jolly-london-20127 need your comment here please 🙂
b
can you connect to the broker from a node using the mosquitto client as outlined in the troubleshooting gist?
c
sure
root@Netmaker-dev:~# sudo docker-compose up -d traefik is up-to-date netmaker is up-to-date coredns is up-to-date netmaker-ui is up-to-date Starting mq ... error ERROR: for mq Cannot start service mq: mkdir /run/containerd/io.containerd.runtime.v2.task/moby/86746f4f3878884fc77a9cbab6fb6f3ac641aa16a10e3d15529b5db95d6bee74: file exists: unknown ERROR: for mq Cannot start service mq: mkdir /run/containerd/io.containerd.runtime.v2.task/moby/86746f4f3878884fc77a9cbab6fb6f3ac641aa16a10e3d15529b5db95d6bee74: file exists: unknown ERROR: Encountered errors while bringing up the project.
unable to start mq itself now
b
something is wrong with your docker setup. I believe you need to purge everything as start over
c
@bored-island-21407 thanks for the suggestion, will start from scratch again
[netclient] 2022-07-19 10:50:42 joining kmd-net at api.hetznerhcm.dima.kmd.dk:443 [netclient] 2022-07-19 10:50:42 UDP hole punching enabled for node Netclient-AZ [netclient] 2022-07-19 10:50:42 starting wireguard [netclient] 2022-07-19 10:51:14 unable to connect to broker, retrying ... Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time= 18.138254ms Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time= 19.364619ms Ping tcp://broker.hetznerhcm.dima.kmd.dk:443(49.12.241.214:443) - Connected - time= 17.148702ms [netclient] 2022-07-19 10:51:18 could not connect to broker broker.hetznerhcm.dima. kmd.dk connect timeout [netclient] 2022-07-19 10:51:18 connection issue detected.. attempt connection with new certs and broker information [netclient] 2022-07-19 10:51:18 certificates/key saved [netclient] 2022-07-19 10:51:50 could not connect to broker at broker.hetznerhcm.dima.kmd.dk:443 [netclient] 2022-07-19 10:51:50 failed to publish update for join connection timeout root@Netclient-AZ:~# -------------------------------------------------------- I restarted and spinned up all docker conatiners but still I can see connection issue with the broker Is there any problem with the domain?
@jolly-london-20127 @bored-island-21407 would you like to suggest something here ? 🙂
b
need more information ... log outpput from netclient node, netmaker container and mq container. did you follow all the steps in the mq troubleshooting gist?
c
yes
b
can you connect from a mosquitto client?
c
can we connect over a call if you have any feasible time ?
b
what were the results of your actions when following the mq troubleshooting gist
c
root@Netmaker-dev:~# docker logs mq 1658226829: mosquitto version 2.0.11 starting 1658226829: Config loaded from /mosquitto/config/mosquitto.conf. 1658226829: Opening ipv4 listen socket on port 8883. 1658226829: Opening ipv6 listen socket on port 8883. 1658226829: Opening ipv4 listen socket on port 1883. 1658226829: Opening ipv6 listen socket on port 1883. 1658226829: mosquitto version 2.0.11 running
b
so no connections in the mq logs at all?
c
there are many
b
and have you tried regenerating certificates for netmaker/mosquitto?
c
yes
and I spinned up everything from scratch again
b
regenerated certs and restart mq
then do netclient pull on clients
c
Yeah I did all the process
root@Netclient-AZ:~# netclient pull -n kmd-net [netclient] 2022-07-19 12:19:53 UDP hole punching enabled for node Netclient-AZ [netclient] 2022-07-19 12:19:55 certificates/key saved root@Netclient-AZ:~#
b
and you successfully connected to the broker with mosquitto client but netclient still has errors?
c
Yes
--------------------- Hey Guys, A little update and deviation from the discussion topic. It's observed that the machine where Netmaker server was configured has been compromised. We are analysing into the potential threat. After that we will be resuming the POC of creation of netmaker networking. Thanks for your support
PreviousNext
Powered by