https://netmaker.org logo
Join Discord
Powered by
How should I configure the broker (mqtt) in regard...
# netmaker
l
How should I configure the broker (mqtt) in regards to the reverse proxy (nginx)? I can see that this is not a simple http-based config. In traefik:
Copy code
yml
      - traefik.enable=true
      - traefik.tcp.routers.mqtts.rule=HostSNI(`broker.NETMAKER_BASE_DOMAIN`)
      - traefik.tcp.routers.mqtts.tls.passthrough=true
      - traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883
      - traefik.tcp.routers.mqtts.service=mqtts-svc
      - traefik.tcp.routers.mqtts.entrypoints=websecure
As far as I understand, I take the 
raw stream
on port 443 and pass it to mqtt port 
8883
without any form of TLS decapsulation? Should nginx apply TLS or leave that to mqtt as well?
b
I am not 100% familiar with nginx proxying of non http traffic. I believe you just pass the raw stream.
l
I tried that, but when I access the broker via browser it returns 
NET::ERR_CERT_INVALID
is this expected?
b
yes, netmaker mqtt connections expect a client cert which your browser will not have. Best way to test would be with mosquitto_pub/mosquitto_sub.
see mq troubleshooting gist for instructions
l
Ah. So just to confirm I got this right, reverse proxy will determine whether the traffic is destined for MQ or any other service based on TLS SNI field, and if it's for MQ, just pass the raw stream without any alteration?
b
I believe that is correct
l
Just to follow up, it turns out that nginx cannot map raw streams and http streams on the same port, at least not without some significant refactoring of my existing configs... So I have exposed a custom port from MQTT directly and use that in the config.
Hmm, I'm pretty sure I did everything correctly, but now i keep getting this error whenever I connect my Windows client to the network
Copy code
1661029220: Client <unknown> disconnected: Protocol error.
1661029224: New connection from xxx:yyy on port zzz.
1661029224: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
I tried a full reinstall of all containers and volumes but nothing changed. I also tried the troubeshooting steps in above gist, but no help either.
b
when you do the troubleshooting steps --- do you get errors with mosquitto_pub?
l
I didn't really understand mosquitto_pub so I didn't test that, but I did try the sqlite delete part
b
when you deleted the certs and after you restarted the netmaker container, did you also wait and then restart the mosquitto container
l
define wait..? I waited 1-2 seconds
as fast as I could write the next restart command basically ^^
b
need to wait until netmaker has completely started.... should wait 10-30 seconds
l
sure, will try now and monitor logs
I changed the port from 8883 to a custom one and updated the mosquitto config as well as 
MQ_PORT
is that okay?
b
yes, that should work
l
Copy code
[netmaker] 2022-08-20 21:23:54 checking keys and certificates
[netmaker] 2022-08-20 21:23:54 generating new root key
[netmaker] 2022-08-20 21:23:54 generating new root CA
[netmaker] 2022-08-20 21:23:54 generating new server key/certificate
[netmaker] 2022-08-20 21:23:54 generating new server client key/certificate
[netmaker] 2022-08-20 21:23:54 ensure the root.pem, root.key, server.pem, and server.key files are updated on your broker
[netmaker] 2022-08-20 21:23:54 REST Server successfully started on port  8081  (REST)
[netmaker] 2022-08-20 21:23:54 connecting to mq broker at netmaker-mq:1883 with TLS? false
[netmaker] 2022-08-20 21:23:55 successfully connected to mq broker
now I restart mqtt?
b
yes
and then you need to do netclient pull -n on your client
l
can I just not delete the client network and use a new one?
that's what I've been doing until now
b
that works too
l
testing
nope same error
it spams bad cert until the client times out and continues without mqtt connection
b
do a netclient pull -n
l
pull succeeds but mqtt logs only show bad certificate
I am using Windows GUI fwiw
from console it just says
Copy code
[netclient.exe] 2022-08-20 22:27:22 could not connect to broker at website:port
b
dns or firewall
l
wait I think linux works
linux client that is
though I am getting a different error here:
Copy code
2022/08/20 22:29:53 daemon restart failed  failed to find pid could not read pid file open /var/run/netclient.pid: no such file or directory
[netclient-arm64] 2022-08-20 22:29:54 error running command: systemctl restart netclient.service 
[netclient-arm64] 2022-08-20 22:29:54 Failed to restart netclient.service: Unit netclient.service not found.
b
how did you install netclient on linux (distro?)
l
wget netclient-arm64 on Ubuntu 22.04 arm64
I thought netclient was portable?
b
run 
.netclient install
before you do the join or run 
./netclient join
with 
--deamon install
flag
l
ah gotcha, testing
yep so I can confirm mqtt works on linux
I can clearly see in the logs incoming connection over public port and the user's username
has windows been tested if it works with latest version?
b
yes it was
l
FWIW this is Windows arm64 ran in Parallels under MacBook Pro M1 Max
So netclient likely runs under emulation
WireGuard is fully arm64 though, so as far as I understand, there should be no issues as you just invoke the already installed (and native) wireguard
actually nvm, mqtt happens before wireguard anyway
hmm I just tried a clean install of netclient on a clean Ubuntu and now I get this:
Copy code
[netclient-arm64] 2022-08-20 22:38:09 network: mesh error setting cfg.Node.Endpoint. 
[netclient-arm64] 2022-08-20 22:38:09 error installing:  public address not found
b
that is output of which command? join? daemon?
l
join
so I did install first, worked, then the join returns this
b
run join with -vvvv flag
l
Copy code
[netclient-arm64] 2022-08-20 22:40:49 [join.go-91] JoinNetwork(): network: mesh error setting cfg.Node.Endpoint. 
[netclient-arm64] 2022-08-20 22:40:49 [commands.go-23] Join(): error installing:  public address not found 
[netclient-arm64] 2022-08-20 22:40:49 [systemd.go-124] RemoveSystemDServices(): removed systemd remnants if any existed
b
what does 
curl ifconfig.me
return on that machine
l
hmm actually it appears that the DNS somehow got broken on this machine
which would make sense, it can't resolve the IP for the server's hostname
I'll deal with that later, it worked once anyway
b
systemctl restart systemd-resolved
l
yeah fixed, interesting
joining also worked, no mqtt errors
so linux is sorted,. thanks! do you have any ideas why windows may be broken?
b
no idea... did you try the msi installer or just downloaded the exe from the release page?
l
so I tried exe first, but ran into: https://github.com/gravitl/netmaker/issues/1282
then I used MSI which didn't hit that error, but then cert issue arose
b
did you say you were running on arm64 .... the only windows exe we build is for amd64
l
yeah agreed, so netmaker is definitely running under emulation, but would that result in cert issues?
b
it shouldn't run at all
l
why not? the built-in Windows amd64 to arm64 emulation is incredibly good
I have yet to encounter any app that malfunctions or doesn't run
b
ok, i will take your word for it
l
the only issue I've encountered so far is with drivers, which for security and stability reasons will not be emulated
but netclient does not add any drivers, right? it only uses the existing wireguard install, and even then, through its CLI
b
netclient does not add any drivers
l
btw, it seems like 
netclient install
breaks my DNS on Ubuntu
I just tried again on a fresh Ubuntu 22.04 arm64 install, and before 
netclient install
, ping 
google.com
works, right after, it fails with 
Temporary failure in name resolution
b
netclient install should not affect your dns... all it does is copy the binary to /usr/sbin install the netclient.service file, systemctl daemon-reload, systemclt enable netclient
l
yeah, this is why it's strange
I can try again and record the commands, 1 sec
b
try 
systemctl --failed
after the install
l
Copy code
root@user:~# systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.
b
systemctl status systemd-resolved
l
(always wanted to do that)
I diffed 
systemctl status systemd-resolved
before and after, absolutely no difference
i.e. the service does not even get restarted
b
but systemctl restart systemd-resolved get dns working again, correct
l
yup
seems like systemd-networkd is restarted after netclient service installation though
here
b
that is weird ... the only thing that could possible to that I think is systemctl daemon-reload
l
I ran 
systemctl daemon-reload
and it did not break DNS
what does the netclient service depend on? it may be restarting something purely on dependency
b
Description=Netclient Daemon Documentation=https://docs.netmaker.org https://k8s.netmaker.org After=network-online.target Wants=network-online.target systemd-networkd-wait-online.service [Service] User=root Type=simple ExecStart=/sbin/netclient daemon Restart=on-failure RestartSec=15s [Install] WantedBy=multi-user.target
l
hmm nothing strange here
cannot reproduce on a cloud instance with Ubuntu amd64, so this is likely a quirk of my VM
btw, is the Windows standalone EXE supposed to work by itself? Or do you need to download 
winsw.exe
and put it next to it?
good news, I reset my Windows VM and now the mqtt certs work!
perhaps something broke during experimentation and netclient cached the "bad" certs on the Windows box
5 Views
PreviousNext
Powered by