Egress to public IP namespaces

Question

bored-island-21407 · Answer

You can have multiple ingress You can have multiple egress as well as long as the egress ranges do not overlap or you use acls to control which other nodes in the network can reach the egress gateway

acoustic-easter-26071 · Answer

I am not sure about the overlap part Can you explain it a bit more please

acoustic-easter-26071 · Answer

my use case is actually I have 5 nodes in 5 different regions UK US AU etc and they are part of my k3s cluster I want to use these 5 nodes to make ingress and egress so that I can access the public internet through any region like a VPN with the netmaker external clients Also want to make sure that these 5 nodes can communicate with each other also

acoustic-easter-26071 · Answer

Also I would like to access the cluster network from these same endpoints Suppose my k3s cluster network is on 10 40 0 0 16 and services at 10 41 0 0 16 I would like to somehow be able to use the external clients to access these cluster pods and services directly and maybe limit which external clients can actually access these cluster networks through ACL Will that be possible in this scenario

bored-island-21407 · Answer

Once you set up one egress to be a VPN all traffic to public internet from all nodes will be be routed through the egress network To do what you want set up two networks One that allows the 5 nodes to communicate and another using acls in which the nodes cannot communicate and create the gateways on the second network Alternatively create 6 networks The first to allow the 5 nodes to communicate and the next 5 would just have a single node which acts as the ingress egress for its region

acoustic-easter-26071 · Answer

Actually the inter nodes commuinication is not a must to have for starter In my k3s cluster these nodes are working fine and communicating with each other I was thinking about this to make sure that those ext clients can communicate with all these 5 nodes But now when I think about it it doesn t actually need to Also I didn t find any good guide on netmaker docs and netmaker k8s yaml files to implement multiple connections on the netclient daemonset I was also thinking about doing PRs when I get this whole setup working so that you guys can keep it as an example doc for other people

acoustic-easter-26071 · Answer

one more thing I wanna say that yesterday I almost crashed my whole k3s cluster trying to do this I think I should not have done all the networking on my host level But the default daemonset yaml file does everything in host level > https github com gravitl netmaker blob master k8s client netclient daemonset yaml

acoustic-easter-26071 · Answer

my k3s installation is already running on flannel wireguard backend

bored-island-21407 · Answer

Any contributions to docs would be most welcome

acoustic-easter-26071 · Answer

I really want to contribute more on the k8s side of things But have to figure out first how to achieve this complex network first

bored-island-21407 · Answer

That would also be we comed

acoustic-easter-26071 · Answer

any idea how do I join multiple networks through that daemonset yaml

acoustic-easter-26071 · Answer

the dockerhub image for netclient also does not document all the environment variables the image can take

bored-island-21407 · Answer

My knowledge of k8 is limited Need < jolly london 20127> to jump in

bored-island-21407 · Answer

https github com gravitl netmaker blob master compose docker compose reference yml

acoustic-easter-26071 · Answer

He seems to be offline right now I will just state the network architecture I am trying to achieve and he might reply when he comes online

acoustic-easter-26071 · Answer

it is for the server Any similar one for the netclient And also what s the difference between netclient and the netclient go image

bored-island-21407 · Answer

Netclient go uses userspace wireguard rather than kernel so it will be slower

acoustic-easter-26071 · Answer

I understand

bored-island-21407 · Answer

netclient help

acoustic-easter-26071 · Answer

alright I can see all the parameters now

acoustic-easter-26071 · Answer

but does all these parameters map directly to docker environment variables

bored-island-21407 · Answer

Short answer no

acoustic-easter-26071 · Answer

https github com gravitl netmaker blob master k8s client netclient daemonset yaml ok I see what is happening here basically netclient is saving a configuration for all its networks and settings I think inside the etc netclient folder And the folder is mapped inside the host machine So if i need to join multiple networks I need to manually execute the netclient join command as many times as I want inside the netclient container and it will save the configuration on host for persistence Later even if the container dies and reschedules it will still have access to those configuration files and it will use them to bootstrap the join for all the networks

acoustic-easter-26071 · Answer

I might be wrong Hope < jolly london 20127> can correct me on this

jolly-london-20127 · Answer

yes < acoustic easter 26071> you figured it out You have 2 options 1 execute the join command from the running netclient containers 2 deploy a second daemonset but mount a different host directory for instance etc netclient 2

acoustic-easter-26071 · Answer

yeah but if one netclient can join multiple networks then I should be a little conservative about the resources and not deploy multiple I guess

jolly-london-20127 · Answer

yes that s true

jolly-london-20127 · Answer

option 2 would be a little more kubernetes native since typically you dont want to execute commands in a running container to keep it stateless but I think option 1 makes more sense here

acoustic-easter-26071 · Answer

but the daemonset yaml is making it stateless anyway because it is storing everything in a hostpath So even if the container dies and spins up again it can still get those configurations and join the networks

acoustic-easter-26071 · Answer

< jolly london 20127> https github com gravitl netmaker blob master k8s server netmaker server yaml in this yaml file there are many containerports defined which are for wg interfaces I suppose starting from 31821 and using the nodeport service for exposing them May I know what these ports are being used for exactly and is it possible to change these ports to different containerports

acoustic-easter-26071 · Answer

And how do I turn off hostnetwork usage on this > https github com gravitl netmaker blob master k8s client netclient daemonset yaml because my k3s is deployed with flannel wg backend so I just wanna run all the netmaker networks contained inside the container itself and not conflict with my host wg network

acoustic-easter-26071 · Answer

don t you think we should use PVC for this instead of mounting directly to host paths to keep it kubernetes native I can do some PR to get the manifests updated with pvc mounts but not sure it will be accepted by you or not

jolly-london-20127 · Answer

this is for the netmaker node on the server You cannot currently change the range of these ports Netmaker on k8s will start joining networks at port 31851 and then iterate upwards so we make 10 available for this purpose

jolly-london-20127 · Answer

you will need to remove hostNetwork true from the yaml but I believe some other changes are necessary as well need to check

acoustic-easter-26071 · Answer

ok I understand 31821 or 31851

jolly-london-20127 · Answer

31821

acoustic-easter-26071 · Answer

got it

acoustic-easter-26071 · Answer

I was thinking remove the hostnetwork true and also use the hostnetwork env variable that is available at https github com gravitl netmaker blob master compose docker compose reference yml

jolly-london-20127 · Answer

for PVC I could not find an easy way to do this with DaemonSet However you could use StatefulSet or something else but then it s not guaranteed to run on all nodes

acoustic-easter-26071 · Answer

PVC with retention policy might do the trick actually Not really need statefulset

jolly-london-20127 · Answer

if you find a good way let me know would be happy to take a PR on that Agreed hostmount is not ideal

acoustic-easter-26071 · Answer

ok I will let you know

acoustic-easter-26071 · Answer

Oh for the SSL problem somehow I couldn t able to make it work fully Not sure what am I doing wrong here I have nginx ingress controller which is doing the TLS termination but from that ingress controller to both the API and UI I am just using non tls connection because everytime I try to go to https netmaker server 8081 without nginx ingress and any TLS terminationa I get SSL method error Same goes for the UI

acoustic-easter-26071 · Answer

i think somehow the deployment yaml should take a secret name so that it can access the tls secret key that nginx and cert manager generates and put it inside the nginx configuration of the container to have a fully secure stream from ingress until api

jolly-london-20127 · Answer

I usually track this down by describing the certificate

jolly-london-20127 · Answer

do kubectl describe certificaterequest <name> it will usually tell you where it s stuck

jolly-london-20127 · Answer

and typically it is because the certificate provider name is incorrect

acoustic-easter-26071 · Answer

netmaker is also using cert manager to generate the certificates

acoustic-easter-26071 · Answer

my ingress is generating the certificates perfectly and I am able to use https on the ingress end actually

acoustic-easter-26071 · Answer

but from ingress to netmaker endpoint is not secure because netmaker ssl is not working

jolly-london-20127 · Answer

what i am wondering is was the correct cert provider name used when creating the netmaker ingress

acoustic-easter-26071 · Answer

yes I think so because the acme challenge was successful and lets encrypt certificated was added to the secret Go to > vpn logicbuff com oh nevermind it is down now and you can check the certificate there I used the letsencrypt production issuer

jolly-london-20127 · Answer

hmmm ok

jolly-london-20127 · Answer

wait you are using 8081 to access the api

acoustic-easter-26071 · Answer

yes in the ingress I am using 8081 to acces the API backend service But the ingress uses port 443 externally to the UI and public

acoustic-easter-26071 · Answer

< jolly london 20127> have you checked if it is possible to disable host networking for the netclient daemon or not Please let me know