Kubernetes Load Balancing

Posted: 11 Dec 2020 Updated: 21 Jun 2022 Verified: 21 Jun 2022

My notes on Kubernetes services and load balancing solutions for cloud and bare metal infrastructure.

Services

Effective scaling of Kubernetes applications involves a range of services that build upon each other to expose the applications that are running on each pod in the cluster onto a known internal or external IP.

ClusterIP

This provides an IP address endpoint that will route traffic to each pod of your deployment. This service is only exposed on the cluster’s internal network, and so can only be accessed internally, by other pods or services running within the k8s cluster.

NodePort

Creating a NodePort also creates a ClusterIP service. The NodePort service will take a randomly assigned high port, listen on that port on every node in the cluster, and take those requests and route them to ClusterIP it created, and from there to a pod associated with the service.

Like a ClusterIP, a NodePort is accessible by pods and services in the cluster, but also by clients on the same LAN (who can ping the host nodes).

A NodePort service in itself doesn’t solve the node load balancing issue, since you would still need to either use one node’s IP address to route requests through, or use another solution to load balance requests across all the nodes.

LoadBalancer

A LoadBalancer service will create a ClusterIP service, then a NodePort service, then create a LoadBalancer service that instantiates and configures a Load Balancer from your cloud provider. Every LoadBalancer k8s service provisions another load balancer instance from your cloud provider.

If you’re using k8s locally, or hosting k8s on prem, you may be wondering why your load balancer never establishes it’s external IP address property. Luckily, I’ve got that covered:

Why doesn’t my LoadBalancer work?

If you try running a LoadBalancer on your own bare metal k8s cluster, or you’ve created a dev environment with something like vagrant, you will find that kubectl get service <load-balancer-name> will be stuck with the external IP value as <pending>.

$ kubectl expose deployment blog --type=LoadBalancer --name=lb-blog --port 80
service/lb-blog exposed
$ kubectl get service lb-blog
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
lb-blog   LoadBalancer   10.43.176.99   <pending>     80:30130/TCP   43s

This is because it k8s requires an actual load balancing service to use.

This means that you need:

access to a cloud provider’s API to provision the load balancer.
custom, provider-specific provisioning calls
to set your k8s nodes security policy to allow this access.

This might be handled by your cloud provider’s modified version of k8s, or you’ll need to configure it yourself.

As a bare metal hoster, you will also need to provide the load balancing service yourself with something like metal lb.

An improvement to this system configuration is the use of an ingress service.

Ingresses

Ingresses are another way of routing traffic to your underlying services.

An ingress controller can be any number of systems that receive traffic and forward that traffic based on a set of routing rules. So, http load balancers like HAProxy, nginx, traefik, or API gateways like Kong or Tyk can provide ingress controller functions.

This allows you to create an internal load balancer, within your cluster. If you are using cloud services, this means that you don’t need to spin up a new instance for every LoadBalancer service you create.

Like a NodePort, an ingress controller will listen on multiple nodes on your cluster. It will then route requests to the services specified in the configuration. For example, if this is a NodePort service, it will route traffic to one of the high ports on one of the nodes. However, if one of these nodes is unavailable, the request will have been routed to a dead server. We still need to do health checks and failovers. That’s what the load balancer part of the set-up will do.

Ingress-nginx

K3s by default comes with traefik, but ingress-nginx is a good alternative.

If you don’t have it installed, you can use helm:

$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
$ kubectl create namespace ingress-nginx
namespace/ingress-nginx created
$ helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx
NAME: ingress-nginx
LAST DEPLOYED: Tue Sep 13 15:31:28 2022
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key
must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

$ kubectl get service ingress-nginx-controller

You will still need a load balancer provided by your cloud service provider, though. Although, if you use this ingress, instead of a LoadBalancer service, you will only need one, instead of one per service.

This is not clear or easy to understand and very poorly covered in most documentation and guides.

If you are implementing your own cluster on bare metal, you can use metallb.

Once you create an ingress server, and you have a domain name that routes requests to your server, you still need to tell the nginx server which service you want to point incoming requests from that domain name to.

This requires an ingress record. Here is an [[Kubernetes Configuration Management Methods|imperative command]] to create this, with the yaml equivalent:

$ kubectl create ingress blog --class=nginx --rule="blog.mehcoleman.com/*=blog:80"

$ kubectl create ingress blog --class=nginx --rule="blog.mehcoleman.com/*=blog:80" --dry-run=client -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  creationTimestamp: null
  name: blog
spec:
  ingressClassName: nginx
  rules:
  - host: blog.mehcoleman.com
    http:
      paths:
      - backend:
          service:
            name: blog
            port:
              number: 80
        path: /
        pathType: Prefix

This will create a host-based route, and route all incoming requests via blog.mehcoleman.com to the blog service connected to port 80.

N.b. You can also create path-based routing. Not shown here.

External-DNS

For self-managed kubernetes dns control, with supporting providers, check out GitHub - kubernetes-sigs/external-dns. Support is somewhat patchy at this stage, unless you go with a big provider.

Metal LB

Metal LB is a controller service that runs on your cluster.

You configure it with a list of spare IP addresses.

When you create a LoadBalancer service, Metal LB will take one of these IP addresses and announce one of the service’s node’s host’s corresponding MAC address as assigned to that IP address via ARP.

Then, if it detects that that node has become unavailable, it can failover by announcing a different node’s host to that IP address. Incoming requests will thus automagically connect to a working node.

Installation

$ kubectl create namespace metallb-system
namespace/metallb-system created

$ helm repo add metallb https://metallb.github.io/metallb
"metallb" has been added to your repositories

$ helm install metallb --namespace metallb-system metallb/metallb
NAME: metallb
LAST DEPLOYED: Mon Sep 12 21:36:16 2020
NAMESPACE: metallb-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
MetalLB is now running in the cluster.

Now you can configure it via its CRs. Please refer to the metallb official docs
on how to use the CRs.

There are installation edge-cases to be aware of. See the installation guide

Configuration

You need to reserve a block of LAN IP addresses that metallb will use to assign services to.

In this example, the cluster nodes are on the network 191.168.1.x

$ kubectl get nodes -o wide
NAME                   STATUS   ROLES                  AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
lima-rancher-desktop   Ready    control-plane,master   18d   v1.24.4+k3s1   192.168.1.62   <none>        Alpine Linux v3.16   5.15.57-0-virt   containerd://1.6.6

So, the IP addresses you reserve would come from this block.

So, we create a custom resource of type IPAddressPool. We can create as many of these as we need, and give them a name so that we can associate them with services later.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: extl-lb-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.240-192.168.1.250

$ kubectl create -f pool.yml

An L2Advertisement associates the block of IP addresses to the metal lb service.

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: anyname
  namespace: metallb-system
spec:
  ipAddressPools:
  - ext-lb-pool

If you were to omit the spec section with the first-pool identifier, k8s would assume all pools are available. (See MetalLB, bare metal load-balancer for Kubernetes)

Now, when you create a LoadBalancer service, it will correctly assign one of these IP addresses.

MEHColeman