Helloooo!

Lately I’ve been getting a lot of requests from customers on how they can leverage NSX-ALB to specify a Static IP Address on their L7 Ingress Objects in a Kubernetes based environment. So I decided to deep dive into it.

It’s no secret that you can specify a preferred Static IP Address for L4 LoadBalancers in Kubernetes as follows:

But what about L7 Ingress Objects? Can NSX-ALB help here?

The short answer is ‘Yes’! Find out how below!

Many thanks to Roberto Casula from the Advanced Load Balancing Team over at VMware for all the explanations around it!!

1. Terminology
2. Prerequisites
3. Static (Preferred) IPs on Dedicated Virtual Services
4. Static (Preferred) IPs on Shared (Sharded) Virtual Services
5. Summary

Terminology

• Shared Virtual Service: Also known as a Sharded Virtual Service is a Virtual Service with children! Meaning you have a Parent Virtual Service where the Virtual IP Address (VIP) lives; and this Parent VS has different Child Virtual Services behind the same VIP. So basically you will have a Parent VS where your different Ingresses live under as Child Virtual Services.
  • On Shared Virtual Services you can only specify a Static (or Preferred) IP Address for the Parent VS and not for your Child VS behind the Parent VS. So you can specify a Static IP per Shared VS Name, but not based on your (Child) Ingress FQDN Names.
• Dedicated Virtual Service: In this case each Ingress gets its own Dedicated Virtual Service and corresponding dedicated VIP.
  • On Dedicated Virtual Services you can specify a Static (or Preferred) IP Address for each Ingress Object (FQDN) you specify. Since each FQDN (Ingress Object) will get their own Virtual Service in NSX-ALB.
• Preferred IP Address: The term Static IP Address & Preferred IP Address can be used interchangeably in this context. Basically we are going to tell to NSX-ALB which Static IP Address we would like / prefer to have on our VS. It’s a Preferred IP Address in the official docs because it’s a preference you specify. If the IP Address is not within the IPAM range or already taken by another VS; then you will not get this IP on your VS.

Let’s get started shall we?

Prerequisites

• Make sure to have an NSX-ALB environment deployed and to have a compatible form of AKO in your (Tanzu) Kubernetes Cluster.
  • How to install AKO in your (Tanzu) Kubernetes Cluster can be found here
  • The AKO Compatibility Guide (for a specific AKO Version) can be found here
• Also make sure to have an Application that you would like to expose externally. In my case: HTML 5 SpaceInvaders!

In my environment I will be using NSX-ALB 21.1.4 together with vSphere with Tanzu on vSphere 7 and I have AKO 1.7.2 installed on a Tanzu Kubernetes Cluster 1.22.9.

Static (Preferred) IPs on Dedicated Virtual Services

If each Ingress you create on Kubernetes side should be having its own Dedicated Virtual Service and corresponding VIP, then you’re in the good section. It’s all about ShardSize.

In this case we want to have the option ‘ShardSize’ set to ‘Dedicated’ because we want to have a Dedicated Virtual Service per Ingress we define. You have various ways of modifying the ShardSize (e.g.: when deploying AKO) but using AviInfraSetting CRDs gives you a lot of flexibility.

1. Create an AviInfraSetting Custom Resource in your (Tanzu) Kubernetes Cluster specifying shardSize: Dedicated

An AviInfraSetting is a Custom Resource that is available to define when you install AKO. In an AviInfraSetting you can define things like the VIP Network to use or also the shardSize. More information about AviInfraSetting CRDs can be found here.

An example is shown below:

apiVersion: ako.vmware.com/v1alpha1
kind: AviInfraSetting
metadata:
  name: dedicated-vs-infra-setting # Give it a meaningful name for you
spec:
  seGroup:
    name: Default-Group # Make sure this matches your Service Engine Group that you've configured in Avi / NSX-ALB!
  network:
    vipNetworks:
      - networkName: "AVI - FrontEnd 60" # Add your Network Name here
        cidr: 10.10.60.0/24 # Add your CIDR here
    enableRhi: false
  l7Settings:
    shardSize: DEDICATED # Set the shardSize to DEDICATED for a Dedicated VS per Ingress

Let’s apply it!

kubectl apply -f AviInfraSetting-dedicated-vs.yaml

2. Create an IngressClass that leverages the AviInfraSetting Object

In order to use our AviInfraSetting Object when deploying Ingress Objects, we need to specify an IngressClass which references the AviInfraSetting.

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: dedicated-vs-ingress-class # Give it a meaningful name for you
spec:
  controller: ako.vmware.com/avi-lb # By default your AKO Ingress Controller is named 'avi-lb'
  parameters:
    apiGroup: ako.vmware.com # Default API Group for AviInfraSetting
    kind: AviInfraSetting # Set the Kind to AviInfraSetting
    name: dedicated-vs-infra-setting # Specify the Dedicated ShardSize / VS AviInfraSetting Object Name here

Let’s apply it!

kubectl apply -f dedicated-vs-ingressclass.yaml

3. Create a HostRule Object specifying the Preferred Static IP Address for your Ingress

A HostRule is a Custom Resource that allows you to specify additional properties for your L7 Ingress Objects in Kubernetes Environments. Those Additional Properties could be specifying a WAF Policy for example or specifying a Preferred Static IP Address for new (to be provisioned) Ingress Objects with a given FQDN.

That’s what we’re going to use HostRules for in this Post. More information can be found here.

Create a new HostRule Object in your (Tanzu) Kubernetes Cluster as follows:

apiVersion: ako.vmware.com/v1alpha1
kind: HostRule
metadata:
   name: my-host-rule-static-ip
   namespace: default
spec:
  virtualhost:
    fqdn: ingress-space-8.avi.potus.local # Mandatory, must match a Kubernetes Ingress Object's FQDN that you will create in a later step
 
    tcpSettings:
      listeners:  # It's mandatory to specify ports and to have at least 1 port with SSL Enabled! Otherwise your HostRule will not be deployed
        - port: 80
        - port: 443
          enableSSL: true 
      loadBalancerIP: 10.10.60.37 # Specify the Preferred Static IP that your future L7 Ingress Object should have; MUST be within your defined IPAM Range + MUST not be in use already

Note: Make sure to specify at least 1 Port with SSL Enabled. Otherwise your HostRule will get the status “Rejected”!

Let’s apply it!

kubectl apply -f hostrule-static-ip-on-dedicated-vs.yaml 

And make sure that the status is “Accepted”.

kubectl get hostrule -o wide

If the status of your HostRule is ‘Rejected’, you can inspect a potential reason by running ‘kubectl describe hostrule’

Now we have everything in place to deploy our Ingress Object! It’s also completely normal to not yet see anything in your NSX-ALB UI yet 😉

4. Create an Ingress Object with the same FQDN as specified in the HostRule

Let’s create our Ingress Object:

apiVersion: networking.k8s.io/v1
kind: Ingress # Set Type to Ingress
metadata:
  name: dedicated-ingress-example # Give it a meaningful name for you
  namespace: default
spec:
  ingressClassName: dedicated-vs-ingress-class # MANDATORY: Specify t he Dedicated Virtual Service Ingress Class we applied earlier
  rules:
  - host: ingress-space-8.avi.potus.local # MANDATORY: This host / FQDN MUST match with your HostRule Object
    http:
      paths:
      - backend:
          service:
            name: spaceinvaders-clusterip
            port:
              number: 8080
        path: /
        pathType: Prefix

Let’s apply it!

kubectl apply -f dedicated-ingress-example.yam

Let’s watch our Ingress Resources from Kubernetes to see if we get our Preferred Static IP!

kubectl get ingress -w

If all went well then we should see that our Ingress Object gets the Static IP Address that we configured in our HostRule Object.

5. What happened in NSX-ALB / Avi?

Let’s find out!

And Browsing to it works!

Static (Preferred) IPs on Shared (Sharded) Virtual Services

We can also specify Preferred Static IP Addresses for our Shared (Sharded) Virtual Service. In this scenario we cannot specify it per FQDN / host! Your Shared Virtual Service usually has the following naming convention in NSX-ALB:

<cluster-name>–Shared-L7-<ingress-class-name>-<number>
e.g.:
tkgs-shared-services-001-Shared-L7-myIngressClass-0

So how do we tell AKO to create Shared Virtual Services?
Usually your Default Ingress Class with AKO installations has the shardSize set to ‘Shared Virtual Services’. This depends on the AKO Values.yaml you used during the deployment of AKO in your environment.

However we can also specify our own IngressClass where we set a shardSize leveraging an AviInfraSetting.

The different Shard Size Options are listed below:

More information can be found here.

So basically to tell AKO to use Shared Virtual Services with specific Ingress Classes, we can create our AviInfraSetting Object in the same manner as with Dedicated Shared Virtual Services and just adjust the Name and shardSize setting to anything else then DEDICATED.

Once you’ve done that you can follow the same procedure as for the Dedicated Virtual Service by creating an IngressClass linking to your AviInfraSetting and creating a HostRule followed by an Ingress that uses our Shared Virtual Service IngressClass. In your HostRule Definition be sure to put your ‘Shared Virtual Service’ name in the FQDN field (see naming convention above) and not the Ingress Hostname. You can also work with wildcards in the FQDN field (aka for each Shared Virtual Service do this 🙂 ).

Summary

Do you see how it all comes together?

1. We can specify which type of Virtual Service (Dedicated vs Shared) we would like to have in AviInfraSetting Objects
2. Then we can use IngressClasses to make sure that our Ingress Objects are using the specific AviInfraSettings we defined.
3. And before we deploy our L7 Ingress Object we can specify a HostRule Object to highlight which Static IP Address we would like to have.
4. And then we can deploy our L7 Ingress Object to see the magic happen!

Of course as you might have guessed by now: yes you can use AviInfraSetting CRDs & HostRule CRDs for A LOT of other things too. Happy deep diving!

Hope it helped you in achieving a Static IP Address on your Virtual Service for your L7 Ingress Object in Kubernetes.

• K8s