Kubernetes Gateway API vs. Kubernetes Ingress

My colleague posted this meme on a Kubernetes group on LinkedIn the other day. It got an overwhelming response, amassing over 20,000 impressions and 150 reactions.

I believe it says a lot about how DevOps and architects are in dire need of a revamped native Kubernetes Ingress. And they do not enjoy writing custom annotations on Ingress, especially for the widely used ingress-nginx controller maintained by the Kubernetes community.

Kubernetes Gateway API is here to solve those problems. Let us see how Gateway API reimagines native Kubernetes Ingress by seeing a direct comparison.

Overview: Kubernetes Ingress and Gateway API

The objective of both types of API is to expose backend services for clients.

Kubernetes Ingress

The native Kubernetes Ingress resource defines the rules to route external HTTP/S traffic to respective services in the K8s cluster. It needs an Ingress controller running in the cluster to implement the rules. Ingress provides SSL termination, load balancing, and name-based virtual hosting capabilities.

Traffic flow with Kubernetes Ingress

Kubernetes Gateway API

The Gateway API splits the functionalities of Ingress resource to dedicated CRDs — GatewayClass, Gateway, and *Route. The Gateway acts as a network endpoint that takes the traffic from outside the cluster to inside the cluster; *Route resources help to define HTTP or TCP routes for the traffic from the Gateway to respective services. There are several implementors for the Kubernetes Gateway API.

Traffic flow with Kubernetes Gateway API.

Dimensions for Comparing Kubernetes Ingress With K8s Gateway API

I’m comparing the two based on four crucial dimensions:

1. Multitenancy 
2. Specifications 
3. Advanced traffic management 
4. Extensibility

Here is a brief tabular summary if you are in a rush:

1. Multitenancy

Ingress poses challenges when used in a shared Kubernetes cluster with multiple tenants. With each tenant working on a single Ingress resource, there is a good chance for one to misconfigure the resource and thus cause problems to another tenant’s route.

Kubernetes Gateway API prevents such conflicts by default with its role delineation nature. Using Gateway API, cluster admin or architect can configure the Gateway resource without sharing it with Devs/DevOps.

DevOps can then create and attach HTTPRoute or TCPRoute to receive the traffic from any Gateway to their apps, allowing them to configure routing rules without any conflicts. This clear separation of roles is much more effective than RBAC implementation with native K8s Ingress. 

(Read more about how to practice RBAC for infra, cluster admins, and DevOps in Gateway API.) 

2. Specifications

Kubernetes Ingress is annotation-heavy. Apart from basic traffic routing functionalities, everything else requires writing annotations, which can also vary based on the controller used. 

Custom annotations are no fun. It takes a good amount of time for its testing and implementation. Kubernetes Gateway API CRDs save the trouble by standardizing and simplifying the specifications to configure traffic routing rules without using annotations.

To give an example, take a look at the following Ingress configuration for header-based matching and routing:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: header-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      if ($http_custom_header = "value1") {
          proxy_pass http://service-1;
      }
      if ($http_custom_header = "value2") {
          proxy_pass http://service-2;
      }
spec:
  ingressClassName: nginx
  rules:
    - host: your-domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: default-backend
                port:
                  number: 80

The annotation in the Ingress resource is configured to look for two header values (value 1 or value 2) and route those requests accordingly to respective destinations (http://service-1 or http://service-2). The requests will be routed to the backend at port 80 if the specified header values are absent.

The same configuration is more simplified and declarative with Gateway API, using matches and backendRefs rules in HTTPRoute resource:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
 name: bar-route
spec:
 parentRefs:
   - name: gateway
 hostnames:
   - foo.example.com
 rules:
 - backendRefs:
   - name: default
 	port: 8080
 - matches:
   - headers:
 	- name: http_custom_header
   	value: value1
   backendRefs:
   - name: service1
 	port: 8080
 - matches:
   - headers:
 	- name: http_custom_header
   	value: value2
   backendRefs:
   - name: service2
 	port: 8080

Apart from not using annotations for core networking functionalities, Kubernetes Gateway API standardizes the specifications across implementations. That is, DevOps and architects need not change anything apart from the GatewayClass while migrating to a new controller implementation (such as Istio, Envoy Gateway, GKE, etc.); the Gateway API CRDs will remain the same.

The learning curve with the migration is also less compared to vendor-specific annotations and CRDs when using Ingress.

3. Advanced Traffic Management

Advanced traffic management means the ability to apply fine-grained rules and policies to the incoming traffic, including header modifications, flexible redirections, canary deployments, etc. 

Kubernetes Ingress provides only limited functionalities when it comes to advanced traffic management. For example, Ingress only supports one backend per route.

To implement a canary rollout, DevOps folks need to create two Ingress resources to route traffic to the stable version and the canary version at the same time, as shown below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-ingress-v1
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
    - host: your-domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-1
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-ingress-v2
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: your-domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-2
                port:
                  number: 80

The dedicated *Route resources in Gateway API open doors for DevOps to configure granular routing rules. K8s Gateway API natively allows advanced traffic management such as canary releases, A/B testing, traffic mirroring, cross-namespace referencing, etc. 

Unlike Ingress, Gateway API supports multiple backends to be defined in a single *Route resource. The above configuration that used two Ingress resources and annotations can be written as follows in HTTPRoute:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: canary-gw-api
spec:
  parentRefs:
    - name: gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: foo-v1
          port: 8080
          weight: 90
        - name: foo-v2
          port: 8080
          weight: 10

You can see the destinations specified under backendRefs, along with the weights.

Another advantage of Kubernetes Gateway API is that the backend need not be in the same namespace as the *Route resource. DevOps can now create a route to a service in another namespace, given that the destination namespace owner allowed such a route using ReferenceGrant:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: bar
  namespace: bar
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: foo
  to:
    - group: ""
      kind: Service
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: foo
  namespace: foo
spec:
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: bar-service
          namespace: bar

Here, the ReferenceGrant in the bar namespace allows HTTPRoute to its service from the foo namespace; HTTPRoute in the foo namespace creates a route to a service in the bar namespace.

Cross-namespace referencing using ReferenceGrant in Gateway API is a flexible way for DevOps to access services and other resources across namespaces.

4. Extensibility 

Ingress’ extensibility has always been using custom annotations. It is limited and tedious to configure. Gateway API far outweighs the extensibility capabilities of Kubernetes Ingress by allowing CRDs, custom filters, and policies. 

Here is an example HTTPRoute resource that has a custom rate limiting filter added under filters: 

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
   kind: Gateway
   name: k8s-gateway
  rules:
  - backendRefs:
   - group: ""
 	kind: Service
 	name: echoserver-service-v1
 	port: 80
   filters:
   - type: ExtensionRef
     extensionRef:
   	 group: stable.example.com
   	 kind: ratelimit
   	 name: limit-echoserver
   matches:
   - path:
   	type: PathPrefix
   	value: /

Here, I’m using a CRD to define the rate limiting configuration on the ehcoserver-service — using Envoy Gateway as implementation. Note that every Gateway API implementor will have its own way of implementing extensions.

Kubernetes Gateway API Is the Future of Ingress

Kubernetes Gateway API primarily standardizes the Ingress specifications that are scattered across annotations and vendor-specific implementations. 

Gateway API not only packs Ingress features that IT teams have been yearning for a while but also has the potential to become the standard for configuring service-to-service (east-west) communication.