Secure access to your tooling components with IAP in GKE

What is Identity-Aware Proxy (IAP)?

Identity-Aware Proxy (IAP) is a Google Cloud service that allows guarding access to Virtual Machines (VMs) and applications using users’ identity and context. It is often used to allow connections from computers to VMs without actually requiring exposing these VMs on the public Internet.

However, it is also useful to guard access to applications by specifying who can access them directly in GCP IAM.

IAP can be plugged into a load balancer targeting services in a cluster to secure access to specific components running in Kubernetes.

How to use IAP to secure access to tooling components running in GKE?

This part describes how to set up a GKE cluster with an Ingress controller plugged behind an HTTP(S) (layer 7) load balancer with IAP, used to secure access to tooling components. The following examples are given using Ingress-NGINX Controller but another Ingress controller can be used.

Create the IAP OAuth credentials

IAP on a load balancer requires an IAP OAuth client and credentials for this client. Everything can be created as code using Terraform:

resource "google_iap_brand" "this" {
  support_email     = "<support email>"
  application_title = "Cloud IAP protected app"
  project           = var.project_id
}

resource "google_iap_client" "iap" {
  display_name = "<IAP client name>"
  brand        =  google_iap_brand.this.name
}

resource "google_secret_manager_secret" "iap_id" {
  secret_id = "iap-id"
  project = var.project_id

  replication {
    automatic = true
  }
}

resource "google_secret_manager_secret_version" "iap_id" {
  secret = google_secret_manager_secret.iap_id.id
  secret_data = google_iap_client.iap.client_id
}

resource "google_secret_manager_secret" "iap_secret" {
  secret_id = "iap-secret"
  project = var.project_id

  replication {
    automatic = true
  }
}

resource "google_secret_manager_secret_version" "iap_secret" {
  secret = google_secret_manager_secret.iap_secret.id
  secret_data = google_iap_client.iap.secret
}

The IAP credentials are stored securely in the Secret Manager where they can be retrieved from the cluster with External Secrets for instance.

Setup the Ingress controller with IAP

The Ingress controller and the load balancer with IAP can directly be created using Kubernetes resources.

The first thing is to retrieve the IAP OAuth credentials from the previous step so that they can be used by the load balancer. This can be done with External Secrets:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gcp-external-iap-secret
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    # This supposes that a ClusterSecretStore "secret-store" is already in the cluster and plugged with GCP Secret Manager
		# See https://external-secrets.io/v0.8.3/provider/google-secrets-manager/ to learn how to configure it
    name: secret-store
  target:
    # Name of the Kubernetes secret ref                                                             
    name: iap-secret
    creationPolicy: Owner
  data:
  - secretKey: client_secret
    remoteRef:
      key: iap-secret
  - secretKey: client_id
    remoteRef:
      key: iap-id

Then to configure the load balancer, three GKE-specific resources must be added: a BackendConfig, a FrontendConfig, and a ManagedCertificate. These resources tell GCP how to configure the front end of the layer 7 load balancer and its backend.

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: ingress-nginx-iap
spec:
  # Automatically redirect HTTP to HTTPS
  redirectToHttps:
    enabled: true
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: managed-cert
spec:
	# Domain names of resources that will be accessed with IAP
  domains:
    - "<FQDN_1>"
    - "<FQDN_>"
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: ingress-nginx-iap
spec:
  iap:
		# Enable IAP
    enabled: true
		# Kubernetes secret ref
    oauthclientCredentials:
      secretName: iap-secret
  healthCheck:
    checkIntervalSec: 30
    timeoutSec: 5
    healthyThreshold: 1
    unhealthyThreshold: 2
    type: HTTPS
    requestPath: /healthz

To create the HTTP(S) load balancer, an ingress with the ingress class GCE must be added to the cluster. GKE clusters have a built-in GCE Ingress controller that creates layer 7 load balancers reading ingress configurations.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: iap-ingress
  annotations:
		# Redriect HTTP to HTTPS
    ingress.kubernetes.io/ssl-redirect: "true"
		# GCE Ingress class
    kubernetes.io/ingress.class: gce
		# Static IP to use for the load balancer
    kubernetes.io/ingress.global-static-ip-name: "<ip resource name>"
		# Name of the FrontendConfig to use
    networking.gke.io/v1beta1.FrontendConfig: ingress-nginx-iap
		# Name of the managed certificate to use
		networking.gke.io/managed-certificates: managed-cert
spec:
  defaultBackend:
    service:
			# Name of the ClusterIP service the load balancer targets
      name: ingress-nginx-iap-controller
      port:
        name: https
	# Secret with the certificate to use for the load balancer (if generated by cert-manager)
	# See https://cert-manager.io/docs/concepts/certificate/
  # tls:
  #   - secretName: <certificate>

Finally, the Ingress-NGINX controller must be configured with additional information for the load balancer to use IAP. The values of the Helm chart can be updated for that:

...
controller:
	...
  service:
    type: ClusterIP
    annotations:
			# Protocol for communications between the LB and the Ingress Controller
      cloud.google.com/app-protocols: '{"https":"HTTPS"}'
			# Name of the BackendConfig (⚠️ Important for IAP to work!)
      cloud.google.com/backend-config: '{"default": "iap-ingress"}'
			# To use container native load balancer
			# See https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing
			cloud.google.com/neg: '{"ingress": true}'

  # Election ID used to elect leader instance (if multiple instances of the same Ingress Controller)
	# 🚧 This value has to differ between two Ingress Controllers
  electionID: iap

  # This section refers to the creation of the IngressClass resource
  # IngressClass resources are supported since k8s >= 1.18 and required since k8s >= 1.19
  ingressClassResource:
    # Name of the ingressClass
		# 🚧 This value has to differ between two Ingress Controllers
    name: iap
    # Controller-value of the controller that is processing this ingressClass
		# 🚧 This value has to differ between two Ingress Controllers
    controllerValue: "k8s.io/ingress-nginx-iap"

  # For backwards compatibility with ingress.class annotation, use ingressClass.
  # Algorithm is as follows, first ingressClassName is considered, if not present, controller looks for ingress.class annotation
	# 🚧 This value has to differ between two Ingress Controllers
  ingressClass: iap

Grant access to the load balancer with IAP

Users still need to be authorized to access the Ingress controller (and tooling components). They need to be given the role roles/iap.httpsResourceAccessor either in the scope of the project or the scope of the load balancer’s IAP application resource.

Make tooling components accessible through the IAP-protected Ingress controller

The final step is to create the right Ingress resources for the tooling components to be reachable from the Ingress controller configured with IAP. It is exactly the same thing as configuring any ingress, except the IngressClass iap must be specified on the Ingress resource.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
	name: iap-protected-ingress
spec:
	# Ingress class of the Ingress controller with IAP
  ingressClassName: iap
  rules:
  - host: "<host>"
    http:
      paths:
      - path: /
        pathType: Prefix
				backend:
          service:
            name: "<service to target>"
            port:
              name: "<service port to target>"

Benefits and drawbacks

The setup that is shown in the previous part has several benefits and drawbacks.

Benefits

• Tooling components are not exposed on the Internet (only connections from authorized users can reach tooling components).
• Access to tooling components is entirely granted through IAM.
• No need to manage an additional component like OAuth2 Proxy in the cluster for a similar need.
• Once the IAP-protected Ingress controller is set up, it is easy to protect new tooling components.

Drawbacks

• User identity is forwarded using headers that can be used for RBAC on applications, but not all components can use them securely. User information can be leveraged by tooling components to grant rights on their UI. X-Goog-Authenticated-User-Email forwards the user's email address and X-Goog-Iap-Jwt-Assertion forwards information on the authenticated user through a signed JWT. Unfortunately, some components (like ArgoCD) are not able to decode JWT headers to access user information. Relying on the X-Goog-Authenticated-User-Email header is way less secure as it does not provide proof of authenticity like the JWT signature that can be verified. This means the right Kubernetes RBAC and Network Policies must be set in the cluster to prevent users to craft their own headers when accessing components.
• groups information is not forwarded by IAP. Components that provide Role-Based Access Control (RBAC) need to be configured on a per-user basis, or a dedicated component to retrieve groups must be installed in the cluster (like Dex for instance).
• Authentication tokens obtained with IAP are limited to a single domain. This is not a problem with tooling components, as APIs and front-ends use the same domain. However, it is not suitable for applications that use XHR to communicate with backends under other domain names.

Conclusion

Using IAP to protect access to the web resources of GKE tooling components is interesting to easily secure and protect them while following Zero Trust principles. However, this method has drawbacks that cannot be ignored.

Especially, it does not forward groups information that allows for efficient management of user rights. It is not too much of an issue when only a few people manage the infrastructure, but it is not manageable at scale.

On large infrastructure, combining IAP with Dex can be a good alternative: IAP grants access to components from the Internet while Dex authenticates users and retrieves the necessary information (e.g. groups) for authorization (performed by components).

Of course, IAP alone is not sufficient to fully protect clusters and components. It comes as an additional layer to other mechanisms such as Role-Based Access Control (RBAC), Network Policies, or workload security hardening.