Fork me 🍴

Willian Antunes

GKE Ingress: How to fix a 502 bad gateway error

4 minute read

kubernetes, ingress, gcp

Table of contents
  1. Is the service running correctly?
    1. Checking backend health status and finding the root cause
  2. Understanding GKE behavior
  3. Fixing the problem
  4. Summing-up

Past few days, I was configuring a GKE cluster for my personal projects. Some of them had to be accessed through the internet; hence I created an ingress to do this job. Here's an example of ingress using Terraform:

resource "kubernetes_ingress" "default" {
  metadata {
    name = "sample-ingress"
    namespace = "production"

    annotations = {
      "" = "spacejam-cert-manager"
      "" = "my-honest-public-ip"

  spec {
    rule {
      host = ""
      http {
        path {
          backend {
            service_name = "spacejam-np-service"
            service_port = 8000

The service named spacejam-np-service was bound to a deployment like the following:

apiVersion: apps/v1
kind: Deployment
  name: spacejam-web-deployment
  namespace: production
  replicas: 1
      app: spacejam-web-deployment
        app: spacejam-web-deployment
        - name: spacejam-web-container
            - configMapRef:
                name: spacejam-configmap
            - name: http
              containerPort: 8000
              path: "/health-check"
              port: 8000
            initialDelaySeconds: 10
            timeoutSeconds: 5

Notice that it does not have a readiness probe entry 🤔. After creating all of the manifests, I tried to access the host, but I received a 502 error 😯.

Is the service running correctly?

My first idea was to check the service. Is it healthy or not? The command kubectl describe deployments spacejam-web-deployment returned that it was fine, then I ran another command kubectl port-forward deployment/spacejam-web-deployment 8000:8000 so I could access the service directly through http://localhost:8000/health-check. Everything was working as expected. My second idea was to understand how the ingress checks if a service is healthy or not.

Checking backend health status and finding the root cause

When you create an ingress, GKE will create a native load balancer afterward. You can know the LB's name through the command:

kubectl get ingress sample-ingress -o jsonpath='{.metadata.annotations.ingress\.kubernetes\.io/url-map}'

Then I opened its configuration, and I saw that one of my backends was unhealthy 👀:

The image shows all the backend services of the ingress native load balancer.

I could understand the root cause of the error when I noticed which path the backend service was using:

Health check attributes of a given backend service. The path attribute has the value of "/".

The targeted service returns 404 instead of 200 for the path /.

Understanding GKE behavior

Looking over GKE Ingress for HTTP(S) Load Balancing guide, the section Default and inferred parameters shows us that the ingress will check the POD's spec:


Then using it value to get the path to verify the service health, but if it's empty, it will use the default value /. Moreover, this statement is important:

When you expose one or more Services through an Ingress using the default Ingress controller, GKE creates a Google Cloud external HTTP(S) load balancer or a Google Cloud internal HTTP(S) load balancer. Both of these load balancers support multiple backend services on a single URL map. Each of the backend services corresponds to a Kubernetes Service, and each backend service must reference a Google Cloud health check. This health check is different from a Kubernetes liveness or readiness probe because the health check is implemented outside of the cluster.

Fixing the problem

To quickly solve it, I added the following to my deployment manifest:

    path: "/health-check"
    port: 8000
  initialDelaySeconds: 10
  timeoutSeconds: 5

Afterward, the backend service was updated to the correct path:

Health check attributes of a given backend service. The path attribute has the value of "/health-check".


There is a considerable debate on this very topic in the issue Ingress Healthcheck Configuration on kubernetes/ingress-gce repository. Looking through issues on GitHub is an excellent way to understand how an application or service is evolving. It's been more than three years, and it's not resolved yet.

In my next blog entry ✍, I'll explain how I configured IPv4 and IPv6 public addresses using only one ingress on GKE. Till next time!

Have you found any mistakes 👀? Feel free to submit a PR editing this blog entry 😄.