Changing app-protocol from HTTP2 to other protocols will cause glbc to hit nil pointer

[freehan]

Symptom: Create an ingress with HTTP2 app protocol. For instance: https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/ingress/http2
Manually change the app protocol on the spec of service from HTTP2 to something like HTTPS,HTTP. This will cause glbc to hit nil pointer and crash with following stacktrace in ingress-gce 1.2:

/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:332
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:244
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:106
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:220
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:283
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:252
/go/src/k8s.io/ingress-gce/pkg/controller/cluster_manager.go:98
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:349
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:299
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:111
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:84
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1674e63]

goroutine 99 [running]:
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x19a12a0, 0x29aa020)
	/usr/local/go/src/runtime/panic.go:502 +0x229
k8s.io/ingress-gce/pkg/healthchecks.NewHealthCheck(...)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:332
k8s.io/ingress-gce/pkg/healthchecks.(*HealthChecks).Get(0xc42007bec0, 0xc420a8e3e0, 0x1e, 0x1cda028, 0x2, 0x0, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:244 +0x313
k8s.io/ingress-gce/pkg/healthchecks.(*HealthChecks).Sync(0xc42007bec0, 0xc42068b880, 0xc, 0xc420518c10, 0xb, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:106 +0xae
k8s.io/ingress-gce/pkg/backends.(*Backends).ensureHealthCheck(0xc420303960, 0xc420518b90, 0xc, 0xc420518c10, 0xb, 0x0, 0x1bb, 0x0, 0x0, 0x7f21, ...)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:220 +0xc0
k8s.io/ingress-gce/pkg/backends.(*Backends).ensureBackendService(0xc420303960, 0xc420518b90, 0xc, 0xc420518c10, 0xb, 0x0, 0x1bb, 0x0, 0x0, 0x7f21, ...)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:283 +0x233
k8s.io/ingress-gce/pkg/backends.(*Backends).Ensure(0xc420303960, 0xc4208ab980, 0x1, 0x1, 0xc4203d1b30, 0x1, 0x1, 0x1670ad8, 0x184a100)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:252 +0x1fd
k8s.io/ingress-gce/pkg/controller.(*ClusterManager).EnsureLoadBalancer(0xc4208f7680, 0xc4203f9b20, 0xc4208aa380, 0x1, 0x1, 0xc4203d1b30, 0x1, 0x1, 0x1, 0x1)
	/go/src/k8s.io/ingress-gce/pkg/controller/cluster_manager.go:98 +0x1ed
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).ensureIngress(0xc4204405a0, 0xc4208b7a20, 0xc420771c00, 0x3, 0x4, 0xc4208aa280, 0x1, 0x1, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:349 +0x4b6
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).sync(0xc4204405a0, 0xc42039dd80, 0x14, 0xc420699b01, 0xc420472d00)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:299 +0x3e5
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).(k8s.io/ingress-gce/pkg/controller.sync)-fm(0xc42039dd80, 0x14, 0xf, 0xc4208dddd8)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:111 +0x3e
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).worker(0xc420400240)
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:84 +0x17a
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).(k8s.io/ingress-gce/pkg/utils.worker)-fm()
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54 +0x2a
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42072dfa8)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420a1ffa8, 0x3b9aca00, 0x0, 0x1, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc42072dfa8, 0x3b9aca00, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).Run(0xc420400240, 0x3b9aca00, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54 +0x55
created by k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).Run
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:231 +0x86

Root Cause:

1. GCE GetHealthCheck V1 API used to return the health check type to "HTTP" while the healthcheck object is create beta API using "HTTP2".

2. Now HTTP2 is promoted to GA on GCE. GCE GetHealthCheck V1 API returns health check type to "HTTP2".

3. The GCE go client used by ingress-gce does not have the new HTTP2 fields in the V1 Healthcheck API.

4. In case 1, this path is hit https://github.com/kubernetes/ingress-gce/blob/release-1.2/pkg/healthchecks/healthchecks.go#L328

5. With 2, this path is hit https://github.com/kubernetes/ingress-gce/blob/release-1.2/pkg/healthchecks/healthchecks.go#L332 hc.Http2HealthCheck is nil because the V1 healthcheck object does not contain HTTP2HealthCheck fields due to 3

Impacted Versions: 1.2, 1.3, 1.4, 1.5, HEAD

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[bowei]

/lifecycle frozen

[rramkumar1]

Closing since this issue is fully mitigated.