Panic when connecting through alloydb auth proxy - panic: runtime error: slice bounds out of range [:1157627904] with capacity 16384

[andriihrachov]

Bug Description

alloydb proxy crashes with error, when deployed as a sidecar on kubernetes

2023/11/17 15:52:42 [********] accepted connection from 127.0.0.1:51470
panic: runtime error: slice bounds out of range [:1157627904] with capacity 16384

Example code (or command)

No response

Stacktrace

goroutine 9 [running]:
cloud.google.com/go/alloydbconn.(*Dialer).metadataExchange(0xc0000bf320, {0x12687f8?, 0xc000510000})
	/go/pkg/mod/cloud.google.com/go/alloydbconn@v1.5.0/dialer.go:337 +0x777
cloud.google.com/go/alloydbconn.(*Dialer).Dial(0xc0000bf320, {0x1263f08, 0xc0004cc000}, {0x7ffd5eaa8c14, 0x52}, {0x0, 0x0, 0x0?})
	/go/pkg/mod/cloud.google.com/go/alloydbconn@v1.5.0/dialer.go:247 +0xc34
github.com/GoogleCloudPlatform/alloydb-auth-proxy/internal/proxy.(*Client).serveSocketMount.func1()
	/go/src/alloydb-auth-proxy/internal/proxy/proxy.go:651 +0x2b1
created by github.com/GoogleCloudPlatform/alloydb-auth-proxy/internal/proxy.(*Client).serveSocketMount in goroutine 8
	/go/src/alloydb-auth-proxy/internal/proxy/proxy.go:631 +0x1da

Steps to reproduce?

1. 1.5.0 version of gcr.io/alloydb-connectors/alloydb-auth-proxy
2. deploy on kubernetes as a sidecar
3. run psql to connect from a neighbor conatiner / port forward from local

Environment

GKE

Additional Details

No response

[andriihrachov]

not reproduceable on 1.4.1

[enocom]

Thanks @andriihrachov. I'll take a look and report back.

[enocom]

What region is your cluster in?

[enocom]

The new Go Connector version (and Proxy version) introduces a metadata exchange check prior to the database protocol taking over. You're seeing this panic because your instance doesn't understand the metadata exchange. This is not expected behavior.

Talking with some folks on the AlloyDB team, I'm told a recent rollout may have failed for a very small number of instances. Your instance seems to be one. If you have a support contract, you can open a ticket to get a fix sooner than the next rollout (tell them to cc me and I'll help get the right people to look at this). Otherwise, I'd suggest staying on v1.4.1 and I'll report back when I see the few stragglers fixed.

[andriihrachov]

What region is your cluster in?

us-east4

Thanks a lot, I downgraded to 1.4.1

[cthumuluru]

Can you point us to the time window when you saw these errors?

[andriihrachov]

17.11.2023 ~17:00 CET

[enocom]

FWIW I tested this on a newly created instance and didn't see any issues. It's possible some existing instances will be affected.

[andriihrachov]

thanks. it was test db anyway, so I'll recreate