Deleting StorageBucketAccessControl or StorageDefaultObjectAccessControl fails if StorageBucket is deleted first

[jcanseco]

Checklist

• I did not find a related open issue.
• I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

Attempting to delete a StorageBucketAccessControl (or StorageDefaultObjectAccessControl) resource after its referenced StorageBucket has already been deleted fails with the following error:

Delete call failed: error fetching live state: error getting ID for resource: error getting value from reference: could not parse reference resolution value '<nil>' as string

It seems that the StorageBucket needs to be deleted first. Otherwise, the only way to unstuck the delete is to do a forced cleanup.

Additional Diagnostic Information

Kubernetes Cluster Version

Client Version: v1.21.0
Server Version: v1.18.16-gke.2100

Config Connector Version

1.49.1

Config Connector Mode

cluster

Log Output

N/A

Steps to Reproduce

Steps to reproduce the issue

1. Apply both the StorageBucket and StorageBucketAccessControl below.
2. Wait for both to be UpToDate.
3. Delete the StorageBucket.
4. Delete the StoageBucketAccessControl.
5. Observe that StorageBucketAccessControl fails with DeleteFailed.

The same is true if you replace StorageBucketAccessControl with a StorageDefaultObjectAccessControl resource instead.

YAML snippets

1. StorageBucket:

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: my-project
  name: my-bucket

2. StorageBucketAccessControl:

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucketAccessControl
metadata:
  name: my-bucket-access-control
spec:
  bucketRef:
    name: my-bucket
  entity: allAuthenticatedUsers
  role: READER

[jcanseco]

We have been able to reproduce the issue and should have a fix out shortly (within the next couple weeks).

[InterestedInTechAndCake]

Thank you for fixing this!

Please note that the same behaviour happened to this too:

bigquerytable.bigquery.cnrm.cloud.google.com

[jcanseco]

Thanks @InterestedInTechAndCake! We'll fix that one as well.

[jcanseco]

@InterestedInTechAndCake, it turns out that we'll need to give the BigQueryTable one a bit more thought before being able to resolve it.

Is this issue blocking you by any chance, or would you consider it more of a friction point?

[InterestedInTechAndCake]

Hi @jcanseco , it is not blocking us so it's fine if you need more time to work on a more proper fix, we have worked around the issue by manually removing the finalizers on the k8s objects that were stuck due to this error. We do need the fix though as we often destroy our clusters and recreate, and likely to hit this again. When you have the information please share with us the timeline and progress on this.

Many thanks!

[jcanseco]

we often destroy our clusters and recreate

@InterestedInTechAndCake gotcha. Am I correct in understanding that the issue here is that you're also trying to delete your BigQueryDataset and BigQueryTable resources when destroying your clusters?

If you need to also destroy your BigQueryDataset resources anyway, the following workaround will likely work for you:

1. Annotate your BigQueryDataset with cnrm.cloud.google.com/delete-contents-on-destroy: true. This will allow the dataset to be deleted on GCP even if it still has tables in it. Note that when the dataset is deleted, its tables are also deleted.
2. Annotate your BigQueryTable with cnrm.cloud.google.com/deletion-policy: abandon. This will make KCC abandon the resource instead of trying to delete it on GCP when the resource is deleted on KCC.

The above combination will allow you to workaround the issue and delete your datasets and tables from both KCC and GCP.

Can you let us know if this workaround works for you?

[InterestedInTechAndCake]

Thank you @jcanseco, what you suggested should be okay as a workaround for us, will bear that in mind if we hit this again next time.

Are you going to fix up StorageBucketAccessControl and StorageDefaultObjectAccessControl first, and spend more time on the BigQuery one? Or are you going to fix both at the same time after you have figured out the solution for BigQuery?

[jcanseco]

Great, thanks for confirming @InterestedInTechAndCake!

We'll release a fix for StorageBucketAccessControl and StorageDefaultObjectAccessControl, then we'll create a separate issue to track the BigQueryTable one.

[maqiuyujoyce]

The issue for Storage resources should be fixed in Config Connector v1.50.0.

[Jonpez2]

Is there an issue we can track for the BigQueryTable deletion problem please?

Thanks!

[xiaobaitusi]

Hi @Jonpez2, thanks for following up on this.

I filed a separate issue to track the BigQueryTable deletion problem. There is not much progress made to resolve the generic deletion ordering issue. But this has been recognized as a common friction to address. We are looking into this!

Closing this particular issue around StorageBucket and StorageBucketAccessControl.

[InterestedInTechAndCake]

Thank you @xiaobaitusi for the update, could you provide the link to the separate issue raised for the BigQueryTable deletion problem please?

[xiaobaitusi]

Sure thing. Those two issues are already cross-linked since I mentioned this particular issue #463 in the separate issue.

For better vis, here is the separate issue #534 about the BigQueryTable deletion problem.