Like always, I ran a lab to show you.

I wrote a Ruby script to publish a message to Pub/Sub:

require "google/cloud/pubsub"

pubsub = Google::Cloud::PubSub.new(
  project_id: "derrick-sandbox",
  credentials: "test-pubsub-sa.json"
)

# Retrieve a topic
topic = pubsub.topic "test-topic"

# Publish a new message
msg = topic.publish "new-message2"

puts 'Sent a message to PubSub!'

I have two test projects: airflow-talk and derrick-sandbox.
To simulate your case, I am running my script with a service account from airflow-talk, while I created a PubSub topic in derrick-sandbox and granted permissions to the PubSub topic there.

I created a PubSub topic and subscription in project derrick-sandbox:

I created a service account in project airflow-talk:

I then granted only Pub/Sub Publisher (roles/pubsub.publisher) to this service account, which in theory, is good enough to publish the message to Pub/Sub:

Now back to airflow-talk project, in Cloud Shell, I was able to use gcloud to publish a message to the topic:
(Note that I did authorize my gcloud using the service account first)

➜  gcloud auth activate-service-account test-pubsub@airflow-talk.iam.gserviceaccount.com  --key-file=test-pubsub-sa.json --project=derrick-sandbox

➜  gcloud pubsub topics publish projects/derrick-sandbox/topics/test-topic --message="Hello World"                                                
messageIds:
- '8303462186509395'

But when running the Ruby script, it got an error:

➜  ruby-pubsub ruby test-pubsub.rb
Traceback (most recent call last):
        11: from test-pubsub.rb:9:in `<main>'
        10: from /home/derrick/.gems/gems/google-cloud-pubsub-2.15.4/lib/google/cloud/pubsub/project.rb:171:in `topic'
         9: from /home/derrick/.gems/gems/google-cloud-pubsub-2.15.4/lib/google/cloud/pubsub/service.rb:109:in `get_topic'
         8: from /home/derrick/.gems/gems/google-cloud-pubsub-v1-0.18.0/lib/google/cloud/pubsub/v1/publisher/client.rb:584:in `get_topic'
         7: from /home/derrick/.gems/gems/gapic-common-0.20.0/lib/gapic/grpc/service_stub.rb:190:in `call_rpc'
         6: from /home/derrick/.gems/gems/gapic-common-0.20.0/lib/gapic/grpc/service_stub/rpc_call.rb:123:in `call'
         5: from /home/derrick/.gems/gems/grpc-1.58.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:173:in `block in request_response'
         4: from /home/derrick/.gems/gems/grpc-1.58.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
         3: from /home/derrick/.gems/gems/grpc-1.58.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:174:in `block (2 levels) in request_response'
         2: from /home/derrick/.gems/gems/grpc-1.58.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:377:in `request_response'
         1: from /home/derrick/.gems/gems/grpc-1.58.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:186:in `attach_status_results_and_complete_call'
/home/derrick/.gems/gems/grpc-1.58.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status': 7:User not authorized to perform this action.. debug_error_string:{UNKNOWN:Error received from peer ipv4:142.251.10.95:443 {grpc_message:"User not authorized to perform this action.", grpc_status:7, created_time:"2023-09-20T06:40:27.019799881+00:00"}} (GRPC::PermissionDenied)
        4: from test-pubsub.rb:9:in `<main>'
        3: from /home/derrick/.gems/gems/google-cloud-pubsub-2.15.4/lib/google/cloud/pubsub/project.rb:171:in `topic'
        2: from /home/derrick/.gems/gems/google-cloud-pubsub-2.15.4/lib/google/cloud/pubsub/service.rb:109:in `get_topic'
        1: from /home/derrick/.gems/gems/google-cloud-pubsub-v1-0.18.0/lib/google/cloud/pubsub/v1/publisher/client.rb:551:in `get_topic'
/home/derrick/.gems/gems/google-cloud-pubsub-v1-0.18.0/lib/google/cloud/pubsub/v1/publisher/client.rb:589:in `rescue in get_topic': 7:User not authorized to perform this action.. debug_error_string:{UNKNOWN:Error received from peer ipv4:142.251.10.95:443 {grpc_message:"User not authorized to perform this action.", grpc_status:7, created_time:"2023-09-20T06:40:27.019799881+00:00"}} (Google::Cloud::PermissionDeniedError)

This is very strange. After digging into the stacktrace, I found it is related to this line:

# Retrieve a topic
topic = pubsub.topic "test-topic"

Apparently, this will make an API call to check if the topic exists, so it requires a Pub/Sub Viewer (roles/pubsub.viewer) IAM role. In SDKs from other languages, this is not the case.

Back to the IAM page, I added Pub/Sub Viewer (roles/pubsub.viewer) to my service account:

The script worked!

➜  ruby-pubsub ruby test-pubsub.rb
Sent a message to PubSub!

One client had an issue using it because their query was too long and it was truncated on Query details page. They asked if there is a config to show longer queries.

As always, I ran a lab to demonstrate how to fix this issue.

Firstly, I created a Cloud SQL MySQL instance and I ran a long query:

SELECT 1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`,   1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`,  
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`,  1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`,  
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`,  1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`,  
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`,  1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`,  
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`,  1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`,  
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`,  1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`,  
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`,  1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`,  
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`,  1 AS `blablahblahblablahblah`, 
1 AS `blablahblahblablahblah`, 1 AS `blablahblahblablahblah`;

Below is what I saw in Query Insight:

I followed the help document to change to 4500 characters:

Customize query lengths

Default: 1024

Sets the query length limit to a specified value from 256 bytes to 4500 bytes. Higher query lengths are more useful for analytical queries, but they also require more memory. Changing the query length requires you to restart the instance. You can still add tags to queries that exceed the length limit.

But the issue is still there.

It turns out that MySQL has a default limit on the query size, there is a flag called max_digest_length in MySQL. I will need to set it via Cloud SQL flag.

I set the flag to 4096 and I found the query not truncated in Query Insight.

There are two required changes in your Cloud Composer Airflow to make it possible.

I'll show these changes in a lab.

I created a Google Sheets file that looks like this:

And created a BigQuery that links to this file:

I can query it without issue on BigQuery Console:

Now on the Airflow part, I created a DAG and uploaded it to Cloud Composer:

import datetime

from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.providers.google.cloud.operators.bigquery import BigQueryCheckOperator


with DAG(
    dag_id="test-bigquery",
    start_date=datetime.datetime(2023, 1, 1),
    schedule=None,
):
    DATASET = "ext"
    TABLE_1 = "gsheets"
    location = "US"

    check_count = BigQueryCheckOperator(
        task_id="check_count",
        sql=f"SELECT COUNT(*) FROM {DATASET}.{TABLE_1}",
        use_legacy_sql=False,
        location=location,
    )

On the first run, Airflow reported an issue:

Diving into the logs, I found that it is about missing Google Drive access. It makes sense because Google Sheets data is stored on Google Drive.

This error is understandable because, by default, Airflow uses google_cloud_default Connection, which doesn't have Google Drive API scope [1]. So I added it:
(Note that you'll need to add cloud-platform in the scope because you don't want to lose access to Google Cloud.)

I ran the DAG again but still getting issues:

What could be the missing puzzle?

I realized that it could be the permission for the service account to access the Google Sheets file, so I shared the Google Sheets file to the service account from Google Sheets UI:

After that, I re-ran the job and it was successful!

Summary:
To access Google Sheets data in Cloud Composer via BigQuery, you will need the following:

1. Google Drive API scope to be added to your Airflow Connection

2. File access to be added to the service account that Composer uses

In this blog post, I’ll show you how to export data from Cloud SQL Postgres (This solution works for MySQL too) database to GCS using BigQuery federated query and its own Scheduled Query feature.

Step 1: Set up the Cloud SQL connection in BigQuery following this document. To avoid impacting the performance of main instance, I used a PostgreSQL read replica.

Step 2: Create a GCS bucket.

Step 3: Use the below SQL query to create a Scheduled Query following this document. For testing purpose, I created an on-demand job so that I can trigger it.

Note that in this SQL, I used BigQuery's EXPORT DATA statement, which can export query result directly to GCS.

EXPORT DATA
  OPTIONS (
    uri = CONCAT('gs://test-sql-ext/', CURRENT_TIMESTAMP(), '--*.csv'),
    format = 'CSV',
    overwrite = true,
    header = true,
    field_delimiter = ';')
AS (
SELECT * FROM EXTERNAL_QUERY("airflow-talk.us.pg-ext", "SELECT * FROM INFORMATION_SCHEMA.TABLES;")
);

Step 4: Trigger the job and check the GCS file.

Step 5: verify the CSV file. It contains the data extracted the SQL query.

You can schedule the query to run. This part of the document talks about how you can set up your schedule. You can set up the schedule using the console or bq CLI. On the console, to specify a custom frequency, select Custom, then enter a Cron-like time specification in the Custom schedule field— for example, every mon 23:30 or every 6 hours .

That's it! We built a data pipeline that can extract data from Cloud SQL to GCS on a schedule. We didn't use Serverless export. The cost of the solution is much cheaper.

The workflow looks like:

Cloud Composer -> BashOperator to trigger a Cloud Run service -> Cloud Run container runs dbt cli

Below is how it works:

Cloud Run service:
When the Cloud Run service is triggered (via HTTP call), it will run a bash script (script.sh)

Dockerfile

FROM golang:1.18-buster as builder

WORKDIR /app

COPY go.* ./
RUN go mod download

COPY invoke.go ./

RUN go build -mod=readonly -v -o server

# Use the official dbt-bigquery image for running
# https://github.com/dbt-labs/dbt-bigquery/pkgs/container/dbt-bigquery

FROM ghcr.io/dbt-labs/dbt-bigquery:1.4.1

WORKDIR /

COPY --from=builder /app/server /app/server
COPY script.sh ./

ENTRYPOINT ["/app/server"]

Invoke.go (Source code from here)

package main

import (
        "log"
        "net/http"
        "os"
        "os/exec"
)

func main() {
        http.HandleFunc("/", scriptHandler)

        // Determine port for HTTP service.
        port := os.Getenv("PORT")
        if port == "" {
                port = "8080"
                log.Printf("Defaulting to port %s", port)
        }

        // Start HTTP server.
        log.Printf("Listening on port %s", port)
        if err := http.ListenAndServe(":"+port, nil); err != nil {
                log.Fatal(err)
        }
}

func scriptHandler(w http.ResponseWriter, r *http.Request) {
        cmd := exec.CommandContext(r.Context(), "/bin/bash", "script.sh")
        cmd.Stderr = os.Stderr
        out, err := cmd.Output()
        if err != nil {
                w.WriteHeader(500)
        }
        w.Write(out)
}

script.sh (This is just an example of running dbt)

dbt --version

I followed this document to deploy the Cloud Run shell service.
gcloud run deploy run-dbt --no-allow-unauthenticated --region=us-central1 --source=.

Terminal output:

Airflow DAG:

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator

default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
    "email": ["airflow@example.com"],
    "email_on_failure": False,
    "email_on_retry": False,
}

dag = DAG(
    "trigger_cloud_run_dag",
    description="trigger_cloud_run_dag",
    schedule_interval="0 3 * * *",
    start_date=datetime(2023, 4, 19),
    catchup=False,
    tags=["custom"],
)

trigger_cloud_run = BashOperator(
    task_id="trigger_cloud_run",
    bash_command='curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://run-dbt-7ensisdq5q-uc.a.run.app',
    do_xcom_push=True,
    dag=dag
)

trigger_cloud_run

To make it secure, the Cloud Run service requires IAM authentication. Therefore I granted my Cloud Composer worker service account roles/run.invoker role.

Airflow logs:

Cloud Run logs show the dbt cli output:

I've uploaded the code to a GitHub repo. Feel free to let me know if you need any help running it.

I found that, behind the scene, user-managed notebook instances are Google-prebuilt Compute Engine instances.
From the Vertex AI console:

From Compute Engine console, it is shown as a normal instance:

Since it is just a normal Compute Engine VM, I wonder if it can leverage the built-in INSTANCE SCHEDULES feature.

During my lab, I set up my Compute Engine schedule looks like this:

From the logs, I can find that the instance was stopped:

Note that for some reason, there was a delay in scheduling the shutdown events. From the logs, it was roughly 14-15 minutes.

Anyway, it worked! Now you can rest assure that your Vertex AI Workbench user-managed notebooks won't cost money during the nights.

In this tutorial, I'll work through a lab, in which I deleted a BigQuery Dataset and then recovered it with all the tables and views. If you are in a rush, feel free to scroll down to the recovery steps.

Set up the test BigQuery Dataset

This is my setup in BigQuery, I have two tables and a view.

Dataset:

Two tables:

View:

Delete the Dataset

From the BigQuery UI, I deleted the Dataset:

Verify if it was deleted:

How to restore a BigQuery Dataset

Time to restore it!

Find out the tables in the Dataset

Owner permission may not be enough to access some INFORMATION_SCHEMA tables. You need to explicitly add at leastbigquery/metadataViewerrole. Also, to restore row-level tables, you needbigquery/admin.

First thing first, let's find out what tables were in my Dataset. Can't remember them? Don't worry.

I understand that you may inherit this from someone or you just don't remember the names of your tables and views. That's fine. Here is a query that you can use:

SELECT
  DISTINCT TABLE_NAME
FROM
  `region-us`.INFORMATION_SCHEMA.TABLE_STORAGE_TIMELINE
WHERE
  TABLE_SCHEMA = "dataset_17102022";

After running it in my BigQuery project, I found the table names:

How does it work? The above query uses TABLE_STORAGE_TIMELINE view in BigQuery INFORMATION SCHEMA to find out the tables that contain data.

Recreate the BigQuery Dataset

In my lab, my Dataset is in the US region. You may need to change the location to the one that you use. You can find all the BigQuery locations from here.

Recreate the BigQuery Dataset using the same name.

 bq --location=US mk -d dataset_17102022

Now I have recreated the Dataset, but it is empty. Let's recreate the tables and recover the data.

Restore the tables and data

To restore the table, we will use time travel feature from BigQuery.

Run the bq command below. Note that I used -3600000, which is specified in milliseconds using a relative offset. It can also be specified as milliseconds since the Unix epoch.

bq --location=US cp airflow-talk:dataset_17102022.table-no-partition@-3600000 airflow-talk:dataset_17102022.table-no-partition

Verified that table-no-partition was recreated with data:

Same process to restore table-partition table:

bq --location=US cp airflow-talk:dataset_17102022.table-partition@-3600000 airflow-talk:dataset_17102022.table-partition

Recreate the view

There is no straightforward way to recreate the views. It is always a good idea to store the DDL in the source repository. But if you don't have it now, you can try finding the view creation logs in Cloud Logging using the name of the view. In my case, the log is here:

To recreate this view, I ran the below SQL statement:

CREATE VIEW
  dataset_17102022.test_view1 AS (
  SELECT
    ID,
    name
  FROM
    `airflow-talk.dataset_17102022.table-partition`
  WHERE
    ID = 1 )

And my view was recreated:

Summary

In this tutorial, I set up a lab to delete and restore a BigQuery Dataset.

Everybody has oops moments, deleting a BigQuery Dataset may be one of them. I hope following the above instructions helps you. Let me know if you have any questions.

Good luck!

There are a few ways you can achieve it:

• Download the files to a VM or local machine, then write a script to convert them
• Use services like Cloud Function, Cloud Run or Cloud Dataflow to read and write the files on Cloud Storage

But what if there is a true serverless way? And you only need to write SQL to convert them?

Let me walk you through it.

Prepare the source data

I wrote a Python script to generate a gzipped JSONL file.

import gzip
import json

def main():
    json_str = ''
    for i in range(10):
        json_str = json_str + json.dumps(dict(id=i, value=i * i)) + "\n"

    json_bytes = json_str.encode("utf-8")

    with gzip.open("test.jsonl.gz", "w") as fout:
        fout.write(json_bytes)

if __name__ == "__main__":
    main()

Uploaded it to my GCS bucket and read it

By following the Google document about BigQuery external table, I was able to query it:

Time to convert it to Parquet format

To achieve this, I used BigQuery EXPORT DATA statement to "export" the data from BigQuery to Cloud Storage in PARQUET format:

After navigating to the Cloud Storage console, I can see the PARQUET file:

Verify the data

Finally, I'd like to verify the data is actually in the PARQUET file:

And yay!

Summary

Using BigQuery's native EXPORT DATA statement, I was able to convert gzipped JSONL files on Google Cloud Storage to PARQUET format.

You can also convert the files to other formats. It currently supports AVRO, CSV, JSON and PARQUET. It is straightforward and serverless.

Logs

Assuming you run your business on Google Cloud. Your API is powered by Cloud Function. When there is a new user registration, the Cloud Function instance writes a log to Cloud Logging.

Create a Log Sink

Sinks control how Cloud Logging routes logs. Using sinks, you can route some or all of your logs to supported destinations.

From the Log Sink page, click CREATE SINK. When creating the sink, route the logs to BigQuery:

After creating the sink, it looks like:

A new BigQuery table was created:

Write some logs

Now I put a few logs to Cloud Logging via gcloud command:

Query the logs and navigate to Looker Studio

On Looker Studio(formerly Data Studio), I built a simple dashboard:

Note that, for testing purposes, I had to switch to minutes so we can see a few bars, but in most cases, the time interval should be hourly or daily.

Summary

Using Cloud Logging, BigQuery and Looker Studio, you can quickly build a dashboard to visualise business data, such as user registrations.

(Updated on 17th May - tested another upgrade from 9.6 to 14, also worked.)

Test run

Create source database

Create a Postgresql 13 database with two flags:

With the flags, pglogical will be enabled

When using Google DMS, the Source instance must include the postgres database. If you don't have this database, then create it. Luckily, Cloud SQL creates postgres database and its user.

Pop up some data

Connect to the database

gcloud sql connect pg-13 --user=postgres -d postgres --quiet

Run the below command

CREATE TABLE entries (guestName VARCHAR(255), content VARCHAR(255),
                        entryID SERIAL PRIMARY KEY);
INSERT INTO entries (guestName, content) values ('first guest', 'I got here!');
INSERT INTO entries (guestName, content) values ('second guest', 'Me too!');

Source databases configuration

Database Migration Service migrates all databases under the source instance other than the following databases:

• For Cloud SQL sources: template databases template0 and template1

Enable pglogical extension for all the databases

As I only have one database postgres, this step is easy.

Connect to the database

gcloud sql connect pg-13 --user=postgres -d postgres --quiet

Run then command

CREATE EXTENSION IF NOT EXISTS pglogical

Set privileges for the user

Set privileges on all schemas (aside from the information schema and schemas starting with "pg_") on each database to migrate, including pglogical.

GRANT USAGE on SCHEMA public to postgres;
GRANT USAGE on SCHEMA pglogical to postgres;
GRANT SELECT on ALL TABLES in SCHEMA pglogical to postgres;
GRANT SELECT on ALL TABLES in SCHEMA public to postgres;
GRANT SELECT on ALL SEQUENCES in SCHEMA public to postgres;
GRANT SELECT on ALL SEQUENCES in SCHEMA pglogical to postgres;
-- Don't forget to grant REPLICATION role to the postgres user that is used to run the job! 
ALTER USER postgres with REPLICATION;

Create and run DMS job

Step 1: Choose the source database. In my case, it is Cloud SQL.

Step 2: Define the source database configuration. This will create a connection profile

Step 3: Create the destination Cloud SQL instance.

Step 4: Config the network connection. In my case, as both source and destination instances are Cloud SQL, I chose private IP which will just do the VPC peering.

Step 5: Test and create the job.

Step 6: Run the job!

This is how it looks like after the job was created:

Verify the replication

Connect to the database

gcloud sql connect pg-13 --user=postgres -d postgres --quiet

Run the following statement

INSERT INTO entries (guestName, content) values ('third guest', 'test dms');

Connect to the new database

gcloud sql connect pg-14-may-16 --user=postgres -d postgres --quiet

Check if the data has been synced:

postgres=> SELECT * FROM entries;
  guestname   |   content   | entryid 
--------------+-------------+---------
 first guest  | I got here! |       1
 second guest | Me too!     |       2
 third guest  | test dms    |       3
(3 rows)

Time to promote the new instance

Click the promote button

Now the new version of Cloud SQL Postgresql instance is promoted!

Conclusion

Google DMS can be used to replicate data from a lower version of Cloud SQL Postgresql to a newer one, then used to upgrade the Postgres version. Note that if you plan to do this, ensure that your applications stop the write operations on the old version Postgres instance, wait for the replication to be finished, then start the promotion of the new instance.

Kubernetes is a new cool kid in the cloud infrastructure world. It is an open-source container orchestration system for automating software deployment, scaling, and management. Nowadays, when we manage our Cloud infrastructure, we have to look after cloud resources such as storage, databases and message queues. In addition, we also need to manage all the stuff within Kubernetes.

In Google Cloud(GCP) world, Terraform becomes the default tool to manage the infrastructure. Engineers use Terraform to deploy GKE, the managed Kubernetes service on GCP. From here, they go to different routes:

• Use Terraform to manage GCP cloud resources and Kubectl plus YAML to manage Kubernetes resources
• Use Terraform to manage both GCP cloud resources and Kubernetes resources using Kubernetes Provider

YAML lovers ask: Is there a way to use YAML to deploy GCP cloud resources?

Yes, Google recently released a service called Config Connector that can make it happen.

Introduce Config Connector

Config Connector is a Kubernetes add-on that allows you to manage GCP resources through Kubernetes. Config Connector provides a collection of Kubernetes Custom Resource Definitions (CRDs) and controllers. The Config Connector CRDs allow Kubernetes to create and manage Google Cloud resources when configuring and applying Objects to your cluster.

Enable Config Connector

To enable Config Connector when creating a GKE cluster:

• Allow Cloud APIs from Access Scopes
• Enable Workload Identity to allow workloads in your GKE clusters to impersonate Identity and Access Management (IAM) service accounts to access Google Cloud services
• Enable Config Connector

If there is a GKE already, follow this link.

Use Config Connector to manage BigQuery datasets

Create a BigQuery Dataset

Use YAML file below:

apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryDataset
metadata:
  annotations:
    cnrm.cloud.google.com/delete-contents-on-destroy: "false"
    cnrm.cloud.google.com/deletion-policy: abandon
    cnrm.cloud.google.com/project-id : airflow-talk
  name: test-bq-dataset
spec:
  resourceID: my_test_dataset
  location: US
  defaultTableExpirationMs: 86400000

Check the deployment

Run kubectl describe bigquerydataset test-bq-dataset

From UI:

Delete the Dataset

Run kubectl delete -f create-bq.yaml

To keep the Dataset, follow this guide.

In short, if cnrm.cloud.google.com/delete-contents-on-destroy: "false", then the Dataset will remain, otherwise Config Connector will delete the Dataset.

Key learning

Config Connector is a new tool to manage GCP resources in the Kubernetes way. However, weather Config Connector should be used instead of Terraform, it is another topic for another day...