I recently assisted a customer in troubleshooting an issue with their Google Cloud Composer environment and found myself delving deep to uncover the problem. They had 22 DAGs, but only 11 were visible in the UI. It turns out this is a known bug that ...
Background Directed Acyclic Graphs (DAGs) in Amazon Managed Workflows for Apache Airflow (MWAA) can be triggered in several ways, depending on how much automation and integration you need. While you can manually trigger DAGs using the Airflow UI, we ...
BigQuery has a useful feature which allows the user to create external table with data on Google Sheets. It is very convenient because BigQuery users can query the data from Google Sheets directly. However, as a data engineer, you may need to build p...
Recently, a client wanted to use dbt core in Cloud Composer Airflow but they encountered Python dependencies issues. To solve this issue, I built a solution to run dbt inside of Cloud Run with Cloud Composer as orchestrator. The workflow looks like: ...
