I recently assisted a customer in troubleshooting an issue with their Google Cloud Composer environment and found myself delving deep to uncover the problem. They had 22 DAGs, but only 11 were visible in the UI. It turns out this is a known bug that isn't likely to be fixed soon, so I decided to document it here for anyone who might encounter the same issue in the future.
What's happening
Airflow is a great tool for orchestrating data pipelines. One of the many reasons that people love it is because of its great UI. In the main page, it shows all the DAGs that are orchestrated by Airflow. For this customer's case, they should have seen 22 DAGs, but for some reason, only 11 were showing up. Weird!
DAGs were running perfectly. And Google Cloud Console UI showed the DAGs perfectly. What could go wrong?
Going down the wrong path
When I first encountered this case, I thought it was one of those common DAG parsing issues in Airflow. I know how it goes—something's wrong with your DAG code, or Airflow can't parse lots of DAGs properly, so they don't show up in the UI.
So I followed Google's documentation about Troubleshooting DAG Processor issues. Started looking into all the Airflow configuration options that define timeouts for parsing DAGs:
[core]dagbag_import_timeout - defines how much time the DAG processor has to parse a single DAG
[core]dag_file_processor_timeout - defines the total amount of time the DAG processor can spend on parsing all DAGs
But one thing bothered me - the customer only had 22 DAGs. Unlike one of my customers, who had 2000+ DAGs, 22 DAGs should not hit parsing timeouts.
Turns out it wasn't a parsing issue at all!
Which versions are broken
I spun up three Composer environments:
I can confirm the bug exists in both Airflow versions 2.10.5 and 2.10.2. Probably 2.10.3, 2.10.4, and 2.11.0 too. 2.9.3 worked fine.
I only tested Cloud Composer, but I suspect the bug exists in MWAA too.
How to reproduce this
Every Cloud Composer environment comes with a default airflow_monitoring DAG to run a health check for the Airflow cluster. This is perfect - I don’t need to write any dummy DAGs.
To reproduce the bug, in 2.10.2 and 2.10.5 environments:
In my Composer, I navigated to Airflow UI. The URL was https://3ca7d775de394874b2bc4ab37359e639-dot-us-central1.composer.googleusercontent.com
It showed the default airflow_monitoring DAG, great. On Airflow UI, I could see All button showed value 1, same with Active and Failed buttons showed 0:
After clicking the Failed button, the button turned blue and both All and Active buttons incorrectly showed 0:
I noticed that the URL had changed to: https://3ca7d775de394874b2bc4ab37359e639-dot-us-central1.composer.googleusercontent.com/home?lastrun=failed
When I navigated to the Airflow UI again via https://3ca7d775de394874b2bc4ab37359e639-dot-us-central1.composer.googleusercontent.com, Chrome dev tools had two redirections:
This indicated Airflow remembered the filter somehow.
In version 2.9.3, after clicking Failed button, both All and Actived buttons showed the correct DAG number:
It is a bug, but won’t get fixed
After some digging, I found the GitHub issue for this bug. It is listed as Closed as not planned. This means it won’t be fixed as I understand Airflow maintainers must be busy working with Airflow 3.
Since Composer is just managed Airflow, Google can't really do much about it. They normally rely on the upstream project to fix it, which apparently isn't happening.
Workarounds
To anyone who encountered the same issue, there are two workarounds:
Delete the ?lastrun=failed part from the URL
Click the "Failed" button again to turn off the filter