Skip to content

Commit 04fef45

Browse files
committed
composer: add workaround for google-api-core Python 3.11 EOL deprecation check
Add a sitecustomize.py import hook and documentation in composer/tools/google_api_core_workaround to mitigate high DAG parsing latency, elevated worker CPU usage, and log spam caused by google-api-core's 365-day EOL deprecation check (importlib.metadata disk scan) on Python 3.11.
1 parent d831f8b commit 04fef45

3 files changed

Lines changed: 179 additions & 0 deletions

File tree

‎composer/tools/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
11
# Composer Tools
22

33
* [Composer DAGs Pausing/Unpausing script](composer_dags.md)
4+
* [google-api-core Python 3.11 Deprecation Workaround](google_api_core_workaround/README.md)
5+
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Workaround for google-api-core Python 3.11 Deprecation Warning
2+
3+
This tool provides a workaround for Cloud Composer environments running Python 3.11 where `google-api-core` triggers high DAG parsing times, elevated worker CPU usage, or deprecation log spam.
4+
5+
## When to Use This
6+
7+
Use this workaround if:
8+
- Your Cloud Composer environment is running on Python 3.11.
9+
- You see `FutureWarning` logs from `google.api_core._python_version_support` indicating that Python 3.11 will reach end of life.
10+
- You notice increased DAG parsing latency or worker CPU usage after entering the 365-day Python 3.11 EOL window (October 2026).
11+
12+
### Root Cause
13+
`google-api-core` emits a deprecation warning starting 365 days before a Python version reaches community EOL. While emitting this warning, it calls `importlib.metadata.packages_distributions()`, which scans all installed packages across `site-packages` on disk. In Cloud Composer, where DAG processors and workers frequently spawn fresh Python subprocesses, repeated filesystem metadata scans cause measurable parse latency and CPU overhead.
14+
15+
## What the Workaround Does
16+
17+
`sitecustomize.py` is an import hook that CPython runs automatically on startup before user code executes. This script:
18+
1. Changes `DEPRECATION_WARNING_PERIOD` from 365 days to 60 days, deferring the warning until August 2027.
19+
2. Stubs `_get_pypi_package_name` to return `None`, preventing `packages_distributions()` from scanning disk.
20+
3. Cleans up its own folder from `sys.path` immediately after load so subsequent imports do not re-scan the patch directory.
21+
22+
Because the file contains no DAG objects and avoids DAG-related keywords, Airflow's default `dag_discovery_safe_mode = True` ignores it during DAG parsing.
23+
24+
## How to Apply
25+
26+
### Step 1: Upload `sitecustomize.py` to GCS
27+
28+
Copy `sitecustomize.py` to a `patches/` directory in your environment's DAGs bucket:
29+
30+
```bash
31+
gcloud storage cp sitecustomize.py gs://<YOUR_DAGS_BUCKET>/dags/patches/sitecustomize.py
32+
```
33+
34+
You can find your DAGs bucket name by running:
35+
```bash
36+
gcloud composer environments describe <ENVIRONMENT_NAME> \
37+
--location <LOCATION> \
38+
--format="value(config.dagGcsPrefix)"
39+
```
40+
41+
### Step 2: Set the `PYTHONPATH` Environment Variable
42+
43+
Add the patch folder to `PYTHONPATH` on your Composer environment:
44+
45+
```bash
46+
gcloud composer environments update <ENVIRONMENT_NAME> \
47+
--location <LOCATION> \
48+
--update-env-variables PYTHONPATH=/home/airflow/gcs/dags/patches
49+
```
50+
51+
> **Important:** If your environment already defines a custom `PYTHONPATH`, preserve your existing paths by separating them with a colon (`:`):
52+
> ```bash
53+
> --update-env-variables PYTHONPATH=/home/airflow/gcs/dags/patches:<EXISTING_PATH>
54+
> ```
55+
56+
This update triggers a rolling restart of Composer workloads. Once restarted, all Python subprocesses (scheduler, workers, DAG processor) will load `sitecustomize.py` on startup.
57+
58+
### Note on `dag_discovery_safe_mode`
59+
60+
If you explicitly disabled safe mode (`[core] dag_discovery_safe_mode = False`), add `patches/*` to your `dags/.airflowignore` file to prevent the DAG file processor from attempting to parse the patch directory:
61+
62+
```text
63+
patches/*
64+
```
65+
66+
## Verification
67+
68+
After the environment finishes updating:
69+
1. Check Cloud Logging or task logs in the Airflow UI. The `_python_version_support` `FutureWarning` messages should stop appearing.
70+
2. In the Cloud Monitoring dashboard for Composer, check the **DAG Processing: Total Parse Time** metric. Parse times should return to baseline levels.
71+
72+
## How to Remove
73+
74+
Once your environment is upgraded to a Composer image or `google-api-core` version with the upstream fix, remove the workaround:
75+
76+
1. Remove `PYTHONPATH` (or restore your original value):
77+
```bash
78+
gcloud composer environments update <ENVIRONMENT_NAME> \
79+
--location <LOCATION> \
80+
--remove-env-variables PYTHONPATH
81+
```
82+
83+
2. Delete the patch file from the bucket:
84+
```bash
85+
gcloud storage rm gs://<YOUR_DAGS_BUCKET>/dags/patches/sitecustomize.py
86+
```
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Copyright 2026 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""CPython sitecustomize hook for patching google.api_core deprecation check."""
16+
17+
import datetime
18+
import importlib.util
19+
import os
20+
import sys
21+
22+
# Remove this patch directory from sys.path to avoid redundant lookups
23+
# on subsequent module imports when loaded via PYTHONPATH.
24+
_patch_file = globals().get("__file__")
25+
if _patch_file:
26+
_patch_dir = os.path.abspath(os.path.dirname(_patch_file))
27+
_is_site_dir = any(
28+
part in ("site-packages", "dist-packages")
29+
for part in _patch_dir.split(os.sep)
30+
)
31+
if not _is_site_dir:
32+
_patch_dir_slash = _patch_dir + os.sep
33+
sys.path[:] = [
34+
p
35+
for p in sys.path
36+
if not isinstance(p, (str, os.PathLike))
37+
or not os.path.isabs(p)
38+
or (
39+
os.path.abspath(p) != _patch_dir
40+
and os.path.abspath(p) != _patch_dir_slash
41+
)
42+
]
43+
sys.path_importer_cache.pop(_patch_dir, None)
44+
sys.path_importer_cache.pop(_patch_dir_slash, None)
45+
46+
TARGET_MODULE = "google.api_core._python_version_support"
47+
48+
49+
class _GoogleApiCorePatcher:
50+
"""Import hook to patch DEPRECATION_WARNING_PERIOD on load."""
51+
52+
def find_spec(self, fullname, path=None, target=None):
53+
if fullname == TARGET_MODULE:
54+
try:
55+
sys.meta_path.remove(self)
56+
except ValueError:
57+
pass
58+
59+
try:
60+
spec = importlib.util.find_spec(fullname)
61+
except Exception:
62+
return None
63+
64+
if spec and spec.loader:
65+
orig_exec_module = spec.loader.exec_module
66+
67+
def exec_module(module):
68+
import importlib.metadata
69+
70+
orig_packages_distributions = getattr(
71+
importlib.metadata, "packages_distributions", None
72+
)
73+
if orig_packages_distributions:
74+
importlib.metadata.packages_distributions = lambda: {}
75+
try:
76+
orig_exec_module(module)
77+
finally:
78+
if orig_packages_distributions:
79+
importlib.metadata.packages_distributions = (
80+
orig_packages_distributions
81+
)
82+
module.DEPRECATION_WARNING_PERIOD = datetime.timedelta(days=60)
83+
module._get_pypi_package_name = lambda module_name: None
84+
85+
spec.loader.exec_module = exec_module
86+
return spec
87+
return None
88+
89+
90+
if not any(type(h).__name__ == "_GoogleApiCorePatcher" for h in sys.meta_path):
91+
sys.meta_path.insert(0, _GoogleApiCorePatcher())

0 commit comments

Comments
 (0)