[ahidalgob]

Description

Add a sitecustomize.py import hook and documentation in composer/tools/google_api_core_workaround to mitigate high DAG parsing latency, elevated worker CPU usage, and log spam caused by google-api-core's 365-day EOL deprecation check (importlib.metadata disk scan) on Python 3.11.

Checklist

Testing

• I have tested this change on a live environment and verified it works as intended.

Compliance & Style

• I have followed the Sample Guidelines.
• The README is updated with all relevant information (if applicable).
• If this PR adds a new sample directory: I updated the CODEOWNERS file.

---

Post-Approval Actions

• Please merge this PR for me once it is approved

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request introduces a workaround for the google-api-core Python 3.11 deprecation warning in Cloud Composer environments, which can cause high DAG parsing latency and elevated CPU usage. It adds a sitecustomize.py import hook to patch the deprecation warning period and stub out disk-scanning package lookups, along with a comprehensive README guide. The review feedback suggests making the import hook more robust by handling potential exceptions in find_spec, temporarily patching importlib.metadata.packages_distributions during module execution to guarantee the disk scan is bypassed, and defensively filtering sys.path against non-string elements and relative paths.

[gemini-code-assist]

To make the import hook extremely robust, we should handle potential exceptions from importlib.util.find_spec (e.g., if parent packages are missing or there are unexpected import errors) by returning None to let the default import system handle it. Additionally, if the deprecation check in google.api_core._python_version_support is executed at module load time (during orig_exec_module), modifying module.DEPRECATION_WARNING_PERIOD afterwards is too late to prevent the disk scan and log spam. Temporarily patching importlib.metadata.packages_distributions during the execution of the module ensures the disk scan is bypassed in all scenarios.

    def find_spec(self, fullname, path=None, target=None):
        if fullname == TARGET_MODULE:
            try:
                sys.meta_path.remove(self)
            except ValueError:
                pass

            try:
                spec = importlib.util.find_spec(fullname)
            except Exception:
                return None

            if spec and spec.loader:
                orig_exec_module = spec.loader.exec_module

                def exec_module(module):
                    import importlib.metadata
                    orig_packages_distributions = getattr(importlib.metadata, "packages_distributions", None)
                    if orig_packages_distributions:
                        importlib.metadata.packages_distributions = lambda: {}
                    try:
                        orig_exec_module(module)
                    finally:
                        if orig_packages_distributions:
                            importlib.metadata.packages_distributions = orig_packages_distributions
                    module.DEPRECATION_WARNING_PERIOD = datetime.timedelta(days=60)
                    module._get_pypi_package_name = lambda module_name: None

                spec.loader.exec_module = exec_module
            return spec
        return None

[ahidalgob]

addressed.

[gemini-code-assist]

When modifying sys.path to remove the patch directory, we should be defensive against non-string elements (which can occasionally be present in sys.path depending on the environment/test runners) and avoid accidentally removing relative paths (like "" or ".") if the current working directory happens to match the patch directory. Checking os.path.isabs(p) ensures we only remove absolute paths.

Suggested change

	sys.path[:] = [
	p
	for p in sys.path
	if os.path.abspath(p) != _patch_dir
	and os.path.abspath(p) != _patch_dir_slash
	]
	sys.path[:] = [
	p
	for p in sys.path
	if not isinstance(p, (str, os.PathLike))
	or not os.path.isabs(p)
	or (os.path.abspath(p) != _patch_dir and os.path.abspath(p) != _patch_dir_slash)
	]

[ahidalgob]

addressed.