diff --git a/people-and-planet-ai/README.md b/people-and-planet-ai/README.md
index f9a477686b7..c2d280f853e 100644
--- a/people-and-planet-ai/README.md
+++ b/people-and-planet-ai/README.md
@@ -2,23 +2,25 @@
## ๐ฆ [Wildlife Insights -- _image-classification_](image-classification)
-> [](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/image-classification/README.ipynb)
+> [](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/feat/modernize-ppai-image-classification/people-and-planet-ai/image-classification/README.ipynb)
>
-> [Watch the video in YouTube ](https://youtu.be/hUzODH3uGg0)
+> [Watch the video on YouTube ](https://youtu.be/hUzODH3uGg0)
-This model is trained to recognize animal species from
+Classify wildlife species and extract ecological recovery metrics from
[camera trap](https://en.wikipedia.org/wiki/Camera_trap)
-pictures.
+pictures in the public [LILA Science](https://lila.science/datasets/wcscameratraps) Cloud Storage bucket (`gs://public-datasets-lila/`).
-* **Model**: [AutoML] Vision
-* **Creating datasets**: [Apache Beam] in [Dataflow]
-* **Training the model**: [AutoML] in [Vertex AI]
-* **Getting predictions**: [Vertex AI]
+* **Multimodal classification and ecological extraction**: [Gemini] (`gemini-3.8-flash`) via the [Google Gen AI SDK]
+* **Rare-species visual similarity and vector search**: [Gemini Embedding 2] (`gemini-embedding-2`)
+* **Class-balanced batch inference pipeline**: [Apache Beam] `RunInference` in [Dataflow]
+* **Metadata and structured observation storage**: [BigQuery]
[Apache Beam]: https://beam.apache.org
+[BigQuery]: https://cloud.google.com/bigquery
[Dataflow]: https://cloud.google.com/dataflow
-[AutoML]: https://cloud.google.com/vertex-ai/docs/beginner/beginners-guide
-[Vertex AI]: https://cloud.google.com/vertex-ai
+[Gemini]: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/3-8-flash
+[Gemini Embedding 2]: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/embedding-2
+[Google Gen AI SDK]: https://googleapis.github.io/python-genai/
---
diff --git a/people-and-planet-ai/image-classification/README.ipynb b/people-and-planet-ai/image-classification/README.ipynb
index cd141cd8e78..54c3c3958cc 100644
--- a/people-and-planet-ai/image-classification/README.ipynb
+++ b/people-and-planet-ai/image-classification/README.ipynb
@@ -50,19 +50,38 @@
"source": [
"# Monitor wildlife health via image classification\n",
"\n",
- "๐ฆ๐ฆง ๐๐ฆฉ๐ฆ ๐ฆ ๐ฆ ๐ฆ ๐ ๐ฆซ ๐ฆ ๐ ๐ ๐ฆ ๐ฆฅ ๐ฆ ๐ฆ\n",
- "\n",
- "- **Time estimate**: 2 hours 15 minutes\n",
- "- **Cost estimate**: less than \\$30.00 (_free_ if you use \\$300 [Cloud credits](https://cloud.google.com/free/docs/gcp-free-tier))\n",
- "\n",
- "> [Watch the video in YouTube ](https://youtu.be/hUzODH3uGg0)\n",
- ">\n",
- "> ๐ Read more about this sample in the [blog post](https://cloud.google.com/blog/topics/developers-practitioners/recovering-global-wildlife-populations-using-ml).\n",
- "\n",
- "This is an interactive notebook that contains all of the code necessary to train an ML model for image classification.\n",
- "This model is trained to recognize animal species from\n",
- "[camera trap](https://en.wikipedia.org/wiki/Camera_trap)\n",
- "pictures."
+ "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "\n",
+ "\n",
+ "Using Colab Enterprise? (Click if the Network dropdown is blank when importing)\n",
+ "\n",
+ "Colab Enterprise runs inside your project's VPC network. In organizations that enforce `constraints/compute.skipDefaultNetworkCreation`, new projects do not have a `default` network, so the **Network** dropdown is blank and the notebook opens in read-only preview mode until a network exists. To create a `default` network and `us-central1` subnet in Cloud Shell:\n",
+ "\n",
+ "```bash\n",
+ "gcloud compute networks create default --project=example-project-id --subnet-mode=custom\n",
+ "gcloud compute networks subnets create default --project=example-project-id --network=default --region=us-central1 --range=10.128.0.0/20 --enable-private-ip-google-access\n",
+ "```\n",
+ "\n",
+ ""
],
"metadata": {
"id": "1LUt29D0MSeu"
@@ -71,16 +90,32 @@
{
"cell_type": "markdown",
"source": [
- "## ๐ Using this interactive notebook\n",
+ "## Overview\n",
+ "\n",
+ "๐ฆ๐ฆง ๐๐ฆฉ๐ฆ ๐ฆ ๐ฆ ๐ฆ ๐ ๐ฆซ ๐ฆ ๐ ๐ ๐ฆ ๐ฆฅ ๐ฆ ๐ฆ\n",
"\n",
- "Click the **run** icons โถ๏ธ of each section within this notebook. \n",
+ "Classify animal species and extract ecological recovery signals from [camera trap](https://en.wikipedia.org/wiki/Camera_trap) pictures using **Gemini 3.8 Flash** (`gemini-3.8-flash`), **Gemini Embedding 2** (`gemini-embedding-2`), and **Apache Beam `RunInference`** on **Dataflow**.\n",
"\n",
- "This notebook code lets you train and deploy an ML model, as well as test its accuracy, from end-to-end. When you run a code cell, the code runs in the notebook's runtime, so you're not making any changes to your personal computer.\n",
+ "- **Time estimate**: ~15โ20 minutes for project setup and interactive exploration, plus ~1.5โ3.5 hours if you run the full 639-species Dataflow batch inference pipeline in the background\n",
+ "- **Cost estimate**: less than \\$1.00 (with zero always-on endpoint charges; see the [Google Cloud Pricing Calculator](https://cloud.google.com/products/calculator))\n",
"\n",
- "> ๐๏ธ **To avoid any errors**, wait for each section to finish in their order before clicking the next โrunโ icon.\n",
+ "> [Watch the video on YouTube ](https://youtu.be/hUzODH3uGg0)\n",
+ ">\n",
+ "> ๐ For more information about the background of this solution, see [Recovering global wildlife populations using ML](https://cloud.google.com/blog/topics/developers-practitioners/recovering-global-wildlife-populations-using-ml).\n",
"\n",
- "This sample must be connected to a Google Cloud project, but nothing else is needed other than your Google Cloud project.\n",
- "You can use an existing project and the cost will be less than \\$30. Alternatively, you can create a new Cloud project with cloud credits for free"
+ "This notebook uses public camera trap datasets hosted in Cloud Storage (`gs://public-datasets-lila/`) by the [Labeled Information Library of Alexandria: Biology and Conservation (LILA)](https://lila.science/datasets/wcscameratraps), including the **Wildlife Conservation Society (WCS) Camera Traps** dataset and the **Felidae Conservation Fund** San Francisco Bay Area dataset.\n",
+ "\n",
+ "In this notebook, you complete the following steps:\n",
+ "\n",
+ "1. **Create a table of image metadata** _(~6โ8 minutes to complete, costs a few cents)_:\n",
+ " Run [`create_images_metadata_table.py`](create_images_metadata_table.py) on [Dataflow](https://cloud.google.com/dataflow) to ingest and clean the WCS Camera Traps metadata JSON archive directly from `gs://public-datasets-lila/wcs/wcs_camera_traps.json.zip` into a [BigQuery](https://cloud.google.com/bigquery) table.\n",
+ "2. **Run class-balanced batch inference in Dataflow** _(~1.5โ3.5 hours for all 639 species; the interactive steps below can be run immediately while the pipeline runs)_:\n",
+ " Run [`run_wildlife_inference.py`](run_wildlife_inference.py) using Apache Beam's `Sample.FixedSizePerKey` and `RunInference` with **Gemini 3.8 Flash** (`gemini-3.8-flash`) to classify images in-place from `gs://public-datasets-lila/wcs-unzipped/` and write structured ecological observations (`species_scientific_name`, `taxon_family`, `common_name`, `animal_count`, `juvenile_present`, `is_empty_or_misfire`, `confidence`) to BigQuery.\n",
+ "3. **Classify and geo-fence camera trap images interactively**:\n",
+ " Use [`classify_wildlife_image.py`](classify_wildlife_image.py) to inspect camera trap photos, see how taxonomic family (`taxon_family`) resolves look-alike species, and pass `habitat_region` context to geo-fence predictions.\n",
+ "4. **Generate multimodal embeddings for rare-species search**:\n",
+ " Use [`embed_wildlife_image.py`](embed_wildlife_image.py) and **Gemini Embedding 2** (`gemini-embedding-2`) to turn a single reference photo of an endangered species into a 768-dimensional vector embedding for similarity search in BigQuery.\n",
+ "5. **Clean up resources** to avoid ongoing storage costs."
],
"metadata": {
"id": "jXDRG8QK2BZD"
@@ -89,36 +124,7 @@
{
"cell_type": "markdown",
"source": [
- "## ๐ดโโ๏ธ Steps included\n",
- "\n",
- "The notebook leverages camera trap images dataset from the\n",
- "[WCS Camera Traps dataset](http://lila.science/datasets/wcscameratraps)\n",
- "from the _Labeled Information Library of Alexandria: Biology and Conservation_ (LILA) organization.\n",
- " \n",
- "Here's a quick summary of what youโll go through:\n",
- "\n",
- "1. **Create a database of image metadata** _(~5 minutes to complete, costs a few cents)_:\n",
- " in a\n",
- " [BigQuery](https://cloud.google.com/bigquery)\n",
- " table with all the image file names along with their respective category. \n",
- "\n",
- "1. **Train an image classifier** _(~2 hours to complete, costs ~$25.00)_:\n",
- " With some desired criteria, a\n",
- " [Dataflow](https://cloud.google.com/dataflow)\n",
- " pipeline creates a balanced dataset from the metadata database.\n",
- " It then physically downloads the necessary images from LILA into\n",
- " [Cloud Storage](https://cloud.google.com/storage),\n",
- " imports the data into\n",
- " [AI Platform Unified](https://cloud.google.com/ai-platform-unified/docs/start/introduction-unified-platform),\n",
- " and starts the model training job.\n",
- "\n",
- "1. **Deploy an ML model** _(costs **$1.25 for every hour** the model is deployed onlineโฆ so complete the โdelete modelโ step soon)_:\n",
- " After the model finishes training, we look at the results and deploy it into a Cloud endpoint in AI Platform Unified.\n",
- "\n",
- "1. **Classify images with the model**:\n",
- " We send some images to the model and get back the model's classification predictions.\n",
- "\n",
- "1. (Optional) **Delete the project** to avoid ongoing costs."
+ "## Get started"
],
"metadata": {
"id": "Kb9doxCpUcpF"
@@ -127,204 +133,221 @@
{
"cell_type": "markdown",
"source": [
- "## โจ Before you begin, you need toโฆ\n",
- "\n",
- "1. Decide on creating a new\n",
- " [free project](https://cloud.google.com/free/docs/gcp-free-tier)\n",
- " _(recommended)_ or using an existing one.\n",
- " Then copy the project ID and paste it in the `google_cloud_project` field in the \"Entering project detailsโ section below.\n",
- "\n",
- " > ๐ก If you _don't plan to keep the resources_ that you create via this sample, we recommend creating a new project instead of selecting an existing project.\n",
- " > After you finish these steps, you can delete the project, removing all the resources associated in bulk.\n",
- "\n",
- "1. [_Click here_](https://console.cloud.google.com/flows/enableapi?apiid=dataflow,aiplatform.googleapis.com)\n",
- " to **enable the following APIs** in your Google Cloud project:\n",
- " _Dataflow_ and _AI Platform_.\n",
- "\n",
- "1. Make sure that **billing is enabled** for your Google Cloud project,\n",
- " [_click here_](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
- " to learn how to confirm that billing is enabled.\n",
- "\n",
- "1. [_Click here_](https://console.cloud.google.com/storage/create-bucket)\n",
- " to create a Cloud Storage bucket.\n",
- " Then copy the bucketโs name and paste it in the `cloud_storage_bucket` field in the โEntering project detailsโ section below.\n",
- "\n",
- " > ๐๏ธ Make sure it's a _regional_ bucket in a location where\n",
- " > [AI Platform is available](https://cloud.google.com/ai-platform-unified/docs/general/locations#available_regions).\n",
- "\n",
- "1. [Create a BigQuery dataset](https://cloud.google.com/bigquery/docs/datasets#create-dataset).\n",
- " Then copy the dataset ID and paste it in the `bigquery_dataset` field in the โEntering project detailsโ section below.\n",
+ "### Install required packages\n",
"\n",
- " > ๐ก You can use the name `samples` for your dataset."
+ "Download and install the required libraries and sample scripts for this notebook _(~1 minute)_."
],
"metadata": {
"id": "VRZjMwyf2BZE"
}
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
+ "execution_count": null,
"source": [
- "### โ๏ธ Preparing the project environment\n",
- "\n",
- "Click run โถ๏ธ for the following cells to download and install the necessary libraries and resources for this solution.\n",
- "\n",
- "> ๐ก You can _optionally_ view the entire\n",
- "> [code in GitHub](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/people-and-planet-ai/image-classification)."
+ "# Takes ~1 minute to fetch the sample scripts and install required packages.\n",
+ "import importlib\n",
+ "import os\n",
+ "import sys\n",
+ "\n",
+ "if \"google.colab\" in sys.modules:\n",
+ " # Install libffi-dev for the Dataflow pipeline in ephemeral Colab runtimes.\n",
+ " !apt-get -qq install libffi-dev\n",
+ "\n",
+ " # Sparse-checkout only people-and-planet-ai/image-classification into an isolated directory.\n",
+ " REPO_DIR = \"ppai-image-classification\"\n",
+ " SAMPLE_SUBPATH = \"people-and-planet-ai/image-classification\"\n",
+ " BRANCH = \"main\"\n",
+ "\n",
+ " if not os.getcwd().endswith(SAMPLE_SUBPATH):\n",
+ " if not os.path.exists(REPO_DIR):\n",
+ " !git clone --filter=blob:none --no-checkout --depth 1 --sparse -b {BRANCH} https://github.com/GoogleCloudPlatform/python-docs-samples.git {REPO_DIR}\n",
+ " !git -C {REPO_DIR} sparse-checkout set {SAMPLE_SUBPATH}\n",
+ " %cd {REPO_DIR}\n",
+ " !git fetch --depth 1 origin {BRANCH}\n",
+ " !git -c advice.detachedHead=false checkout -f FETCH_HEAD\n",
+ " %cd {SAMPLE_SUBPATH}\n",
+ "\n",
+ "if os.getcwd() not in sys.path:\n",
+ " sys.path.insert(0, os.getcwd())\n",
+ "\n",
+ "# Install the sample requirements in-place (no kernel restart required).\n",
+ "%pip install --upgrade --quiet pip\n",
+ "%pip install --upgrade --quiet -r requirements.txt\n",
+ "importlib.invalidate_caches()"
],
+ "outputs": [],
"metadata": {
"id": "5S5_eLKE-M2o"
}
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "### Import libraries"
+ ],
+ "metadata": {
+ "id": "LevEkQNG94Lx"
+ }
+ },
{
"cell_type": "code",
"execution_count": null,
"source": [
- "# We need libffi-dev to launch the Dataflow pipeline.\n",
- "!apt-get -qq install libffi-dev\n",
- "\n",
- "# Clone the python-docs-samples respository.\n",
- "!git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git\n",
+ "import io\n",
+ "import os\n",
+ "import sys\n",
"\n",
- "# Navigate to the sample code directory.\n",
- "%cd python-docs-samples/people-and-planet-ai/image-classification\n",
+ "from google import genai\n",
+ "from IPython.display import display\n",
+ "from PIL import Image\n",
+ "import requests\n",
"\n",
- "# Install the sample requirements.\n",
- "!pip install --quiet -U pip\n",
- "!pip install -r requirements.txt"
+ "import classify_wildlife_image\n",
+ "import embed_wildlife_image"
],
"outputs": [],
"metadata": {
- "id": "zdiQJibg9iEn"
+ "id": "cipdLUp92BZF"
}
},
{
"cell_type": "markdown",
"source": [
- "> ๐๏ธ **\\[DONโT PANIC\\]** Itโs safe to _ignore the warnings_.\n",
- "> When we `pip install` the requirements, there might be some warnings about conflicting dependency versions.\n",
- "> For the scope of this sample, thatโs ok.\n",
+ "### Authenticate your notebook environment (Colab only)\n",
"\n",
- "> โ ๏ธ **Restart the runtime**: Running the previous cell just updated some libraries and requires to restart the runtime to load those libraries correctly.\n",
- ">\n",
- "> In the top-left menu, click _**\"Runtime\" > \"Restart runtime\"**_."
+ "Authenticate your environment on Google Colab."
],
"metadata": {
- "id": "j_Us7ESa90WN"
+ "id": "lWfoIueZMO9P"
}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
- "# After restarting the runtime, navigate back to the sample code directory.\n",
- "%cd python-docs-samples/people-and-planet-ai/image-classification"
+ "if \"google.colab\" in sys.modules:\n",
+ " from google.colab import auth\n",
+ "\n",
+ " auth.authenticate_user()"
],
"outputs": [],
"metadata": {
- "id": "LevEkQNG94Lx"
+ "id": "sWFY2a_P2BZF"
}
},
{
"cell_type": "markdown",
"source": [
- "### โ๏ธ Entering project details\n",
+ "### Set Google Cloud project information\n",
"\n",
- "Fill these in with the Google Cloud project resources you just created."
+ "To get started using Gemini Enterprise Agent Platform, Dataflow, and BigQuery, you must have an existing Google Cloud project with billing enabled. Enter your `PROJECT_ID` (and optional `LOCATION`) below:"
],
"metadata": {
- "id": "cipdLUp92BZF"
+ "id": "arH5wwVq2BZG"
}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
- "google_cloud_project = \"\" # @param {type:\"string\"}\n",
- "cloud_storage_bucket = \"\" # @param {type:\"string\"}\n",
- "bigquery_dataset = \"samples\" # @param {type:\"string\"}\n",
- "\n",
- "# Validate inputs.\n",
- "if not google_cloud_project:\n",
- " raise ValueError(\"Please provide your google_cloud_project\")\n",
- "if not cloud_storage_bucket:\n",
- " raise ValueError(\"Please provide your cloud_storage_bucket\")\n",
- "\n",
- "# Authenticate to use the Google Cloud resources.\n",
- "try:\n",
- " from google.colab import auth\n",
+ "PROJECT_ID = \"example-project-id\" # @param {type: \"string\", placeholder: \"example-project-id\", isTemplate: true}\n",
+ "if not PROJECT_ID or PROJECT_ID == \"example-project-id\":\n",
+ " PROJECT_ID = os.environ.get(\"GOOGLE_CLOUD_PROJECT\", \"example-project-id\")\n",
"\n",
- " auth.authenticate_user()\n",
- " print(\"Authenticated\")\n",
- "except ModuleNotFoundError:\n",
- " import os\n",
+ "LOCATION = \"us-central1\" # @param {type: \"string\"}\n",
+ "if not LOCATION:\n",
+ " LOCATION = os.environ.get(\"GOOGLE_CLOUD_LOCATION\", \"us-central1\")\n",
"\n",
- " if os.environ.get(\"GOOGLE_APPLICATION_CREDENTIALS\") is None:\n",
- " raise ValueError(\n",
- " \"Please set your GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account JSON file path.\"\n",
- " )\n",
- " print(f\"GOOGLE_APPLICATION_CREDENTIALS: {service_account_file}\")\n",
- "\n",
- "%env GOOGLE_CLOUD_PROJECT={google_cloud_project}"
+ "%env GOOGLE_CLOUD_PROJECT={PROJECT_ID}"
],
"outputs": [],
"metadata": {
- "id": "lWfoIueZMO9P",
- "cellView": "form"
+ "cellView": "form",
+ "id": "wc5yFA9W2BZG"
}
},
{
"cell_type": "markdown",
"source": [
- "1. Click the run button โถ๏ธ for the cells above.\n",
- "1. Paste the \"verification code\" that is presented, and press **\\[ENTER\\]** to authenticate.\n",
+ "### Create a Cloud Storage bucket and BigQuery dataset\n",
"\n",
- "Run the following cell, you can leave them with these values or change them if you prefer."
+ "Run the following cell _(~30 seconds)_ to automatically derive your staging bucket (`{PROJECT_ID}-wildlife-insights`) and BigQuery dataset (`samples`), enable the required APIs, and create the bucket and dataset if they don't already exist:"
],
"metadata": {
- "id": "sWFY2a_P2BZF"
+ "id": "5f1a08ef"
}
},
{
"cell_type": "code",
- "execution_count": null,
"source": [
- "cloud_storage_directory = \"samples/wildlife-insights\" # @param {type:\"string\"}\n",
- "bigquery_table = \"wildlife_images_metadata\" # @param {type:\"string\"}\n",
- "ai_platform_name_prefix = \"wildlife_classifier\" # @param {type:\"string\"}\n",
- "region = \"us-central1\" # @param {type:\"string\"}"
+ "# Takes ~15-30 seconds to enable APIs and create the staging bucket and dataset.\n",
+ "BUCKET_NAME = f\"{PROJECT_ID}-wildlife-insights\"\n",
+ "CLOUD_STORAGE_DIRECTORY = \"samples/wildlife-insights\"\n",
+ "BIGQUERY_DATASET = \"samples\"\n",
+ "BIGQUERY_TABLE = \"wildlife_images_metadata\"\n",
+ "PREDICTIONS_TABLE = \"wildlife_predictions\"\n",
+ "\n",
+ "# Enable the required Google Cloud APIs.\n",
+ "!gcloud services enable \\\n",
+ " dataflow.googleapis.com \\\n",
+ " aiplatform.googleapis.com \\\n",
+ " bigquery.googleapis.com \\\n",
+ " --project \"{PROJECT_ID}\"\n",
+ "\n",
+ "# Create the regional Cloud Storage staging bucket with soft-delete disabled (avoids Dataflow temp storage warnings and charges).\n",
+ "!gcloud storage buckets describe \"gs://{BUCKET_NAME}\" --project=\"{PROJECT_ID}\" >/dev/null 2>&1 || \\\n",
+ " gcloud storage buckets create \"gs://{BUCKET_NAME}\" --project=\"{PROJECT_ID}\" --location=\"{LOCATION}\" --soft-delete-duration=0s\n",
+ "!gcloud storage buckets update \"gs://{BUCKET_NAME}\" --project=\"{PROJECT_ID}\" --clear-soft-delete >/dev/null 2>&1 || true\n",
+ "\n",
+ "# Create the BigQuery dataset if it doesn't already exist.\n",
+ "!bq --project_id=\"{PROJECT_ID}\" --location=\"{LOCATION}\" mk --dataset \"{PROJECT_ID}:{BIGQUERY_DATASET}\" 2>/dev/null || true\n",
+ "\n",
+ "print(f\"Cloud Storage bucket: gs://{BUCKET_NAME} (soft-delete disabled)\")\n",
+ "print(f\"BigQuery dataset: {PROJECT_ID}.{BIGQUERY_DATASET}\")"
],
- "outputs": [],
"metadata": {
- "cellView": "form",
- "id": "arH5wwVq2BZG"
- }
+ "id": "2e39c724"
+ },
+ "execution_count": null,
+ "outputs": []
},
{
"cell_type": "markdown",
"source": [
- "> ๐ Make sure the `region` matches the region you chose for your Cloud Storage bucket."
+ "### Initialize Gen AI SDK"
],
"metadata": {
- "id": "wc5yFA9W2BZG"
+ "id": "34a58e2f"
}
},
+ {
+ "cell_type": "code",
+ "source": [
+ "client = genai.Client(vertexai=True, project=PROJECT_ID, location=\"global\")"
+ ],
+ "metadata": {
+ "id": "45bdd3ac"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
{
"cell_type": "markdown",
"source": [
- "## ๐ป Creating the database of images metadata\n",
+ "## ๐ป Create the image metadata table\n",
"\n",
- "To build our model we need data, so we create a database table from the metadata of all the images.\n",
- "This is a _one-time only_ process.\n",
+ "Before running batch wildlife inference, create a BigQuery metadata table containing the species annotations and image file paths from the WCS Camera Traps archive. This is a _one-time_ setup step _(~6โ8 minutes total: ~3โ4 minutes for Dataflow worker VM startup + ~3โ4 minutes to stream 736,952 annotations into BigQuery)_:\n",
"\n",
- "Click run on the first Dataflow pipeline to create the images metadata table in BigQuery, which contains 2 fields of information from each image from the JSON file in WCS Camera Traps database:\n",
+ "Click **run** โถ๏ธ on the following cell to launch the Dataflow pipeline, which extracts two fields for each camera trap image from the WCS Camera Traps JSON archive:\n",
"\n",
- " - `category`: The _species_ we want to predict, this is our _label_.\n",
- " - `file_name`: The _file path_ where the image file is located.\n",
- " \n",
- "We do some very basic _data cleaning_ like discarding rows with categories that are note useful like `#ref!`, `empty`, `unidentifiable`, `unidentified`, `unknown`.\n",
+ "- `category`: The ground-truth species label.\n",
+ "- `file_name`: The relative file path of the image in `gs://public-datasets-lila/wcs-unzipped/`.\n",
"\n",
- "> ๐ก As you collect new images from the field via your own camera traps, you continue to store the new imageโs category and file name into the BigQuery table.\n",
- "> You could extend this sample to collect more pertinent data for reporting purposes such as location, date, etc."
+ "The pipeline filters out rows with categories that are not useful species labels, such as `#ref!`, `empty`, `unidentifiable`, `unidentified`, and `unknown`.\n",
+ "\n",
+ "> ๐ก As you collect new images from the field from your own camera traps, you can insert the new image categories and file paths into the BigQuery table.\n",
+ "> You can also extend this table schema to store additional ecological metadata, such as camera location and capture timestamp."
],
"metadata": {
"id": "GHs2UDDnT1YG"
@@ -335,14 +358,15 @@
"execution_count": null,
"source": [
"# [One time only] Create the images metadata table.\n",
+ "# Takes ~6-8 minutes on Dataflow (~3-4 min worker VM startup + ~3-4 min to ingest 736,952 rows).\n",
"!python create_images_metadata_table.py \\\n",
- " --bigquery-dataset \"{bigquery_dataset}\" \\\n",
- " --bigquery-table \"{bigquery_table}\" \\\n",
+ " --bigquery-dataset \"{BIGQUERY_DATASET}\" \\\n",
+ " --bigquery-table \"{BIGQUERY_TABLE}\" \\\n",
" --runner \"DataflowRunner\" \\\n",
" --job_name \"wildlife-images-metadata-`date +%Y%m%d-%H%M%S`\" \\\n",
- " --project \"{google_cloud_project}\" \\\n",
- " --temp_location \"gs://{cloud_storage_bucket}/{cloud_storage_directory}/temp\" \\\n",
- " --region \"{region}\" \\\n",
+ " --project \"{PROJECT_ID}\" \\\n",
+ " --temp_location \"gs://{BUCKET_NAME}/{CLOUD_STORAGE_DIRECTORY}/temp\" \\\n",
+ " --region \"{LOCATION}\" \\\n",
" --worker_machine_type \"n1-standard-2\""
],
"outputs": [],
@@ -360,11 +384,9 @@
"\n",
"\n",
"\n",
- "> ๐๏ธ We need at least `n1-standard-2` [worker machines](https://cloud.google.com/compute/docs/machine-types#n1_machine_types) due to large RAM usage when parsing the metadata JSON file.\n",
- "\n",
- "You can look up the job details in the Dataflow jobs page:\n",
+ "> ๐๏ธ The pipeline uses the `n1-standard-2` [machine type](https://cloud.google.com/compute/docs/machine-types#n1_machine_types) to provide sufficient memory when parsing the full WCS metadata JSON archive.\n",
"\n",
- "- [console.cloud.google.com/dataflow/jobs](https://console.cloud.google.com/dataflow/jobs)"
+ "You can view the job progress on the [Dataflow Jobs page in the Google Cloud console](https://console.cloud.google.com/dataflow/jobs)."
],
"metadata": {
"id": "p8rDiRzZcWvK"
@@ -373,49 +395,26 @@
{
"cell_type": "markdown",
"source": [
- "# ๐ Training the image classifier\n",
+ "## ๐ Run class-balanced wildlife inference with Apache Beam and Gemini\n",
"\n",
- "One of the challenges of a camera trap dataset is that it usually contains a _different number of images per species_, making it very _unbalanced_.\n",
+ "One of the classic challenges of a camera trap dataset is that it contains a _heavily skewed distribution of images per species_.\n",
"\n",
- "For example, there are tens of thousands of pictures of some species like [tayassu pecari](https://www.google.com/search?q=tayassu+pecari&tbm=isch), while only a handful of pictures of other species like [tolypeutes matacus](https://www.google.com/search?q=tolypeutes+matacus&tbm=isch).\n",
+ "In the LILA WCS Camera Traps dataset (`736,952` valid annotations across `639` species), there are **`95,788` photos** of common species like [tayassu pecari](https://www.google.com/search?q=tayassu+pecari&tbm=isch) (white-lipped peccary), while only **`54` photos** of critically endangered species like [dicerorhinus sumatrensis](https://www.google.com/search?q=dicerorhinus+sumatrensis&tbm=isch) (Sumatran rhinoceros) and just **`1` photo** of [tolypeutes matacus](https://www.google.com/search?q=tolypeutes+matacus&tbm=isch) (southern three-banded armadillo).\n",
"\n",
- "In _emojis_, the data looks _skewed_ like this: \n",
+ "In _emojis_, the data looks _skewed_ like this:\n",
"\n",
"- ๐ฆ๐ฆ๐ฆ๐ฆ๐ฆ (5 Green Macaws)\n",
"- ๐ฆง (1 Bornean orangutan)\n",
- "- ๐๐๐ (3 Sumatran elephant)\n",
+ "- ๐๐๐ (3 Sumatran elephants)\n",
"- ๐๐๐๐๐๐๐๐๐๐๐๐ (12 wild boars)\n",
"\n",
- "This could introduce a\n",
- "[_bias_](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias)\n",
- "into your model which can lead to it predicting a species just because it's more common rather than identifying it by itโs actual features.\n",
- "So instead of using the entire database of images, we leverage Dataflow to create a balanced dataset by specifying the _minimum_ and _maximum_ number of images we want for every category.\n",
- "For categories with too many images, it chooses at most `max_images_per_class` randomly selected images.\n",
- "And categories with less than `min_images_per_class` are discarded as _\"too little information to teach our model to classify this species\"_.\n",
- "\n",
- "> ๐ก For this sample, we defaulted to using _between 50 and 100 images_ per species/class to keep the training dataset small. This reduces the training time and the _potential cost_ of prediction accuracy.\n",
- "> Feel free to play around with other numbers.\n",
- "\n",
- "Once Dataflow chooses what images meet our criteria by counting the number of species per categories in BigQuery, it\n",
- "[lazily](https://en.wikipedia.org/wiki/Lazy_evaluation)\n",
- "downloads the actual JPEG images into Cloud Storage from the Lila Science database using itโs file name.\n",
- "This means we are only preprocessing the images we need for training rather than processing the entire dataset of JPEG images.\n",
- " \n",
- "When retraining in the future, if an image already exists in Cloud Storage, there's nothing else to do for that image, as we have created a _cache_ for images that have already been processed in our prior run.\n",
- "\n",
- "After all the images are stored in Cloud Storage, Dataflow creates a\n",
- "[CSV file for AI Platform](https://cloud.google.com/ai-platform-unified/docs/datasets/prepare-image#csv).\n",
- "Each row includes the Cloud Storage path of an image, alongside with the category, our label.\n",
- "\n",
- "Next, Dataflow tells AI Platform to import the CSV file and create a dataset from it. This is where the Dataflow job stops and AI Platform begins to train the model.\n",
- " \n",
- "We let AI Platform do the _data splitting_ automatically for us. By default, it uses _80% of the data for training_, _10% for validation_, and _10% for testing_. See [About data splits for AI Platform models](https://cloud.google.com/ai-platform-unified/docs/general/ml-use) for more information.\n",
- "\n",
- "> ๐๏ธ For simplicity, in this sample we are training a `CLOUD` model.\n",
- "> This allows us to deploy it to an HTTP endpoint and get predictions via the browser.\n",
- ">\n",
- "> See [Train an Edge model](https://cloud.google.com/ai-platform-unified/docs/training/automl-edge-console)\n",
- "> for information on how to train a model for a microcontroller."
+ "In traditional supervised ML (such as legacy AutoML Vision), species with fewer than `min_images_per_class = 50` had to be **discarded** because there was not enough training dataโmeaning **`359` out of `639` species (`56%`)**, including the rarest and most endangered animals, were excluded from the model.\n",
+ "\n",
+ "With **Gemini 3.8 Flash** (`gemini-3.8-flash`) inside Apache Beam's `RunInference` transform:\n",
+ "1. **No rare species are left behind**: You can set `min_images_per_class = 1` while still using `Sample.FixedSizePerKey(max_images_per_class)` so common species don't dominate your batch inference budget.\n",
+ "2. **Zero image duplication**: Instead of downloading and copying gigabytes of JPEGs into your own bucket, Gemini reads `gs://public-datasets-lila/wcs-unzipped/{file_name}` directly in place.\n",
+ "3. **Richer ecological recovery signals in one pass**: Using Structured Outputs (`WildlifeObservation`), Gemini returns not just `species_scientific_name` and `confidence`, but also `taxon_family`, `common_name`, `animal_count`, `juvenile_present` (to track breeding and population recovery), and `is_empty_or_misfire`.\n",
+ "4. **Zero idle endpoint cost**: Because `RunInference` calls the serverless Gemini API on Gemini Enterprise Agent Platform directly from Dataflow workers, there is no always-on model endpoint to deploy or pay for by the hour."
],
"metadata": {
"id": "I9sD-nSOmbK6"
@@ -425,37 +424,28 @@
"cell_type": "code",
"execution_count": null,
"source": [
- "min_images_per_class = 50 # @param {type:\"integer\"}\n",
- "max_images_per_class = 100 # @param {type:\"integer\"}"
- ],
- "outputs": [],
- "metadata": {
- "id": "V5Y4byrzxVMC",
- "cellView": "form"
- }
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "source": [
- "# Create a balanced dataset and signal AI Platform to train a model.\n",
- "!python train_model.py \\\n",
- " --cloud-storage-path \"gs://{cloud_storage_bucket}/{cloud_storage_directory}\" \\\n",
- " --bigquery-dataset \"{bigquery_dataset}\" \\\n",
- " --bigquery-table \"{bigquery_table}\" \\\n",
- " --ai-platform-name-prefix \"{ai_platform_name_prefix}\" \\\n",
+ "min_images_per_class = 1 # @param {type:\"integer\"}\n",
+ "max_images_per_class = 5 # @param {type:\"integer\"}\n",
+ "\n",
+ "# Run class-balanced sampling and Gemini 3.8 Flash RunInference into BigQuery.\n",
+ "# Takes ~1.5-3.5 hours across all 639 species (~3,000 images); interactive cells below can run immediately.\n",
+ "!python run_wildlife_inference.py \\\n",
+ " --project \"{PROJECT_ID}\" \\\n",
+ " --bigquery-dataset \"{BIGQUERY_DATASET}\" \\\n",
+ " --bigquery-table \"{BIGQUERY_TABLE}\" \\\n",
+ " --predictions-table \"{PREDICTIONS_TABLE}\" \\\n",
" --min-images-per-class \"{min_images_per_class}\" \\\n",
" --max-images-per-class \"{max_images_per_class}\" \\\n",
" --runner \"DataflowRunner\" \\\n",
- " --job_name \"wildlife-train-model-`date +%Y%m%d-%H%M%S`\" \\\n",
- " --project \"{google_cloud_project}\" \\\n",
- " --temp_location \"gs://{cloud_storage_bucket}/{cloud_storage_directory}/temp\" \\\n",
+ " --job_name \"wildlife-run-inference-`date +%Y%m%d-%H%M%S`\" \\\n",
+ " --temp_location \"gs://{BUCKET_NAME}/{CLOUD_STORAGE_DIRECTORY}/temp\" \\\n",
" --requirements_file \"requirements.txt\" \\\n",
- " --region \"{region}\""
+ " --region \"{LOCATION}\""
],
"outputs": [],
"metadata": {
- "id": "BgBiq2CzQ9Cp"
+ "id": "V5Y4byrzxVMC",
+ "cellView": "form"
}
},
{
@@ -464,21 +454,11 @@
"\n",
"\n",
- "> ๐๏ธ It can take several minutes for the job to show up in the\n",
- "> [Dataflow jobs page](https://console.cloud.google.com/dataflow/jobs).\n",
- "> See [[ARROW-8983]](https://issues.apache.org/jira/browse/ARROW-8983) for more information.\n",
- "\n",
- "Training the AI Platform model can take a while, depending on the dataset size and the training budget you allow.\n",
- "\n",
- "> ๐ก You can adjust the training budget using the `--budget-milli-node-hours` flag. We default to `8000` which is the minimum.\n",
- ">\n",
- "> See [the pricing page](https://cloud.google.com/vision/automl/pricing) and\n",
- "> [`train_budget_milli_node_hours`](https://cloud.google.com/automl/docs/reference/rpc/google.cloud.automl.v1#imageclassificationmodelmetadata)\n",
- "> for more information."
+ "You can monitor the pipeline progress in the [Dataflow jobs page](https://console.cloud.google.com/dataflow/jobs) and query the structured predictions table in [BigQuery](https://console.cloud.google.com/bigquery)."
],
"metadata": {
"id": "h_xtp4SrwLZN"
@@ -487,20 +467,11 @@
{
"cell_type": "markdown",
"source": [
- "# ๐ฃ Deploying the model\n",
+ "## ๐บ Visualize camera trap classification with Gemini 3.8 Flash\n",
"\n",
- "> ๐ Make sure your model has finished training.\n",
- "> You can look at the training progress on the\n",
- "> [AI Platform training jobs page](https://console.cloud.google.com/ai/platform/training/training-pipelines).\n",
+ "Because Gemini 3.8 Flash is a pre-trained multimodal foundation model on Gemini Enterprise Agent Platform (`location=\"global\"`), you don't need to wait hours to train or deploy a custom endpoint before classifying camera trap images.\n",
"\n",
- "> ๐ก If you were disconnected from the session due to inactivity, please make sure to re-run the cells at the _\"Before you begin\"_ section at the beginning of the notebook.\n",
- "\n",
- "We need to deploy the trained model into an endpoint to get predictions from it.\n",
- "Since we don't have a micro controller readily available, letโs deploy it in the cloud as an HTTP endpoint.\n",
- "\n",
- "You can deploy it through the Cloud Console: [console.cloud.google.com/ai/platform/models](https://console.cloud.google.com/ai/platform/models)\n",
- "\n",
- "Or alternatively, you can deploy it through the API by running the following cells:"
+ "Define helper functions to render images directly from the public LILA Science Cloud Storage bucket (`https://storage.googleapis.com/public-datasets-lila/wcs-unzipped`) and invoke [`classify_wildlife_image.py`](classify_wildlife_image.py):"
],
"metadata": {
"id": "WgHVM2Nwxw9r"
@@ -510,17 +481,27 @@
"cell_type": "code",
"execution_count": null,
"source": [
- "# First we need the model path, we can get it with gcloud.\n",
- "# ๐ก If you get an error here, please make sure your model has finished training.\n",
- "cmd_output = !gcloud beta ai models list \\\n",
- " --project {google_cloud_project} \\\n",
- " --region {region} \\\n",
- " --filter \"display_name:{ai_platform_name_prefix}*\" \\\n",
- " --format \"table[no-heading](display_name,name)\" 2>/dev/null\n",
- "models = sorted([line.split() for line in cmd_output])\n",
- "model_path = models[0][1]\n",
+ "LILA_WCS_HTTP_BASE = \"https://storage.googleapis.com/public-datasets-lila/wcs-unzipped\"\n",
+ "LILA_WCS_GCS_BASE = \"gs://public-datasets-lila/wcs-unzipped\"\n",
+ "\n",
+ "\n",
+ "def display_image(image_file: str, base_url: str = LILA_WCS_HTTP_BASE, width: int = 400) -> None:\n",
+ " response = requests.get(f\"{base_url}/{image_file}\", timeout=60)\n",
+ " response.raise_for_status()\n",
+ " image = Image.open(io.BytesIO(response.content))\n",
+ " display(image.resize((int(width), int(width / image.size[0] * image.size[1]))))\n",
+ "\n",
"\n",
- "print(f\"model_path: {model_path}\")"
+ "def predict(\n",
+ " image_file: str,\n",
+ " habitat_region: str = \"Southeast Asia tropical rainforest\",\n",
+ ") -> None:\n",
+ " display_image(image_file)\n",
+ " classify_wildlife_image.classify_wildlife_image(\n",
+ " project_id=PROJECT_ID,\n",
+ " image_gcs_uri=f\"{LILA_WCS_GCS_BASE}/{image_file}\",\n",
+ " habitat_region=habitat_region,\n",
+ " )"
],
"outputs": [],
"metadata": {
@@ -528,86 +509,74 @@
}
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"source": [
- "# Create an endpoint and deploy the model to it.\n",
- "!python deploy_model.py \\\n",
- " --project {google_cloud_project} \\\n",
- " --region {region} \\\n",
- " --model-path {model_path} \\\n",
- " --model-endpoint-name {ai_platform_name_prefix}"
+ "\n",
+ "\n",
+ "Now, run the following cells to see how `gemini-3.8-flash` classifies each camera trap photo into structured ecological metadata (`Species`, `Taxon Family`, `Common Name`, `Animal Count`, `Juvenile Present`, `Empty or Misfire`, and `Confidence`).\n",
+ "\n",
+ "First, classify a **Sumatran rhinoceros** (`Dicerorhinus sumatrensis`) camera trap frame:"
],
- "outputs": [],
"metadata": {
"id": "lRRg39FcYILC"
}
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
"source": [
- ""
+ "# Species: dicerorhinus sumatrensis (Sumatran rhinoceros)\n",
+ "predict(\"animals/0325/1529.jpg\")"
],
"metadata": {
"id": "VQPz8h9z2BZK"
- }
+ },
+ "execution_count": null,
+ "outputs": []
},
{
"cell_type": "markdown",
"source": [
- "# ๐บ Visualizing classification of images online\n",
- "\n",
- "Now that we have a deployed model, we can classify images using it.\n",
- " \n",
- "First, lets define some functions to help us visualize and navigate the images, click run:"
+ "Classify a **Guianan white-eared opossum** (`Didelphis imperfecta`) camera trap frame:"
+ ],
+ "metadata": {
+ "id": "bf03ee3d"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Species: didelphis imperfecta (Guianan white-eared opossum)\n",
+ "predict(\n",
+ " \"animals/0667/1214.jpg\",\n",
+ " habitat_region=\"Central and South America tropical rainforest\",\n",
+ ")"
],
"metadata": {
"id": "1jDpAELnxzf7"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Classify a **Malayan tapir** (`Tapirus indicus`) camera trap frame:"
+ ],
+ "metadata": {
+ "id": "9386b435"
}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
- "import io\n",
- "import requests\n",
- "from PIL import Image\n",
- "from IPython.display import display\n",
- "\n",
- "from google.cloud import bigquery\n",
- "\n",
- "\n",
- "def display_image(image_file, width=400):\n",
- " base_url = \"https://lilablobssc.blob.core.windows.net/wcs-unzipped\"\n",
- " image_bytes = requests.get(f\"{base_url}/{image_file}\").content\n",
- " if b\"\" in image_bytes:\n",
- " raise ValueError(\n",
- " f\"Error requesting image: {base_url}/{image_file}\\n{image_bytes.decode('utf-8')}\"\n",
- " )\n",
- " image = Image.open(io.BytesIO(image_bytes))\n",
- " display(image.resize((int(width), int(width / image.size[0] * image.size[1]))))\n",
- "\n",
- "\n",
- "def display_samples_for_category(category, num_samples=3, width=400):\n",
- " client = bigquery.Client()\n",
- " query_job = client.query(\n",
- " f\"\"\"\n",
- " SELECT file_name\n",
- " FROM `{google_cloud_project}.{bigquery_dataset}.{bigquery_table}`\n",
- " WHERE category = '{category}'\n",
- " LIMIT {num_samples}\n",
- " \"\"\"\n",
- " )\n",
- "\n",
- " for row in query_job:\n",
- " image_file = row[\"file_name\"]\n",
- " print(f\"{category}: {image_file}\")\n",
- " display_image(image_file, width)"
+ "# Species: tapirus indicus (Malayan tapir)\n",
+ "predict(\"animals/0036/0072.jpg\")"
],
"outputs": [],
"metadata": {
@@ -617,43 +586,66 @@
{
"cell_type": "markdown",
"source": [
- "Now run the following cell to get the model endpoint ID, which we need to make predictions."
+ "Classify a **margay** (`Leopardus wiedii`) camera trap frame:"
+ ],
+ "metadata": {
+ "id": "d021f39a"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "# Species: leopardus wiedii (Margay)\n",
+ "predict(\n",
+ " \"animals/0000/1705.jpg\",\n",
+ " habitat_region=\"Central and South America tropical rainforest\",\n",
+ ")"
],
"metadata": {
"id": "AFCtx6I195DA"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Classify a **banded palm civet** (`Hemigalus derbyanus`) camera trap frame:"
+ ],
+ "metadata": {
+ "id": "dac341c7"
}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
- "# First we need the endpoint ID, we can get it with gcloud.\n",
- "stdout = !gcloud beta ai endpoints list \\\n",
- " --project {google_cloud_project} \\\n",
- " --region {region} \\\n",
- " --filter \"display_name={ai_platform_name_prefix}\" \\\n",
- " --format \"table[no-heading](ENDPOINT_ID)\" 2>/dev/null\n",
- "model_endpoint_id = stdout[0]\n",
- "\n",
- "print(f\"model_endpoint_id: {model_endpoint_id}\")"
+ "# Species: hemigalus derbyanus (Banded palm civet)\n",
+ "predict(\"animals/0036/0566.jpg\")"
],
"outputs": [],
"metadata": {
"id": "M9jn39iv6EDu"
}
},
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Classify a **nine-banded armadillo** (`Dasypus novemcinctus`) camera trap frame:"
+ ],
+ "metadata": {
+ "id": "5a0d7818"
+ }
+ },
{
"cell_type": "code",
"execution_count": null,
"source": [
- "def predict(image_file):\n",
- " # First, display the image, and then we run our prediction script.\n",
- " display_image(image_file)\n",
- " !python predict.py \\\n",
- " --project \"{google_cloud_project}\" \\\n",
- " --region \"{region}\" \\\n",
- " --model-endpoint-id \"{model_endpoint_id}\" \\\n",
- " --image-file \"{image_file}\""
+ "# Species: dasypus novemcinctus (Nine-banded armadillo)\n",
+ "predict(\n",
+ " \"animals/0000/0425.jpg\",\n",
+ " habitat_region=\"Central and South America tropical rainforest\",\n",
+ ")"
],
"outputs": [],
"metadata": {
@@ -663,15 +655,15 @@
{
"cell_type": "markdown",
"source": [
- "\n",
+ "With Gemini's structured output schema (`WildlifeObservation`), you can address this in two ways without training a separate model:\n",
+ "1. **Taxonomic hierarchy (`taxon_family`)**: Every prediction returns both `species_scientific_name` and `taxon_family` (for example, `Felidae`), giving ecologists reliable family-level rollups even on blurry or partially occluded frames.\n",
+ "2. **Habitat geo-fencing (`habitat_region`)**: Because camera traps are stationary, passing the trap's geographic region in the prompt constrains predictions to species native to that ecosystem.\n",
"\n",
- "Now, run the following cells to see what our model thinks about each of the following images.\n",
- "The model gives us a list of species it predicts it is, alongside with the confidence score for each prediction."
+ "Run the following cell to test habitat geo-fencing on a camera trap photo from the **Felidae Conservation Fund** dataset (`gs://public-datasets-lila/felidae-conservation-fund/`), captured in the **San Francisco Bay Area, California**:"
],
"metadata": {
"id": "nScTC8Si2BZL"
@@ -681,8 +673,17 @@
"cell_type": "code",
"execution_count": null,
"source": [
- "# Species: dicerorhinus sumatrensis\n",
- "predict(\"animals/0325/1529.jpg\")"
+ "# San Francisco Bay Area mountain lion (Puma concolor) from Felidae Conservation Fund (LILA Science)\n",
+ "sf_bay_image = \"1/2025-03/53c186d3a51ede0a8a002d10dc9af6a985fbc70d04a17a335847e7179ce1a812.jpg\"\n",
+ "display_image(\n",
+ " sf_bay_image,\n",
+ " base_url=\"https://storage.googleapis.com/public-datasets-lila/felidae-conservation-fund\",\n",
+ ")\n",
+ "classify_wildlife_image.classify_wildlife_image(\n",
+ " project_id=PROJECT_ID,\n",
+ " image_gcs_uri=f\"gs://public-datasets-lila/felidae-conservation-fund/{sf_bay_image}\",\n",
+ " habitat_region=\"San Francisco Bay Area, California, USA\",\n",
+ ")"
],
"outputs": [],
"metadata": {
@@ -690,13 +691,21 @@
}
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"source": [
- "# Species: didelphis imperfecta\n",
- "predict(\"animals/0667/1214.jpg\")"
+ "## ๐งฌ Discover rare species with Gemini Embedding 2\n",
+ "\n",
+ "When rangers capture a single sighting of a critically endangered animalโsuch as the **Sumatran rhinoceros** (`Dicerorhinus sumatrensis`, with fewer than 50 individuals remaining in the wild)โthey often need to search millions of historical camera trap frames for visually similar animals.\n",
+ "\n",
+ "Using [`embed_wildlife_image.py`](embed_wildlife_image.py) and **Gemini Embedding 2** (`gemini-embedding-2`), you can generate a 768-dimensional multimodal vector embedding for any camera trap image:\n",
+ "\n",
+ ""
],
- "outputs": [],
"metadata": {
"id": "VF-S9qpnEqjD"
}
@@ -705,8 +714,11 @@
"cell_type": "code",
"execution_count": null,
"source": [
- "# Species: tapirus indicus\n",
- "predict(\"animals/0036/0072.jpg\")"
+ "# Generate a 768-dimensional multimodal embedding for a Sumatran rhinoceros camera trap frame\n",
+ "embed_wildlife_image.embed_wildlife_image(\n",
+ " project_id=PROJECT_ID,\n",
+ " image_gcs_uri=\"gs://public-datasets-lila/wcs-unzipped/animals/0325/1529.jpg\",\n",
+ ")"
],
"outputs": [],
"metadata": {
@@ -714,106 +726,120 @@
}
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"source": [
- "# Species: leopardus wiedii\n",
- "predict(\"animals/0000/1705.jpg\")"
+ "Run the following cell _(~10 seconds)_ to embed the sample camera trap images into a BigQuery table (`samples.wildlife_image_embeddings`) with an `ARRAY` column and execute a live `VECTOR_SEARCH` query to rank the closest matching frames by cosine distance:"
],
- "outputs": [],
"metadata": {
"id": "qJp96gNqHUwn"
}
},
{
"cell_type": "code",
- "execution_count": null,
"source": [
- "# Species: hemigalus derbyanus\n",
- "predict(\"animals/0036/0566.jpg\")"
+ "# Takes ~10 seconds to embed 6 camera trap frames, load into BigQuery, and run VECTOR_SEARCH.\n",
+ "import json\n",
+ "from google.genai import types\n",
+ "\n",
+ "EMBEDDINGS_TABLE = \"wildlife_image_embeddings\"\n",
+ "gallery = [\n",
+ " (\"gs://public-datasets-lila/wcs-unzipped/animals/0325/1529.jpg\", \"Dicerorhinus sumatrensis (Sumatran rhinoceros)\"),\n",
+ " (\"gs://public-datasets-lila/wcs-unzipped/animals/0036/0072.jpg\", \"Tapirus indicus (Malayan tapir)\"),\n",
+ " (\"gs://public-datasets-lila/wcs-unzipped/animals/0667/1214.jpg\", \"Didelphis imperfecta (Guianan white-eared opossum)\"),\n",
+ " (\"gs://public-datasets-lila/wcs-unzipped/animals/0000/1705.jpg\", \"Leopardus wiedii (Margay)\"),\n",
+ " (\"gs://public-datasets-lila/wcs-unzipped/animals/0036/0566.jpg\", \"Hemigalus derbyanus (Banded palm civet)\"),\n",
+ " (\"gs://public-datasets-lila/wcs-unzipped/animals/0000/0425.jpg\", \"Dasypus novemcinctus (Nine-banded armadillo)\"),\n",
+ "]\n",
+ "with open(\"embeddings.jsonl\", \"w\") as f:\n",
+ " for gcs_uri, species_label in gallery:\n",
+ " part = types.Part.from_uri(file_uri=gcs_uri, mime_type=\"image/jpeg\")\n",
+ " resp = client.models.embed_content(\n",
+ " model=\"gemini-embedding-2\",\n",
+ " contents=[part],\n",
+ " config=types.EmbedContentConfig(output_dimensionality=768),\n",
+ " )\n",
+ " row = {\"image_gcs_uri\": gcs_uri, \"species_scientific_name\": species_label, \"embedding\": list(resp.embeddings[0].values)}\n",
+ " f.write(json.dumps(row) + \"\\n\")\n",
+ "\n",
+ "schema = [\n",
+ " {\"name\": \"image_gcs_uri\", \"type\": \"STRING\", \"mode\": \"NULLABLE\"},\n",
+ " {\"name\": \"species_scientific_name\", \"type\": \"STRING\", \"mode\": \"NULLABLE\"},\n",
+ " {\"name\": \"embedding\", \"type\": \"FLOAT64\", \"mode\": \"REPEATED\"},\n",
+ "]\n",
+ "with open(\"embeddings_schema.json\", \"w\") as f:\n",
+ " json.dump(schema, f)\n",
+ "\n",
+ "!bq --project_id=\"{PROJECT_ID}\" load --source_format=NEWLINE_DELIMITED_JSON --replace \\\n",
+ " \"{PROJECT_ID}:{BIGQUERY_DATASET}.{EMBEDDINGS_TABLE}\" embeddings.jsonl embeddings_schema.json\n",
+ "!bq --project_id=\"{PROJECT_ID}\" query --use_legacy_sql=false \\\n",
+ " \"SELECT base.species_scientific_name, base.image_gcs_uri, ROUND(distance, 4) AS cosine_distance FROM VECTOR_SEARCH(TABLE \\`{PROJECT_ID}.{BIGQUERY_DATASET}.{EMBEDDINGS_TABLE}\\`, 'embedding', (SELECT embedding FROM \\`{PROJECT_ID}.{BIGQUERY_DATASET}.{EMBEDDINGS_TABLE}\\` WHERE image_gcs_uri = 'gs://public-datasets-lila/wcs-unzipped/animals/0325/1529.jpg'), top_k => 6, distance_type => 'COSINE') ORDER BY cosine_distance ASC\""
],
- "outputs": [],
"metadata": {
- "id": "yl523QeUGXmR"
- }
- },
- {
- "cell_type": "code",
+ "id": "f9a99c63"
+ },
"execution_count": null,
- "source": [
- "# Species: dasypus novemcinctus\n",
- "predict(\"animals/0000/0425.jpg\")"
- ],
- "outputs": [],
- "metadata": {
- "id": "xlgjmCEQRUsS"
- }
+ "outputs": []
},
{
"cell_type": "markdown",
"source": [
- "๐ต๏ธ While analyzing the modelโs evaluation, you may notice that many species of the same family are very similar and it may be confusing the model. This means it might be worth trying to classify the image to a family instead of a specific species, or get more images from different angles/proximity to detect it better."
+ "## Cleaning up\n",
+ "\n",
+ "To avoid incurring ongoing storage charges for the resources created in this notebook, run the following cell to delete the BigQuery dataset and the temporary Dataflow staging files in Cloud Storage:"
],
"metadata": {
- "id": "r7INPuD2I2vV"
+ "id": "MGl_1UyiGVl6"
}
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
"source": [
- "# ๐งน Cleaning up to avoid additional costs\n",
+ "delete_resources = True # @param {type:\"boolean\"}\n",
+ "\n",
+ "if delete_resources or os.getenv(\"IS_TESTING\"):\n",
+ " # Delete the BigQuery dataset (including the metadata, predictions, and embeddings tables).\n",
+ " !bq rm -r -f -d \"{PROJECT_ID}:{BIGQUERY_DATASET}\"\n",
"\n",
- "To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.\n",
- " \n",
- "If you wish to delete the project you created for this tutorial (itโs the easiest way to eliminate billing), here are the steps:"
+ " # Delete the temporary Dataflow staging files in Cloud Storage.\n",
+ " !gcloud storage rm -r \"gs://{BUCKET_NAME}/{CLOUD_STORAGE_DIRECTORY}\""
],
"metadata": {
- "id": "MGl_1UyiGVl6"
- }
+ "id": "8418262e"
+ },
+ "execution_count": null,
+ "outputs": []
},
{
"cell_type": "markdown",
"source": [
- "## Deleting the project\n",
- "\n",
- "The easiest way to eliminate billing is to delete the project that you created for the tutorial.\n",
+ "### Optional: Delete a dedicated project\n",
"\n",
- "To delete the project:\n",
+ "If you created a dedicated Google Cloud project solely for this notebook and want to delete the entire project, uncomment and run the command in the following cell:\n",
"\n",
- "> โ ๏ธ **Caution**: Deleting a project has the following effects:\n",
- ">\n",
- "> - **Everything in the project is deleted.** If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.\n",
- "> - **Custom project IDs are lost.** When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an `appspot.com` URL, delete selected resources inside the project instead of deleting the whole project.\n",
- ">\n",
- "> If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.\n",
- "\n",
- "1. In the Cloud Console, go to the **Manage resources** page.\n",
- "\n",
- " \n",
- "\n",
- "1. In the project list, select the project that you want to delete, and then click **Delete**.\n",
- "\n",
- "1. In the dialog, type the project ID, and then click **Shut down** to delete the project.\n"
+ "> โ ๏ธ **Caution**: Deleting a project permanently deletes everything in the project and releases the project ID. Only delete the project if you created it exclusively for this notebook."
],
"metadata": {
"id": "nv7phC7WGZuT"
}
},
+ {
+ "cell_type": "code",
+ "source": [
+ "# Uncomment and run the following line ONLY if you created a dedicated project exclusively for this notebook:\n",
+ "# !gcloud projects delete \"{PROJECT_ID}\" --quiet"
+ ],
+ "metadata": {
+ "id": "a64bdcae"
+ },
+ "execution_count": null,
+ "outputs": []
+ },
{
"cell_type": "markdown",
"source": [
- "## ๐ Congratulations, you have successfully completed this code lab!\n",
- " \n",
- "๐ You made it this far, congrats.\n",
- "Dive deeper by visiting the\n",
- "[code on GitHub](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/people-and-planet-ai/image-classification).\n",
+ "## ๐ What's next\n",
"\n",
- "> ๐๏ธ Was this a meaningful project for you? If so you can you share your experience in\n",
- "> [these 2 quick questions here](https://docs.google.com/forms/d/e/1FAIpQLSeClJev12m2CGE5uwJAvfJRZe9LYUZR4nimsDjrRjvc5xBifA/viewform). "
+ "Explore the complete sample code on [GitHub](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/people-and-planet-ai/image-classification)."
],
"metadata": {
"id": "Em4E_RNv2BZM"
diff --git a/people-and-planet-ai/image-classification/README.md b/people-and-planet-ai/image-classification/README.md
index 8de9c232fd2..a8a62651008 100644
--- a/people-and-planet-ai/image-classification/README.md
+++ b/people-and-planet-ai/image-classification/README.md
@@ -2,20 +2,29 @@
[](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/image-classification/README.ipynb)
-> [Watch the video in YouTube ](https://youtu.be/hUzODH3uGg0)
+> [Watch the video on YouTube ](https://youtu.be/hUzODH3uGg0)
-This model is trained to recognize animal species from
+Classify wildlife species and extract ecological recovery metrics from
[camera trap](https://en.wikipedia.org/wiki/Camera_trap)
-pictures.
+pictures in the public [LILA Science](https://lila.science/datasets/wcscameratraps) Cloud Storage bucket (`gs://public-datasets-lila/`).
-* **Model**: [AutoML] Vision
-* **Creating datasets**: [Apache Beam] in [Dataflow]
-* **Training the model**: [AutoML] in [Vertex AI]
-* **Getting predictions**: [Vertex AI]
+* **Multimodal classification and ecological extraction**: [Gemini] (`gemini-3.8-flash`) via the [Google Gen AI SDK]
+* **Rare-species visual similarity and vector search**: [Gemini Embedding 2] (`gemini-embedding-2`)
+* **Class-balanced batch inference pipeline**: [Apache Beam] `RunInference` in [Dataflow]
+* **Metadata and structured observation storage**: [BigQuery]
+
+## Snippet files
+
+* [`create_images_metadata_table.py`](create_images_metadata_table.py) (`dataflow_wildlife_ingest_metadata`): Ingests and cleans LILA WCS camera trap annotations into BigQuery.
+* [`classify_wildlife_image.py`](classify_wildlife_image.py) (`googlegenaisdk_wildlife_classify_image`): Classifies a camera trap image into structured ecological metadata (`species_scientific_name`, `taxon_family`, `common_name`, `animal_count`, `juvenile_present`, `is_empty_or_misfire`, `confidence`) with optional habitat geo-fencing (`habitat_region`).
+* [`embed_wildlife_image.py`](embed_wildlife_image.py) (`googlegenaisdk_wildlife_embed_image`): Generates a 768-dimensional multimodal vector embedding for a camera trap image using `gemini-embedding-2`.
+* [`run_wildlife_inference.py`](run_wildlife_inference.py) (`dataflow_wildlife_run_inference`): Runs class-balanced sampling (`Sample.FixedSizePerKey`) and batch Gemini inference (`RunInference`) directly against `gs://public-datasets-lila/wcs-unzipped/` and writes results to BigQuery.
[Apache Beam]: https://beam.apache.org
+[BigQuery]: https://cloud.google.com/bigquery
[Dataflow]: https://cloud.google.com/dataflow
-[AutoML]: https://cloud.google.com/vertex-ai/docs/beginner/beginners-guide
-[Vertex AI]: https://cloud.google.com/vertex-ai
+[Gemini]: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/3-8-flash
+[Gemini Embedding 2]: https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/embedding-2
+[Google Gen AI SDK]: https://googleapis.github.io/python-genai/
-> ๐ Read more about this sample in the [blog post](https://cloud.google.com/blog/topics/developers-practitioners/recovering-global-wildlife-populations-using-ml).
+> ๐ For more information about the background of this solution, see [Recovering global wildlife populations using ML](https://cloud.google.com/blog/topics/developers-practitioners/recovering-global-wildlife-populations-using-ml).
diff --git a/people-and-planet-ai/image-classification/classify_wildlife_image.py b/people-and-planet-ai/image-classification/classify_wildlife_image.py
new file mode 100644
index 00000000000..7027bba487f
--- /dev/null
+++ b/people-and-planet-ai/image-classification/classify_wildlife_image.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python
+
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# [START googlegenaisdk_wildlife_classify_image]
+import mimetypes
+import sys
+
+from google import genai
+from google.genai import errors
+from google.genai import types
+from pydantic import BaseModel
+
+
+class WildlifeObservation(BaseModel):
+ species_scientific_name: str
+ taxon_family: str
+ common_name: str
+ animal_count: int
+ juvenile_present: bool
+ is_empty_or_misfire: bool
+ confidence: float
+
+
+def classify_wildlife_image(
+ project_id: str,
+ image_gcs_uri: str,
+ habitat_region: str = "Southeast Asia tropical rainforest",
+) -> None:
+ """Classifies a camera trap image into structured ecological metadata.
+
+ Uses Gemini 3.8 Flash on Gemini Enterprise Agent Platform using the global
+ endpoint with a Pydantic response schema and geographic habitat context
+ (geo-fencing) to identify the species and taxonomic family, count
+ individuals, detect juveniles, and flag empty or misfired frames.
+
+ Args:
+ project_id: Google Cloud project identifier (for example, 'example-project-id').
+ image_gcs_uri: Cloud Storage URI of the camera trap JPEG image.
+ habitat_region: Geographic region used to ground species predictions.
+ """
+ # gemini-3.8-flash serves on the global endpoint.
+ location = "global"
+ client = genai.Client(vertexai=True, project=project_id, location=location)
+
+ mime_type, _ = mimetypes.guess_type(image_gcs_uri)
+ image_part = types.Part.from_uri(
+ file_uri=image_gcs_uri,
+ mime_type=mime_type or "image/jpeg",
+ )
+ habitat_prefix = (
+ f"Camera trap habitat region: {habitat_region}. "
+ if habitat_region
+ else ""
+ )
+ prompt = (
+ f"{habitat_prefix}"
+ "Identify the wildlife species (scientific name, taxonomic family, and "
+ "common name) in this camera trap photo, count the individuals, "
+ "indicate whether juvenile offspring are present, and flag whether the "
+ "frame is empty or a camera misfire."
+ )
+ thinking_config = types.ThinkingConfig(
+ thinking_level=types.ThinkingLevel.LOW,
+ )
+ config = types.GenerateContentConfig(
+ response_mime_type="application/json",
+ response_schema=WildlifeObservation,
+ thinking_config=thinking_config,
+ )
+
+ try:
+ response = client.models.generate_content(
+ model="gemini-3.8-flash",
+ contents=[image_part, prompt],
+ config=config,
+ )
+ except errors.APIError as err:
+ print(
+ f"error: failed to classify camera trap image '{image_gcs_uri}' in "
+ f"location '{location}' for project '{project_id}': {err.message}",
+ file=sys.stderr,
+ )
+ else:
+ observation = WildlifeObservation.model_validate_json(response.text)
+ print(f"Resource ID: {image_gcs_uri}")
+ print(f"Animal Count: {observation.animal_count}")
+ print(f"Common Name: {observation.common_name}")
+ print(f"Confidence: {observation.confidence:.2f}")
+ print(f"Empty or Misfire: {observation.is_empty_or_misfire}")
+ print(f"Habitat Region: {habitat_region}")
+ print(f"Juvenile Present: {observation.juvenile_present}")
+ print(f"Species: {observation.species_scientific_name}")
+ print(f"Taxon Family: {observation.taxon_family}")
+# [END googlegenaisdk_wildlife_classify_image]
+
+
+if __name__ == "__main__":
+ import argparse
+ import os
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--project",
+ default=os.environ.get("GOOGLE_CLOUD_PROJECT", "example-project-id"),
+ help="Google Cloud project ID.",
+ )
+ parser.add_argument(
+ "--image-gcs-uri",
+ default=(
+ "gs://public-datasets-lila/wcs-unzipped/animals/0036/0072.jpg"
+ ),
+ help="Cloud Storage URI of the camera trap JPEG image.",
+ )
+ parser.add_argument(
+ "--habitat-region",
+ default="Southeast Asia tropical rainforest",
+ help="Geographic region of the camera trap for geo-fenced classification.",
+ )
+ args = parser.parse_args()
+
+ classify_wildlife_image(
+ project_id=args.project,
+ image_gcs_uri=args.image_gcs_uri,
+ habitat_region=args.habitat_region,
+ )
diff --git a/people-and-planet-ai/image-classification/create_images_metadata_table.py b/people-and-planet-ai/image-classification/create_images_metadata_table.py
index e8120b2fd18..3657b675395 100644
--- a/people-and-planet-ai/image-classification/create_images_metadata_table.py
+++ b/people-and-planet-ai/image-classification/create_images_metadata_table.py
@@ -1,6 +1,6 @@
#!/usr/bin/env python
-# Copyright 2021 Google LLC
+# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -14,20 +14,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+# [START dataflow_wildlife_ingest_metadata]
from __future__ import annotations
from collections.abc import Iterable
import io
import json
-import logging
import os
+import sys
+import time
import zipfile
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
+from google.api_core import exceptions
import requests
-METADATA_URL = "https://lilablobssc.blob.core.windows.net/wcs/wcs_camera_traps.json.zip"
+METADATA_URL = (
+ "https://storage.googleapis.com/public-datasets-lila/wcs/wcs_camera_traps.json.zip"
+)
INVALID_CATEGORIES = {
"#ref!",
@@ -42,89 +47,114 @@
}
-def run(
+class _ExtractMetadataFn(beam.DoFn):
+ """Downloads and parses camera trap image annotations from LILA Science."""
+
+ def process(self, metadata_url: str) -> Iterable[dict[str, str]]:
+ content = b""
+ for attempt in range(3):
+ try:
+ content = requests.get(metadata_url, timeout=120).content
+ break
+ except requests.RequestException:
+ if attempt == 2:
+ raise
+ time.sleep(2**attempt)
+
+ with zipfile.ZipFile(io.BytesIO(content)) as archive:
+ filename = os.path.splitext(os.path.basename(metadata_url))[0]
+ with archive.open(filename) as metadata_file:
+ metadata = json.load(metadata_file)
+
+ categories = {
+ category["id"]: category["name"]
+ for category in metadata["categories"]
+ }
+ file_names = {
+ image["id"]: image["file_name"] for image in metadata["images"]
+ }
+
+ for annotation in metadata["annotations"]:
+ category_id = annotation["category_id"]
+ image_id = annotation["image_id"]
+ if category_id not in categories or image_id not in file_names:
+ continue
+ category_name = categories[category_id]
+ if (
+ category_name in INVALID_CATEGORIES
+ or category_name.startswith("unknown ")
+ or category_name.endswith(" desconocida")
+ or category_name.endswith(" desconocido")
+ ):
+ continue
+ yield {
+ "category": category_name,
+ "file_name": file_names[image_id],
+ }
+
+
+def create_images_metadata_table(
+ project_id: str,
bigquery_dataset: str,
bigquery_table: str,
pipeline_options: PipelineOptions | None = None,
) -> None:
- """Creates the images metadata table in BigQuery.
+ """Ingests LILA WCS camera trap image metadata into a BigQuery table.
- This is a one time only process. It reads the metadata file from the LILA
- science WCS database, gets rid of invalid rows and uploads all the
- `file_names` alongside their respective `category` into BigQuery.
-
- To learn more about the WCS Camera Traps dataset:
- http://lila.science/datasets/wcscameratraps
+ Reads the Wildlife Conservation Society (WCS) Camera Traps metadata archive
+ from the public LILA Science Cloud Storage bucket, filters out unannotated
+ or ambiguous records, and writes the species category and relative image
+ path to BigQuery.
Args:
- bigquery_dataset: Dataset ID for the images database, the dataset must exist.
- bigquery_table: Table ID for the images database, it is created if it doesn't exist.
- pipeline_options: PipelineOptions for Apache Beam.
+ project_id: Google Cloud project identifier (for example, 'example-project-id').
+ bigquery_dataset: BigQuery dataset identifier where the table is stored.
+ bigquery_table: BigQuery table identifier to create or overwrite.
+ pipeline_options: Optional Apache Beam pipeline execution configuration.
"""
- # We create a simple schema for the BigQuery table.
- # For more information on BigQuery schemas, see:
- # https://cloud.google.com/bigquery/docs/schemas
- schema = ",".join(
- [
- "category:STRING",
- "file_name:STRING",
- ]
- )
-
- with beam.Pipeline(options=pipeline_options) as pipeline:
- (
- pipeline
- | "Create None" >> beam.Create([METADATA_URL])
- | "Get images info" >> beam.FlatMap(get_images_metadata)
- | "Filter invalid rows"
- >> beam.Filter(
- lambda x: x["category"] not in INVALID_CATEGORIES
- or x["category"].startswith("unknown ")
- or x["category"].endswith(" desconocida")
- or x["category"].endswith(" desconocido")
- )
- | "Write images database"
- >> beam.io.WriteToBigQuery(
- dataset=bigquery_dataset,
- table=bigquery_table,
- schema=schema,
- write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
- create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+ table_spec = f"{project_id}:{bigquery_dataset}.{bigquery_table}"
+ schema = "category:STRING,file_name:STRING"
+
+ try:
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ (
+ pipeline
+ | "SeedMetadataUrl" >> beam.Create([METADATA_URL])
+ | "ParseValidAnnotations" >> beam.ParDo(_ExtractMetadataFn())
+ | "WriteMetadataToBigQuery"
+ >> beam.io.WriteToBigQuery(
+ table=table_spec,
+ schema=schema,
+ write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
+ create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+ )
)
+ except (exceptions.GoogleAPICallError, RuntimeError) as err:
+ print(
+ f"error: failed to populate BigQuery table '{table_spec}' for "
+ f"project '{project_id}': {err}",
+ file=sys.stderr,
)
-
-
-def get_images_metadata(metadata_url: str) -> Iterable[dict[str, str]]:
- """Returns an iterable of {'category', 'file_name'} dicts."""
- content = requests.get(metadata_url).content
- with zipfile.ZipFile(io.BytesIO(content)) as zf:
- filename = os.path.splitext(os.path.basename(metadata_url))[0]
- with zf.open(filename) as f:
- metadata = json.load(f)
-
- categories = {
- category["id"]: category["name"] for category in metadata["categories"]
- }
- file_names = {image["id"]: image["file_name"] for image in metadata["images"]}
-
- for annotation in metadata["annotations"]:
- category_id = annotation["category_id"]
- image_id = annotation["image_id"]
- if category_id not in categories:
- logging.error(f"invalid category ID {category_id}, skipping")
- elif image_id not in file_names:
- logging.error(f"invalid image ID {image_id}, skipping")
- else:
- yield {
- "category": categories[category_id],
- "file_name": file_names[image_id],
- }
+ else:
+ print(f"Resource ID: {table_spec}")
+ print(f"BigQuery Dataset: {bigquery_dataset}")
+ print(f"BigQuery Table: {bigquery_table}")
+ print(f"Metadata Archive: {METADATA_URL}")
+# [END dataflow_wildlife_ingest_metadata]
if __name__ == "__main__":
import argparse
+ import logging
+
+ logging.getLogger().setLevel(logging.INFO)
parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--project",
+ default=os.environ.get("GOOGLE_CLOUD_PROJECT", "example-project-id"),
+ help="Google Cloud project ID.",
+ )
parser.add_argument(
"--bigquery-dataset",
required=True,
@@ -137,5 +167,12 @@ def get_images_metadata(metadata_url: str) -> Iterable[dict[str, str]]:
)
args, pipeline_args = parser.parse_known_args()
- pipeline_options = PipelineOptions(pipeline_args, save_main_session=True)
- run(args.bigquery_dataset, args.bigquery_table, pipeline_options)
+ options = PipelineOptions(
+ pipeline_args, project=args.project, save_main_session=True
+ )
+ create_images_metadata_table(
+ project_id=args.project,
+ bigquery_dataset=args.bigquery_dataset,
+ bigquery_table=args.bigquery_table,
+ pipeline_options=options,
+ )
diff --git a/people-and-planet-ai/image-classification/deploy_model.py b/people-and-planet-ai/image-classification/deploy_model.py
deleted file mode 100644
index ca52e26a439..00000000000
--- a/people-and-planet-ai/image-classification/deploy_model.py
+++ /dev/null
@@ -1,128 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2021 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from google.cloud import aiplatform
-
-
-def create_model_endpoint(project: str, region: str, model_endpoint_name: str) -> str:
- """Creates a model endpoint in AI Platform.
-
- Args:
- project: Google Cloud Project Id.
- region: Location for AI Platform resources.
- model_endpoint_name: AI Platform deployment endpoint name.
-
- Returns:
- The deployed model_endpoint_id.
- """
- client = aiplatform.gapic.EndpointServiceClient(
- client_options={"api_endpoint": "us-central1-aiplatform.googleapis.com"}
- )
-
- response = client.create_endpoint(
- parent=f"projects/{project}/locations/{region}",
- endpoint={"display_name": model_endpoint_name},
- )
- print(f"Creating model endpoint, operation: {response.operation.name}")
- model_endpoint = response.result()
- print(f"Model endpoint created\n{model_endpoint}")
- model_endpoint_id = model_endpoint.name.split("/")[-1]
- return model_endpoint_id
-
-
-def deploy_model(
- project: str, region: str, model_path: str, model_name: str, model_endpoint_id: str
-) -> str:
- """Deploys a model into an endpoint in AI Platform.
-
- Args:
- project: Google Cloud Project Id.
- region: Location for AI Platform resources.
- model_path: full model path.
- model_name: deployed model name.
- model_endpoint_id: deployment endpoint ID.
-
- Returns:
- The deployed_model_id.
- """
- client = aiplatform.gapic.EndpointServiceClient(
- client_options={"api_endpoint": "us-central1-aiplatform.googleapis.com"}
- )
-
- response = client.deploy_model(
- endpoint=client.endpoint_path(project, region, model_endpoint_id),
- deployed_model={
- "model": model_path,
- "display_name": model_name,
- "automatic_resources": {
- "min_replica_count": 1,
- "max_replica_count": 1,
- },
- },
- # key '0' assigns traffic for the newly deployed model
- # Traffic percentage values must add up to 100
- traffic_split={"0": 100},
- )
- print(f"Deploying model, operation: {response.operation.name}")
- deployed_model = response.result()
- print(f"Model deployed\n{deployed_model}")
- return deployed_model.deployed_model.id
-
-
-def run(project: str, region: str, model_path: str, model_endpoint_name: str) -> None:
- """Creates an endpoint in AI Platform and deploys a model to it.
-
- Args:
- project: Google Cloud Project Id.
- region: Location for AI Platform resources.
- model_path: full model path.
- model_endpoint_name: deployment endpoint name.
-
- Returns:
- The deployed model_endpoint_id.
- """
- model_endpoint_id = create_model_endpoint(project, region, model_endpoint_name)
- deploy_model(project, region, model_path, model_endpoint_name, model_endpoint_id)
- return model_endpoint_id
-
-
-if __name__ == "__main__":
- import argparse
-
- parser = argparse.ArgumentParser()
- parser.add_argument(
- "--project",
- required=True,
- help="Google Cloud Project Id",
- )
- parser.add_argument(
- "--region",
- required=True,
- help="Location for AI Platform resources",
- )
- parser.add_argument(
- "--model-path",
- required=True,
- help="Full model path",
- )
- parser.add_argument(
- "--model-endpoint-name",
- required=True,
- help="Deployment endpoint name",
- )
- args = parser.parse_args()
-
- run(args.project, args.region, args.model_path, args.model_endpoint_name)
diff --git a/people-and-planet-ai/image-classification/e2e_test.py b/people-and-planet-ai/image-classification/e2e_test.py
index 0cd14601803..b0b249c4f0c 100644
--- a/people-and-planet-ai/image-classification/e2e_test.py
+++ b/people-and-planet-ai/image-classification/e2e_test.py
@@ -1,4 +1,4 @@
-# Copyright 2021 Google LLC
+# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -12,216 +12,147 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+from __future__ import annotations
+
+from collections.abc import Iterable
import io
+import json
import os
-import subprocess
-import uuid
-
-import google
-from google.cloud import aiplatform
-from google.cloud import bigquery
-from google.cloud import storage
+from typing import Any
+from unittest import mock
+import zipfile
+
+import apache_beam as beam
+from apache_beam.ml.inference.base import RunInference
+from apache_beam.testing.util import assert_that
+import classify_wildlife_image
+import create_images_metadata_table
+import embed_wildlife_image
import pytest
+import run_wildlife_inference
-import deploy_model
-import predict
-
-SUFFIX = uuid.uuid4().hex[0:6]
-PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"]
-BUCKET_NAME = f"wildlife-insights-{SUFFIX}"
-BIGQUERY_DATASET = f"wildlife_insights_{SUFFIX}"
-BIGQUERY_TABLE = "images_database"
-MODEL_ENDPOINT = f"wildlife_insights_{SUFFIX}"
-REGION = "us-central1"
-MIN_IMAGES_PER_CLASS = 1
-MAX_IMAGES_PER_CLASS = 1
-
-# Use a pre-trained pre-existing model, training one takes too long.
-MODEL_PATH = f"projects/{PROJECT}/locations/{REGION}/models/8785722428534816768"
-
-
-@pytest.fixture(scope="session")
-def bucket_name() -> str:
- storage_client = storage.Client()
- bucket = storage_client.create_bucket(BUCKET_NAME)
+PROJECT = os.environ.get("GOOGLE_CLOUD_PROJECT")
+if not PROJECT:
+ pytest.skip(
+ "GOOGLE_CLOUD_PROJECT environment variable is required.",
+ allow_module_level=True,
+ )
- print(f"bucket_name: {repr(BUCKET_NAME)}")
- yield BUCKET_NAME
- bucket.delete(force=True)
+def test_classify_wildlife_image(capsys: pytest.CaptureFixture[str]) -> None:
+ image_uri = "gs://public-datasets-lila/wcs-unzipped/animals/0036/0072.jpg"
+ classify_wildlife_image.classify_wildlife_image(
+ project_id=PROJECT,
+ image_gcs_uri=image_uri,
+ habitat_region="Southeast Asia tropical rainforest",
+ )
+ out, err = capsys.readouterr()
+ assert err == ""
+ assert f"Resource ID: {image_uri}" in out
+ assert "Taxon Family: Tapiridae" in out
+ assert "Species: Tapirus indicus" in out
-@pytest.fixture(scope="session")
-def bigquery_dataset() -> str:
- bigquery_client = bigquery.Client()
+def test_classify_wildlife_image_geofenced_sf_bay(
+ capsys: pytest.CaptureFixture[str],
+) -> None:
+ image_uri = (
+ "gs://public-datasets-lila/felidae-conservation-fund/"
+ "1/2025-03/53c186d3a51ede0a8a002d10dc9af6a985fbc70d04a17a335847e7179ce1a812.jpg"
+ )
+ classify_wildlife_image.classify_wildlife_image(
+ project_id=PROJECT,
+ image_gcs_uri=image_uri,
+ habitat_region="San Francisco Bay Area, California, USA",
+ )
+ out, err = capsys.readouterr()
+ assert err == ""
+ assert f"Resource ID: {image_uri}" in out
+ assert "Taxon Family: Felidae" in out
+ assert "Species: Puma concolor" in out
+ assert "Habitat Region: San Francisco Bay Area, California, USA" in out
+
+
+def test_embed_wildlife_image(capsys: pytest.CaptureFixture[str]) -> None:
+ image_uri = "gs://public-datasets-lila/wcs-unzipped/animals/0036/0072.jpg"
+ embed_wildlife_image.embed_wildlife_image(
+ project_id=PROJECT,
+ image_gcs_uri=image_uri,
+ )
+ out, err = capsys.readouterr()
+ assert err == ""
+ assert f"Resource ID: {image_uri}" in out
+ assert "Embedding Dimensions: 768" in out
+ assert "Model: gemini-embedding-2" in out
+
+
+def test_extract_metadata_fn_filters_invalid_categories() -> None:
+ sample_metadata = {
+ "categories": [
+ {"id": 1, "name": "tapirus indicus"},
+ {"id": 2, "name": "unknown"},
+ {"id": 3, "name": "unknown rodent"},
+ {"id": 4, "name": "especie desconocida"},
+ {"id": 5, "name": "empty"},
+ ],
+ "images": [
+ {"id": "img1", "file_name": "animals/0036/0072.jpg"},
+ {"id": "img2", "file_name": "animals/0000/0001.jpg"},
+ {"id": "img3", "file_name": "animals/0000/0002.jpg"},
+ {"id": "img4", "file_name": "animals/0000/0003.jpg"},
+ {"id": "img5", "file_name": "animals/0000/0004.jpg"},
+ ],
+ "annotations": [
+ {"image_id": "img1", "category_id": 1},
+ {"image_id": "img2", "category_id": 2},
+ {"image_id": "img3", "category_id": 3},
+ {"image_id": "img4", "category_id": 4},
+ {"image_id": "img5", "category_id": 5},
+ ],
+ }
+ zip_buffer = io.BytesIO()
+ with zipfile.ZipFile(zip_buffer, "w") as zf:
+ zf.writestr("wcs_camera_traps.json", json.dumps(sample_metadata))
- dataset_id = f"{PROJECT}.{BIGQUERY_DATASET}"
- bigquery_client.create_dataset(bigquery.Dataset(dataset_id))
+ mock_response = mock.MagicMock()
+ mock_response.content = zip_buffer.getvalue()
- print(f"bigquery_dataset: {repr(BIGQUERY_DATASET)}")
- yield BIGQUERY_DATASET
+ with mock.patch("requests.get", return_value=mock_response):
+ fn = create_images_metadata_table._ExtractMetadataFn()
+ results = list(fn.process(create_images_metadata_table.METADATA_URL))
- bigquery_client.delete_dataset(dataset_id, delete_contents=True, not_found_ok=True)
+ assert results == [
+ {"category": "tapirus indicus", "file_name": "animals/0036/0072.jpg"}
+ ]
-@pytest.fixture(scope="session")
-def bigquery_table(bigquery_dataset: str) -> str:
- # Create a small test table.
- table_id = f"{PROJECT}.{bigquery_dataset}.{BIGQUERY_TABLE}"
- schema = [
- bigquery.SchemaField("category", "STRING"),
- bigquery.SchemaField("file_name", "STRING"),
- ]
+def test_beam_run_inference_with_gemini_model_handler() -> None:
rows = [
- "alectoris rufa,animals/0059/1810.jpg",
- "equus quagga,animals/0378/0118.jpg",
- "fossa fossana,animals/0620/0242.jpg",
- "human,humans/0379/0877.jpg",
- "human,humans/0640/0467.jpg",
- "lophotibis cristataa,animals/0605/1478.jpg",
- "mazama temama,animals/0532/0525.jpg",
- "odontophorus balliviani,animals/0523/1368.jpg",
- "tayassu pecari,animals/0049/0849.jpg",
- "tayassu pecari,animals/0090/1218.jpg",
+ {"category": "tapirus indicus", "file_name": "animals/0036/0072.jpg"},
+ {"category": "tapirus indicus", "file_name": "animals/0036/0072.jpg"},
]
-
- bigquery_client = bigquery.Client()
- with io.StringIO("\n".join(rows)) as source_file:
- bigquery_client.load_table_from_file(
- source_file,
- table_id,
- job_config=bigquery.LoadJobConfig(
- source_format=bigquery.SourceFormat.CSV,
- schema=schema,
- ),
- ).result()
-
- # The table is deleted when we delete the dataset.
- table = bigquery_client.get_table(table_id)
- print(f"bigquery_table: {repr(BIGQUERY_TABLE)}")
- print(f" table_id: {repr(table_id)}")
- print(f" num_rows: {repr(table.num_rows)}")
- print(f" schema: {repr(table.schema)}")
- yield BIGQUERY_TABLE
-
-
-@pytest.fixture(scope="session")
-def model_endpoint_id() -> str:
- print(f"model_path: {repr(MODEL_PATH)}")
- endpoint_id = deploy_model.create_model_endpoint(PROJECT, REGION, MODEL_ENDPOINT)
-
- try:
- deployed_model_id = deploy_model.deploy_model(
- PROJECT, REGION, MODEL_PATH, MODEL_ENDPOINT, endpoint_id
- )
- except google.api_core.exceptions.NotFound:
- print(
- "NOTE: The permanent model from the testing infrastructure was not "
- "found (probably deleted), but we can still keep going."
- )
- deployed_model_id = "0"
-
- print(f"model_endpoint_id: {repr(endpoint_id)}")
- yield endpoint_id
-
- client = aiplatform.gapic.EndpointServiceClient(
- client_options={"api_endpoint": "us-central1-aiplatform.googleapis.com"}
- )
-
- endpoint_path = client.endpoint_path(PROJECT, REGION, endpoint_id)
- try:
- client.undeploy_model(
- endpoint=endpoint_path, deployed_model_id=deployed_model_id
- ).result()
- except google.api_core.exceptions.NotFound as err:
- if deployed_model_id != "0":
- raise err
- client.delete_endpoint(name=endpoint_path).result()
-
-
-@pytest.fixture(scope="session")
-def cache_dependencies() -> None:
- # The Dataflow staging procedure involves downloading all the requirements and
- # rebuilding everything from scratch.
- # Recent Apache Beam versions include dependencies that require a C++ and Rust compiler
- # and compiling all the dependencies can take a long time.
- # We download the pre-compiled dependencies and then set PIP_NO_DEPS to force
- # pip to not rebuild any indirect dependencies.
- subprocess.run(
- [
- "pip",
- "download",
- "--dest",
- "/tmp/dataflow-requirements-cache",
- "-r",
- "requirements.txt",
- ],
- check=True,
- )
- os.environ["PIP_NO_DEPS"] = "True"
-
-
-def test_create_images_database_table(
- bucket_name: str, bigquery_dataset: str, cache_dependencies: None
-) -> None:
- # The table is deleted when we delete the dataset.
- subprocess.run(
- [
- "python",
- "create_images_metadata_table.py",
- f"--bigquery-dataset={bigquery_dataset}",
- f"--bigquery-table={BIGQUERY_TABLE}_test",
- "--runner=DataflowRunner",
- f"--job_name=wildlife-images-database-{SUFFIX}",
- f"--project={PROJECT}",
- f"--temp_location=gs://{bucket_name}/temp",
- f"--region={REGION}",
- "--worker_machine_type=n1-standard-2",
- ],
- check=True,
+ handler = run_wildlife_inference.GeminiWildlifeModelHandler(
+ project_id=PROJECT
)
-
-def test_train_model(
- bucket_name: str,
- bigquery_dataset: str,
- bigquery_table: str,
- cache_dependencies: None,
-) -> None:
- subprocess.run(
- [
- "python",
- "train_model.py",
- f"--cloud-storage-path=gs://{bucket_name}",
- f"--bigquery-dataset={bigquery_dataset}",
- f"--bigquery-table={bigquery_table}",
- "--ai-platform-name-prefix=", # empty skips the AI Platform operations.
- f"--min-images-per-class={MIN_IMAGES_PER_CLASS}",
- f"--max-images-per-class={MAX_IMAGES_PER_CLASS}",
- "--runner=DataflowRunner",
- f"--project={PROJECT}",
- f"--region={REGION}",
- f"--job_name=wildlife-train-{SUFFIX}",
- f"--temp_location=gs://{bucket_name}/temp",
- "--requirements_file=requirements.txt",
- "--requirements_cache=skip",
- "--pickle_library=cloudpickle",
- ],
- check=True,
- )
-
-
-def test_predict(model_endpoint_id: str) -> None:
- try:
- predictions = predict.run(
- project=PROJECT,
- region=REGION,
- model_endpoint_id=model_endpoint_id,
- image_file="animals/0036/0072.jpg", # tapirus indicus
- )
- assert len(predictions) > 0, f"predictions: {repr(predictions)}"
- except google.api_core.exceptions.FailedPrecondition:
- print(
- "NOTE: The model was not deployed, but it was called "
- "correctly so it's all good ๐"
+ def _verify_predictions(actual: Iterable[dict[str, Any]]) -> None:
+ results = list(actual)
+ assert len(results) == 1
+ pred = results[0]
+ assert pred["ground_truth_category"] == "tapirus indicus"
+ assert pred["taxon_family"] == "Tapiridae"
+ assert pred["animal_count"] >= 1
+ assert pred["is_empty_or_misfire"] is False
+ assert 0.0 <= pred["confidence"] <= 1.0
+
+ with beam.Pipeline() as pipeline:
+ predictions = (
+ pipeline
+ | "CreateRows" >> beam.Create(rows)
+ | "KeyBySpecies" >> beam.WithKeys(lambda row: row["category"])
+ | "SampleOnePerSpecies" >> beam.combiners.Sample.FixedSizePerKey(1)
+ | "DropKey" >> beam.Values()
+ | "Flatten" >> beam.FlatMap(lambda s: s)
+ | "RunGeminiInference" >> RunInference(handler)
)
+ assert_that(predictions, _verify_predictions)
diff --git a/people-and-planet-ai/image-classification/embed_wildlife_image.py b/people-and-planet-ai/image-classification/embed_wildlife_image.py
new file mode 100644
index 00000000000..4d92a6ba49b
--- /dev/null
+++ b/people-and-planet-ai/image-classification/embed_wildlife_image.py
@@ -0,0 +1,94 @@
+#!/usr/bin/env python
+
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# [START googlegenaisdk_wildlife_embed_image]
+import mimetypes
+import sys
+
+from google import genai
+from google.genai import errors
+from google.genai import types
+
+
+def embed_wildlife_image(project_id: str, image_gcs_uri: str) -> None:
+ """Generates a multimodal vector embedding for a camera trap image.
+
+ Uses Gemini Embedding 2 on Gemini Enterprise Agent Platform using the
+ global endpoint to project camera trap imagery into a 768-dimensional
+ multimodal vector space suitable for few-shot rare species discovery and
+ semantic search in BigQuery.
+
+ Args:
+ project_id: Google Cloud project identifier (for example, 'example-project-id').
+ image_gcs_uri: Cloud Storage URI of the camera trap JPEG image.
+ """
+ # gemini-embedding-2 serves on the global endpoint.
+ location = "global"
+ client = genai.Client(vertexai=True, project=project_id, location=location)
+
+ mime_type, _ = mimetypes.guess_type(image_gcs_uri)
+ image_part = types.Part.from_uri(
+ file_uri=image_gcs_uri,
+ mime_type=mime_type or "image/jpeg",
+ )
+ config = types.EmbedContentConfig(output_dimensionality=768)
+
+ try:
+ response = client.models.embed_content(
+ model="gemini-embedding-2",
+ contents=[image_part],
+ config=config,
+ )
+ except errors.APIError as err:
+ print(
+ f"error: failed to embed camera trap image '{image_gcs_uri}' in "
+ f"location '{location}' for project '{project_id}': {err.message}",
+ file=sys.stderr,
+ )
+ else:
+ embeddings = response.embeddings or [response.embedding]
+ values = embeddings[0].values
+ preview = [round(v, 5) for v in values[:3]]
+ print(f"Resource ID: {image_gcs_uri}")
+ print(f"Embedding Dimensions: {len(values)}")
+ print("Model: gemini-embedding-2")
+ print(f"Sample Vector Values: {preview}")
+# [END googlegenaisdk_wildlife_embed_image]
+
+
+if __name__ == "__main__":
+ import argparse
+ import os
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--project",
+ default=os.environ.get("GOOGLE_CLOUD_PROJECT", "example-project-id"),
+ help="Google Cloud project ID.",
+ )
+ parser.add_argument(
+ "--image-gcs-uri",
+ default=(
+ "gs://public-datasets-lila/wcs-unzipped/animals/0036/0072.jpg"
+ ),
+ help="Cloud Storage URI of the camera trap JPEG image.",
+ )
+ args = parser.parse_args()
+
+ embed_wildlife_image(
+ project_id=args.project,
+ image_gcs_uri=args.image_gcs_uri,
+ )
diff --git a/people-and-planet-ai/image-classification/predict.py b/people-and-planet-ai/image-classification/predict.py
deleted file mode 100644
index 86cad8f9049..00000000000
--- a/people-and-planet-ai/image-classification/predict.py
+++ /dev/null
@@ -1,109 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2021 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import annotations
-
-import base64
-
-from google.cloud import aiplatform
-from google.cloud.aiplatform.gapic.schema import predict
-import requests
-
-from train_model import with_retries
-
-
-def run(
- project: str, region: str, model_endpoint_id: str, image_file: str
-) -> list[tuple[str, float]]:
- """Sends an image from the LILA WCS database for prediction.
-
- Args:
- project: Google Cloud Project ID.
- region: Location for AI Platform resources.
- model_endpoint_id: Deployed model endpoint ID.
- image_file: The image file path from LILA.
-
- Returns:
- The predictions as a list of (category, confidence) tuples, sorted by confidence.
- """
- client = aiplatform.gapic.PredictionServiceClient(
- client_options={
- "api_endpoint": "us-central1-prediction-aiplatform.googleapis.com"
- }
- )
-
- base_url = "https://lilablobssc.blob.core.windows.net/wcs-unzipped"
- image_bytes = with_retries(lambda: requests.get(f"{base_url}/{image_file}").content)
-
- response = client.predict(
- endpoint=client.endpoint_path(
- project=project, location=region, endpoint=model_endpoint_id
- ),
- instances=[
- predict.instance.ImageClassificationPredictionInstance(
- content=base64.b64encode(image_bytes).decode("utf-8"),
- ).to_value()
- ],
- parameters=predict.params.ImageClassificationPredictionParams(
- confidence_threshold=0.1,
- max_predictions=5,
- ).to_value(),
- )
-
- prediction = [dict(pred) for pred in response.predictions][0]
- return sorted(
- [
- (category, confidence)
- for category, confidence in zip(
- prediction["displayNames"], prediction["confidences"]
- )
- ],
- reverse=True,
- key=lambda x: x[1],
- )
-
-
-if __name__ == "__main__":
- import argparse
-
- parser = argparse.ArgumentParser()
- parser.add_argument(
- "--project",
- required=True,
- help="Google Cloud Project Id",
- )
- parser.add_argument(
- "--region",
- required=True,
- help="Location for AI Platform resources",
- )
- parser.add_argument(
- "--model-endpoint-id",
- required=True,
- help="Deployed model endpoint ID",
- )
- parser.add_argument(
- "--image-file",
- required=True,
- help="The image file path from LILA",
- )
- args = parser.parse_args()
-
- predictions = run(
- args.project, args.region, args.model_endpoint_id, args.image_file
- )
- for category, confidence in predictions:
- print(f"{category}: {confidence * 100.0 : .2f}% confidence")
diff --git a/people-and-planet-ai/image-classification/requirements-test.txt b/people-and-planet-ai/image-classification/requirements-test.txt
index 21e891cc05a..9a6d2bf61e1 100644
--- a/people-and-planet-ai/image-classification/requirements-test.txt
+++ b/people-and-planet-ai/image-classification/requirements-test.txt
@@ -1,3 +1,2 @@
-google-cloud-storage==2.16.0
pytest-xdist==3.5.0
pytest==9.0.3; python_version >= "3.10"
diff --git a/people-and-planet-ai/image-classification/requirements.txt b/people-and-planet-ai/image-classification/requirements.txt
index ccfe52b49a4..1db18250480 100644
--- a/people-and-planet-ai/image-classification/requirements.txt
+++ b/people-and-planet-ai/image-classification/requirements.txt
@@ -1,3 +1,5 @@
+apache-beam[gcp]==2.77.0
+google-genai==2.29.0
pillow==12.3.0
-apache-beam[gcp]==2.55.1
-google-cloud-aiplatform==1.157.0
+pydantic==2.14.1
+requests==2.34.2
diff --git a/people-and-planet-ai/image-classification/run_wildlife_inference.py b/people-and-planet-ai/image-classification/run_wildlife_inference.py
new file mode 100644
index 00000000000..038dc12041f
--- /dev/null
+++ b/people-and-planet-ai/image-classification/run_wildlife_inference.py
@@ -0,0 +1,255 @@
+#!/usr/bin/env python
+
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# [START dataflow_wildlife_run_inference]
+from __future__ import annotations
+
+from collections.abc import Iterable, Sequence
+import mimetypes
+import sys
+from typing import Any
+
+import apache_beam as beam
+from apache_beam.ml.inference.base import ModelHandler
+from apache_beam.ml.inference.base import RunInference
+from apache_beam.options.pipeline_options import PipelineOptions
+from google import genai
+from google.api_core import exceptions
+from google.genai import errors
+from google.genai import types
+from pydantic import BaseModel
+from pydantic import ValidationError
+
+LILA_WCS_GCS_PREFIX = "gs://public-datasets-lila/wcs-unzipped"
+
+
+class WildlifeObservation(BaseModel):
+ species_scientific_name: str
+ taxon_family: str
+ common_name: str
+ animal_count: int
+ juvenile_present: bool
+ is_empty_or_misfire: bool
+ confidence: float
+
+
+class GeminiWildlifeModelHandler(
+ ModelHandler[dict[str, str], dict[str, Any], genai.Client]
+):
+ """Beam ModelHandler that classifies camera trap images with Gemini."""
+
+ def __init__(self, project_id: str) -> None:
+ self._project_id = project_id
+
+ def load_model(self) -> genai.Client:
+ return genai.Client(
+ vertexai=True, project=self._project_id, location="global"
+ )
+
+ def run_inference(
+ self,
+ batch: Sequence[dict[str, str]],
+ model: genai.Client,
+ inference_args: dict[str, Any] | None = None,
+ ) -> Iterable[dict[str, Any]]:
+ prompt = (
+ "Identify the wildlife species (scientific name, taxonomic family, "
+ "and common name) in this camera trap photo, count the individuals, "
+ "indicate whether juvenile offspring are present, and flag whether "
+ "the frame is empty or a camera misfire."
+ )
+ config = types.GenerateContentConfig(
+ response_mime_type="application/json",
+ response_schema=WildlifeObservation,
+ thinking_config=types.ThinkingConfig(
+ thinking_level=types.ThinkingLevel.LOW,
+ ),
+ )
+ for item in batch:
+ image_gcs_uri = f"{LILA_WCS_GCS_PREFIX}/{item['file_name']}"
+ mime_type, _ = mimetypes.guess_type(image_gcs_uri)
+ image_part = types.Part.from_uri(
+ file_uri=image_gcs_uri,
+ mime_type=mime_type or "image/jpeg",
+ )
+ try:
+ response = model.models.generate_content(
+ model="gemini-3.8-flash",
+ contents=[image_part, prompt],
+ config=config,
+ )
+ obs = WildlifeObservation.model_validate_json(response.text)
+ except (errors.APIError, ValidationError) as err:
+ print(
+ f"error: failed to classify camera trap image "
+ f"'{image_gcs_uri}': {err}",
+ file=sys.stderr,
+ )
+ continue
+
+ yield {
+ "image_gcs_uri": image_gcs_uri,
+ "ground_truth_category": item["category"],
+ "species_scientific_name": obs.species_scientific_name,
+ "taxon_family": obs.taxon_family,
+ "common_name": obs.common_name,
+ "animal_count": obs.animal_count,
+ "juvenile_present": obs.juvenile_present,
+ "is_empty_or_misfire": obs.is_empty_or_misfire,
+ "confidence": obs.confidence,
+ }
+
+
+def run_wildlife_inference(
+ project_id: str,
+ bigquery_dataset: str,
+ source_table: str,
+ predictions_table: str,
+ min_images_per_class: int = 1,
+ max_images_per_class: int = 5,
+ pipeline_options: PipelineOptions | None = None,
+) -> None:
+ """Runs class-balanced camera trap inference with Apache Beam RunInference.
+
+ Samples a balanced set of camera trap records per species from BigQuery,
+ classifies each image in-place from the public LILA Science Cloud Storage
+ bucket using Gemini 3.8 Flash inside Apache Beam RunInference, and writes
+ the structured ecological observations to BigQuery.
+
+ Args:
+ project_id: Google Cloud project identifier (for example, 'example-project-id').
+ bigquery_dataset: BigQuery dataset identifier containing the tables.
+ source_table: Source BigQuery table containing 'category' and 'file_name'.
+ predictions_table: Destination BigQuery table for structured predictions.
+ min_images_per_class: Minimum images required to include a species class.
+ max_images_per_class: Maximum images sampled per species class.
+ pipeline_options: Optional Apache Beam pipeline execution configuration.
+ """
+ source_spec = f"{project_id}:{bigquery_dataset}.{source_table}"
+ dest_spec = f"{project_id}:{bigquery_dataset}.{predictions_table}"
+ output_schema = ",".join(
+ [
+ "image_gcs_uri:STRING",
+ "ground_truth_category:STRING",
+ "species_scientific_name:STRING",
+ "taxon_family:STRING",
+ "common_name:STRING",
+ "animal_count:INTEGER",
+ "juvenile_present:BOOLEAN",
+ "is_empty_or_misfire:BOOLEAN",
+ "confidence:FLOAT",
+ ]
+ )
+ model_handler = GeminiWildlifeModelHandler(project_id=project_id)
+
+ try:
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ (
+ pipeline
+ | "ReadMetadataFromBigQuery"
+ >> beam.io.ReadFromBigQuery(table=source_spec)
+ | "KeyBySpecies" >> beam.WithKeys(lambda row: row["category"])
+ | "SamplePerSpecies"
+ >> beam.combiners.Sample.FixedSizePerKey(max_images_per_class)
+ | "DropKey" >> beam.Values()
+ | "FilterMinImages"
+ >> beam.Filter(
+ lambda sample: len(sample) >= min_images_per_class
+ )
+ | "FlattenSampledRows"
+ >> beam.FlatMap(lambda sample: sample)
+ | "ReshuffleSampledRows" >> beam.Reshuffle()
+ | "RunGeminiInference" >> RunInference(model_handler)
+ | "WritePredictionsToBigQuery"
+ >> beam.io.WriteToBigQuery(
+ table=dest_spec,
+ schema=output_schema,
+ write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
+ create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+ )
+ )
+ except (
+ exceptions.GoogleAPICallError,
+ errors.APIError,
+ RuntimeError,
+ ) as err:
+ print(
+ f"error: failed to run wildlife inference pipeline writing to "
+ f"'{dest_spec}' for project '{project_id}': {err}",
+ file=sys.stderr,
+ )
+ else:
+ print(f"Resource ID: {dest_spec}")
+ print(f"Max Images Per Class: {max_images_per_class}")
+ print(f"Min Images Per Class: {min_images_per_class}")
+ print(f"Source Table: {source_spec}")
+# [END dataflow_wildlife_run_inference]
+
+
+if __name__ == "__main__":
+ import argparse
+ import logging
+ import os
+
+ logging.getLogger().setLevel(logging.INFO)
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--project",
+ default=os.environ.get("GOOGLE_CLOUD_PROJECT", "example-project-id"),
+ help="Google Cloud project ID.",
+ )
+ parser.add_argument(
+ "--bigquery-dataset",
+ required=True,
+ help="BigQuery dataset ID.",
+ )
+ parser.add_argument(
+ "--bigquery-table",
+ default="wildlife_images_metadata",
+ help="Source BigQuery metadata table ID.",
+ )
+ parser.add_argument(
+ "--predictions-table",
+ default="wildlife_predictions",
+ help="Destination BigQuery table ID for structured Gemini predictions.",
+ )
+ parser.add_argument(
+ "--min-images-per-class",
+ type=int,
+ default=1,
+ help="Minimum number of images required per class.",
+ )
+ parser.add_argument(
+ "--max-images-per-class",
+ type=int,
+ default=5,
+ help="Maximum number of images sampled per class.",
+ )
+ args, pipeline_args = parser.parse_known_args()
+
+ options = PipelineOptions(
+ pipeline_args, project=args.project, save_main_session=True
+ )
+ run_wildlife_inference(
+ project_id=args.project,
+ bigquery_dataset=args.bigquery_dataset,
+ source_table=args.bigquery_table,
+ predictions_table=args.predictions_table,
+ min_images_per_class=args.min_images_per_class,
+ max_images_per_class=args.max_images_per_class,
+ pipeline_options=options,
+ )
diff --git a/people-and-planet-ai/image-classification/train_model.py b/people-and-planet-ai/image-classification/train_model.py
deleted file mode 100644
index 4521cb2ed9b..00000000000
--- a/people-and-planet-ai/image-classification/train_model.py
+++ /dev/null
@@ -1,388 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2021 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import annotations
-
-from collections.abc import Callable, Iterable
-from datetime import datetime
-import io
-import logging
-import random
-import time
-from typing import TypeVar
-
-import apache_beam as beam
-from apache_beam.options.pipeline_options import PipelineOptions
-from google.cloud.aiplatform.gapic import DatasetServiceClient
-from google.cloud.aiplatform.gapic import PipelineServiceClient
-from google.cloud.aiplatform.gapic.schema import trainingjob
-from PIL import Image, ImageFile
-import requests
-
-a = TypeVar("a")
-
-
-def run(
- project: str,
- region: str,
- cloud_storage_path: str,
- bigquery_dataset: str,
- bigquery_table: str,
- ai_platform_name_prefix: str,
- min_images_per_class: int,
- max_images_per_class: int,
- budget_milli_node_hours: int,
- pipeline_options: PipelineOptions | None = None,
-) -> None:
- """Creates a balanced dataset and signals AI Platform to train a model.
-
- Args:
- project: Google Cloud Project ID.
- region: Location for AI Platform resources.
- bigquery_dataset: Dataset ID for the images database, the dataset must exist.
- bigquery_table: Table ID for the images database, the table must exist.
- ai_platform_name_prefix: Name prefix for AI Platform resources.
- min_images_per_class: Minimum number of images required per class for training.
- max_images_per_class: Maximum number of images allowed per class for training.
- budget_milli_node_hours: Training budget.
- pipeline_options: PipelineOptions for Apache Beam.
-
- """
- with beam.Pipeline(options=pipeline_options) as pipeline:
- images = (
- pipeline
- | "Read images info"
- >> beam.io.ReadFromBigQuery(dataset=bigquery_dataset, table=bigquery_table)
- | "Key by category" >> beam.WithKeys(lambda x: x["category"])
- | "Random samples"
- >> beam.combiners.Sample.FixedSizePerKey(max_images_per_class)
- | "Remove key" >> beam.Values()
- | "Discard small samples"
- >> beam.Filter(lambda sample: len(sample) >= min_images_per_class)
- | "Flatten elements" >> beam.FlatMap(lambda sample: sample)
- | "Get image" >> beam.FlatMap(get_image, cloud_storage_path)
- )
-
- dataset_csv_filename = f"{cloud_storage_path}/dataset.csv"
- dataset_csv_file = (
- pipeline
- | "Dataset filename" >> beam.Create([dataset_csv_filename])
- | "Write dataset file"
- >> beam.Map(write_dataset_csv_file, images=beam.pvalue.AsIter(images))
- )
-
- if ai_platform_name_prefix:
- timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
- (
- dataset_csv_file
- | "Create dataset"
- >> beam.Map(
- create_dataset,
- project=project,
- region=region,
- dataset_name=f"{ai_platform_name_prefix}_{timestamp}",
- )
- | "Import images" >> beam.MapTuple(import_images_to_dataset)
- | "Train model"
- >> beam.Map(
- train_model,
- project=project,
- region=region,
- model_name=f"{ai_platform_name_prefix}_{timestamp}",
- budget_milli_node_hours=budget_milli_node_hours,
- )
- )
-
-
-def get_image(
- image_info: dict[str, str], cloud_storage_path: str
-) -> Iterable[tuple[str, str]]:
- """Makes sure an image exists in Cloud Storage.
-
- Checks if the image file_name exists in Cloud Storage.
- If it doesn't exist, it downloads it from the LILA WCS dataset.
- If the image can't be downloaded, it is skipped.
-
- Args:
- image_info: dict of {'category', 'file_name'}.
- cloud_storage_path: Cloud Storage path to look for and download images.
-
- Returns:
- A (category, image_gcs_path) tuple.
- """
- import apache_beam as beam
-
- base_url = "https://lilablobssc.blob.core.windows.net/wcs-unzipped"
- category = image_info["category"]
- file_name = image_info["file_name"]
-
- # If the image file does not exist, try downloading it.
- image_gcs_path = f"{cloud_storage_path}/{file_name}"
- logging.info(f"loading image: {image_gcs_path}")
- if not beam.io.gcp.gcsio.GcsIO().exists(image_gcs_path):
- image_url = f"{base_url}/{file_name}"
- logging.info(f"image not found, downloading: {image_gcs_path} [{image_url}]")
- try:
- ImageFile.LOAD_TRUNCATED_IMAGES = True
- image = Image.open(io.BytesIO(url_get(image_url)))
- with beam.io.gcp.gcsio.GcsIO().open(image_gcs_path, "w") as f:
- image.save(f, format="JPEG")
- except Exception as e:
- logging.warning(f"Failed to load image [{image_url}]: {e}")
- return
-
- yield category, image_gcs_path
-
-
-def write_dataset_csv_file(
- dataset_csv_filename: str, images: Iterable[tuple[str, str]]
-) -> str:
- """Writes the dataset image file names and categories in a CSV file.
-
- Each line in the output dataset CSV file is in the format:
- image_gcs_path,category
-
- For more information on the CSV format AI Platform expects:
- https://cloud.google.com/ai-platform-unified/docs/datasets/prepare-image#csv
-
- Args:
- dataset_csv_filename: Cloud Storage path for the output dataset CSV file.
- images: List of (category, image_gcs_path) tuples.
-
- Returns:
- The unchanged dataset_csv_filename.
- """
- import apache_beam as beam
-
- logging.info(f"Writing dataset CSV file: {dataset_csv_filename}")
- with beam.io.gcp.gcsio.GcsIO().open(dataset_csv_filename, "w") as f:
- for category, image_gcs_path in images:
- f.write(f"{image_gcs_path},{category}\n".encode())
- return dataset_csv_filename
-
-
-def create_dataset(
- dataset_csv_filename: str, project: str, region: str, dataset_name: str
-) -> tuple[str, str]:
- """Creates an dataset for AI Platform.
-
- For more information:
- https://cloud.google.com/ai-platform-unified/docs/datasets/create-dataset-api#create-dataset
-
- Args:
- dataset_csv_filename: Cloud Storage path for the dataset CSV file.
- project: Google Cloud Project ID.
- region: Location for AI Platform resources.
- dataset_name: Dataset name.
-
- Returns:
- A (dataset_full_path, dataset_csv_filename) tuple.
- """
- client = DatasetServiceClient(
- client_options={"api_endpoint": "us-central1-aiplatform.googleapis.com"}
- )
-
- response = client.create_dataset(
- parent=f"projects/{project}/locations/{region}",
- dataset={
- "display_name": dataset_name,
- "metadata_schema_uri": "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml",
- },
- )
- logging.info(f"Creating dataset, operation: {response.operation.name}")
- dataset = response.result() # wait until the operation finishes
- logging.info(f"Dataset created:\n{dataset}")
- return dataset.name, dataset_csv_filename
-
-
-def import_images_to_dataset(dataset_full_path: str, dataset_csv_filename: str) -> str:
- """Imports the images from the dataset CSV file into the AI Platform dataset.
-
- For more information:
- https://cloud.google.com/ai-platform-unified/docs/datasets/create-dataset-api#import-data
-
- Args:
- dataset_full_path: The AI Platform dataset full path.
- dataset_csv_filename: Cloud Storage path for the dataset CSV file.
-
- Returns:
- The dataset_full_path.
- """
- client = DatasetServiceClient(
- client_options={"api_endpoint": "us-central1-aiplatform.googleapis.com"}
- )
-
- response = client.import_data(
- name=dataset_full_path,
- import_configs=[
- {
- "gcs_source": {"uris": [dataset_csv_filename]},
- "import_schema_uri": "gs://google-cloud-aiplatform/schema/dataset/ioformat/image_classification_single_label_io_format_1.0.0.yaml",
- }
- ],
- )
- logging.info(f"Importing data into dataset, operation: {response.operation.name}")
- _ = response.result() # wait until the operation finishes
- logging.info(f"Data imported: {dataset_full_path}")
- return dataset_full_path
-
-
-def train_model(
- dataset_full_path: str,
- project: str,
- region: str,
- model_name: str,
- budget_milli_node_hours: int,
-) -> str:
- """Starts a model training job.
-
- For more information:
- https://cloud.google.com/ai-platform-unified/docs/training/automl-api#training_an_automl_model_using_the_api
-
- Args:
- dataset_full_path: The AI Platform dataset full path.
- project: Google Cloud Project ID.
- region: Location for AI Platform resources.
- model_name: Model name.
- budget_milli_node_hours: Training budget.
-
- Returns:
- The training pipeline full path.
- """
- client = PipelineServiceClient(
- client_options={
- "api_endpoint": "us-central1-aiplatform.googleapis.com",
- }
- )
-
- training_pipeline = client.create_training_pipeline(
- parent=f"projects/{project}/locations/{region}",
- training_pipeline={
- "display_name": model_name,
- "input_data_config": {"dataset_id": dataset_full_path.split("/")[-1]},
- "model_to_upload": {"display_name": model_name},
- "training_task_definition": "gs://google-cloud-aiplatform/schema/trainingjob/definition/automl_image_classification_1.0.0.yaml",
- "training_task_inputs": trainingjob.definition.AutoMlImageClassificationInputs(
- model_type="CLOUD",
- budget_milli_node_hours=budget_milli_node_hours,
- ).to_value(),
- },
- )
- logging.info(f"Training model, training pipeline:\n{training_pipeline}")
- return training_pipeline.name
-
-
-def url_get(url: str) -> bytes:
- """Sends an HTTP GET request with retries.
-
- Args:
- url: URL for the request.
-
- Returns:
- The response content bytes.
- """
- logging.info(f"url_get: {url}")
- return with_retries(lambda: requests.get(url).content)
-
-
-def with_retries(f: Callable[[], a], max_attempts: int = 3) -> a:
- """Runs a function with retries, using exponential backoff.
-
- For more information:
- https://developers.google.com/drive/api/v3/handle-errors?hl=pt-pt#exponential-backoff
-
- Args:
- f: A function that doesn't receive any input.
- max_attempts: The maximum number of attempts to run the function.
-
- Returns:
- The return value of `f`, or an Exception if max_attempts was reached.
- """
- for n in range(max_attempts + 1):
- try:
- return f()
- except Exception as e:
- if n < max_attempts:
- logging.warning(f"Got an error, {n+1} of {max_attempts} attempts: {e}")
- time.sleep(2**n + random.random()) # 2^n seconds + random jitter
- else:
- raise e
-
-
-if __name__ == "__main__":
- import argparse
-
- parser = argparse.ArgumentParser()
- parser.add_argument(
- "--cloud-storage-path",
- required=True,
- help="Cloud Storage path to store the AI Platform dataset files.",
- )
- parser.add_argument(
- "--bigquery-dataset",
- required=True,
- help="BigQuery dataset ID for the images database.",
- )
- parser.add_argument(
- "--bigquery-table",
- default="wildlife_images_metadata",
- help="BigQuery table ID for the images database.",
- )
- parser.add_argument(
- "--ai-platform-name-prefix",
- default="wildlife_classifier",
- help="Name prefix for AI Platform resources.",
- )
- parser.add_argument(
- "--min-images-per-class",
- type=int,
- default=50,
- help="Minimum number of images required per class for training.",
- )
- parser.add_argument(
- "--max-images-per-class",
- type=int,
- default=80,
- help="Maximum number of images allowed per class for training.",
- )
- parser.add_argument(
- "--budget-milli-node-hours",
- type=int,
- default=8000,
- help="Training budget, see: https://cloud.google.com/automl/docs/reference/rpc/google.cloud.automl.v1#imageclassificationmodelmetadata",
- )
- args, pipeline_args = parser.parse_known_args()
-
- pipeline_options = PipelineOptions(pipeline_args, save_main_session=True)
- project = pipeline_options.get_all_options().get("project")
- if not project:
- parser.error("please provide a Google Cloud project ID with --project")
- region = pipeline_options.get_all_options().get("region")
- if not region:
- parser.error("please provide a Google Cloud compute region with --region")
-
- run(
- project=project,
- region=region,
- cloud_storage_path=args.cloud_storage_path,
- bigquery_dataset=args.bigquery_dataset,
- bigquery_table=args.bigquery_table,
- ai_platform_name_prefix=args.ai_platform_name_prefix,
- min_images_per_class=args.min_images_per_class,
- max_images_per_class=args.max_images_per_class,
- budget_milli_node_hours=args.budget_milli_node_hours,
- pipeline_options=pipeline_options,
- )
![\"Google](\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\"){kind=logo}
![\"Google](\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-enterprise-logo.png\"){kind=logo}
![\"GitHub](\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\"){kind=icon}