Skip to content

docs: Add deduplication sample for Cloud Scheduler job requests - #14702

Open
camiekim wants to merge 6 commits into
GoogleCloudPlatform:mainfrom
camiekim:master
Open

camiekim wants to merge 6 commits into
GoogleCloudPlatform:mainfrom
camiekim:master

Conversation

@camiekim

Copy link
Copy Markdown
Contributor

Description

Fixes b/574556578

Checklist

Testing

  • I have tested this change on a live environment and verified it works as intended.

Compliance & Style


Post-Approval Actions

  • Please merge this PR for me once it is approved

This script demonstrates how to deduplicate retried Cloud Scheduler HTTP requests using Firestore, ensuring that duplicate job requests are handled appropriately.

Per b/574556578
Add unit tests for deduplicating job requests handling.
@camiekim
camiekim requested review from a team as code owners October 30, 2026 13:13
@snippet-bot

snippet-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Here is the summary of changes.

You are about to add 1 region tag.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@product-auto-label product-auto-label Bot added the samples Issues that are directly related to samples. label Oct 30, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Python snippet and corresponding unit tests demonstrating how to deduplicate retried Cloud Scheduler HTTP requests using Firestore. The feedback highlights a contradiction where manual runs (lacking the schedule time header) are rejected with a 400 error despite docstring claims, and suggests allowing them to run directly. Additionally, the reviewer recommends handling expired locks to prevent crashed jobs from blocking retries due to Firestore's asynchronous TTL deletion, along with adding unit tests to cover both scenarios.

Reordered import statements for consistency.
Added a link to the Cloud Scheduler documentation for deduplication.
@iennae

iennae commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Thanks @camiekim for this sample and for the docs page it supports. I've opened #14772 as a rewrite built on your work. You're credited as co-author, and it keeps the cloudscheduler_deduplicate_job_requests region tag, so your docs page shouldn't need changes.

The main changes:

  • Lease for crashed attempts: If an instance crashes before deleting its IN_PROGRESS record, retries currently get a 409 until TTL cleanup, which can take more than 24 hours. The rewrite claims runs in a Firestore transaction with a lease, so an expired claim can be taken over without the get/set race you pointed out in review.
  • Own directory: It's a functions-framework function in cloud_scheduler/deduplicate_job_requests/ with its own requirements, instead of adding Flask and Firestore to snippets/, which no longer uses Flask after feat(cloud_scheduler): modernize samples and cover documented job tasks #14767.
  • Hashed key: The job name and schedule time are hashed into the document ID instead of validated and normalized. The header stays the same across retries.
  • Tests without credentials: Unit tests use a fake Firestore, and the integration tests are opt-in.

I also tested what the headers look like on each target type, which might be useful for the page:

  • HTTP targets get X-CloudScheduler-JobName, which is the job ID, and X-CloudScheduler-ScheduleTime, which is RFC 3339 with an offset and microseconds. The API reference documents the same headers for App Engine targets.
  • Pub/Sub messages carry no run identifier, only the job's attributes, a message ID, and a publish time. Pub/Sub targets need different guidance, such as Pub/Sub exactly-once delivery.
  • A forced run (gcloud scheduler jobs run) sends the next scheduled time. Header-based deduplication then skips that scheduled run.
  • A groc-schedule job also sent ScheduleTime in testing, but the reference only promises it for unix-cron schedules.

If #14772 works for you, would you be open to closing this PR in favor of it? Feedback on the rewrite is very welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

samples Issues that are directly related to samples.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants