Implement soft delete for Fuzzer, Job, FuzzerJob, and FuzzTarget and discard unprocessable messages

[javanlacerda]

Problem

1. Pub/Sub Poisoned Task Messages: When tasks in Pub/Sub reference deleted or non-existent Job, Fuzzer, FuzzerJob, or FuzzTarget entities, worker exceptions or unhandled missing lookups prevent clean task completion. Without message acknowledgment (ack), Pub/Sub redelivers the message upon deadline expiration indefinitely, wedging task queues.
2. Hard Deletion & Dangling References: Hard-deleting entities leaves orphaned mapping records and references in Datastore, causing errors during cron scheduling (schedule_fuzz.py), batching (batch_fuzzer_jobs.py), and UI rendering.

---

Solution

1. Soft Deletion Models & Web Handlers

• data_types.py:
  • Added deleted = ndb.BooleanProperty(default=False) to Fuzzer, Job, FuzzerJob, and FuzzTarget.
• fuzzers.py:
  • Replaced hard deletion with soft deletion (fuzzer.deleted = True).
  • Filtered out soft-deleted fuzzers from /fuzzers UI rendering and API endpoints.
  • Added checks to reject requests targeting already-deleted fuzzers in DeleteHandler and LogHandler.
• jobs.py:
  • Replaced hard deletion with soft deletion (job.deleted = True).
  • Added un-delete logic in UpdateJob.post (job.deleted = False) when recreating/updating jobs.
  • Filtered out soft-deleted jobs and fuzzers from pagination results (get_results), _job_to_dict, and Handler.get.
  • Added checks in DeleteJobHandler and GetEnvironmentHandler to handle soft-deleted jobs cleanly.

2. Worker Task Validation & Discard Logic

• commands.py:
  • Added validation in process_command_impl to check that Job, Fuzzer, and FuzzerJob mappings exist and are not marked deleted.
  • Returning None allows the worker's task lease loop to cleanly acknowledge and discard poisoned messages rather than crashing and triggering Pub/Sub redelivery.
• setup.py:
  • Validated that Fuzzer exists and is not deleted in preprocess_update_fuzzer_and_data_bundles.
• corpus_pruning_task.py:
  • Validated FuzzTarget in utask_preprocess and passed the resolved entity directly to _utask_preprocess to eliminate redundant Datastore queries.
  • Cleanly logged and returned None when encountering missing or deleted fuzz targets.

3. Cron Schedulers & Datastore Helpers

• cleanup.py:
  • Added cleanup_invalid_fuzzer_jobs() to purge orphaned or deleted FuzzerJob mapping entities from Datastore.
  • Soft-deleted unused FuzzTarget entities (fuzz_target.deleted = True) in cleanup_unused_fuzz_targets_and_jobs().
• schedule_fuzz.py:
  • Filtered out soft-deleted Job, Fuzzer, and FuzzerJob entities during candidate generation.
  • Lowered log level from error to warning when skipping deleted or missing entities during periodic scheduling.
• batch_fuzzer_jobs.py:
  • Filtered out soft-deleted FuzzerJob records before batching.
• data_handler.py:
  • Excluded soft-deleted jobs in get_all_job_type_names() and soft-deleted fuzz targets in get_fuzz_target().
• fuzzer_selection.py:
  • Cleaned up soft-deleted job mappings in update_mappings_for_fuzzer() and ignored soft-deleted targets in get_fuzz_target_weights().

---

Testing

• Added unit tests covering soft deletion, filtering, cleanup, and poison task handling across:
  • src/clusterfuzz/_internal/tests/appengine/handlers/cron/cleanup_test.py (CleanupInvalidFuzzerJobsTest)
  • src/clusterfuzz/_internal/tests/appengine/handlers/cron/schedule_fuzz_test.py (deleted entities in OSSFuzz and Chrome schedulers)
  • src/clusterfuzz/_internal/tests/appengine/handlers/fuzzers_test.py (DeleteHandlerTest)
  • src/clusterfuzz/_internal/tests/appengine/handlers/jobs_test.py (DeleteJobHandlerTest)
  • src/clusterfuzz/_internal/tests/core/bot/tasks/commands_test.py (ProcessCommandImplPoisonedTest)
  • src/clusterfuzz/_internal/tests/core/bot/tasks/utasks/corpus_pruning_task_test.py (CorpusPruningPreprocessPoisonedTest)
  • src/clusterfuzz/_internal/tests/core/datastore/data_handler_test.py (GetFuzzTargetAndJobsTest)
  • src/clusterfuzz/_internal/tests/core/fuzzing/fuzzer_selection_test.py (test_soft_deleted_job)
• Verified code formatting and linting:

  pipenv run python butler.py format
  pipenv run python butler.py lint