Clean up invalid FuzzerJob entities in cleanup task

[javanlacerda]

fix b/538683766

Problem

When Job or Fuzzer entities are deleted, orphaned FuzzerJob mappings can linger in Datastore because the cleanup cron task does not detect and prune them.

Solution

• Added cleanup_invalid_fuzzer_jobs() in src/clusterfuzz/_internal/cron/cleanup.py to delete FuzzerJob entities referencing non-existent Fuzzer or Job entities.
• Called cleanup_invalid_fuzzer_jobs() in cleanup.py's main().
• Added unit tests in src/clusterfuzz/_internal/tests/appengine/handlers/cron/cleanup_test.py.

Testing

python butler.py py_unittest -t appengine -p cleanup_test.py
python butler.py lint

[ViniciustCosta]

It should be ok to add this in cleanup, but I think this deletion is already done in fuzzer_selection.py here: https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/fuzzing/fuzzer_selection.py#L32,

which should be called by the update/delete handlers (ex: https://github.com/google/clusterfuzz/blob/master/src/appengine/handlers/fuzzers.py#L303)

[dylanjew]

It should be ok to add this in cleanup, but I think this deletion is already done in fuzzer_selection.py here: https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/fuzzing/fuzzer_selection.py#L32,

which should be called by the update/delete handlers (ex: https://github.com/google/clusterfuzz/blob/master/src/appengine/handlers/fuzzers.py#L303)

I think there's still a possibility that we're left with dangling FuzzerJobs in the Datastore. If you remove a fuzzer from a job via the /jobs page, we only look for existing fuzzers with the Job.name in Fuzzer.jobs when we go to delete the associated FuzzerJobs here

This means that if there exists a FuzzerJob that doesn't have a corresponding entry in Fuzzer.jobs, we will never delete it. It's not supposed to happen, but it's possible there's a bug somewhere that introduces the invalid config.

We don't validate the Fuzzer actually exists before we schedule FuzzerJobs, so having this repair cron would allow us to recover from these scenarios.

[dylanjew]

It should be ok to add this in cleanup, but I think this deletion is already done in fuzzer_selection.py here: https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/fuzzing/fuzzer_selection.py#L32,
which should be called by the update/delete handlers (ex: https://github.com/google/clusterfuzz/blob/master/src/appengine/handlers/fuzzers.py#L303)

I think there's still a possibility that we're left with dangling FuzzerJobs in the Datastore. If you remove a fuzzer from a job via the /jobs page, we only look for existing fuzzers with the Job.name in Fuzzer.jobs when we go to delete the associated FuzzerJobs here

This means that if there exists a FuzzerJob that doesn't have a corresponding entry in Fuzzer.jobs, we will never delete it. It's not supposed to happen, but it's possible there's a bug somewhere that introduces the invalid config.

We don't validate the Fuzzer actually exists before we schedule FuzzerJobs, so having this repair cron would allow us to recover from these scenarios.

I wonder if we should expand this repair cron to just keep the Fuzzer.jobs list in sync with the FuzzerJobs, rather than only deleting dangling/invalid FuzzerJobs?

[ViniciustCosta]

I wonder if we should expand this repair cron to just keep the Fuzzer.jobs list in sync with the FuzzerJobs, rather than only deleting dangling/invalid FuzzerJobs?

I agree that this might be a cleaner solution