Added runbooks for RCA and replaced GCS URLs with the repsective paths#2108
Added runbooks for RCA and replaced GCS URLs with the repsective paths#2108pulkeet-wq wants to merge 1 commit into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive set of troubleshooting guides and runbooks to assist with import validation, root cause analysis, and resolution of production failures. The reviewer's feedback is highly actionable and focuses on polishing these documents by correcting copy-paste errors (such as incorrect website and import references), removing redundant sections and empty headers or list items, replacing absolute or personal fork links with relative or main repository links, removing internal shortlinks, and avoiding hardcoded local paths in command examples.
|
|
||
| This section highlights the steps that can be taken to perform RCA for the failed production pipelines. | ||
|
|
||
| Step 1: Go to the looker [dashboard](https://lookerstudio.google.com/c/reporting/e88fda74-50c9-46c6-88aa-c84342ceba48/page/eaXdF) and get the latest status of all the pipelines from the dashboard. The probable different states of each pipelines could be as below: |
There was a problem hiding this comment.
According to the style guidelines, please use 'Data Studio' and 'datastudio.google.com' instead of 'Looker Studio' and 'lookerstudio.google.com' when referring to the dashboard branding.
| Step 1: Go to the looker [dashboard](https://lookerstudio.google.com/c/reporting/e88fda74-50c9-46c6-88aa-c84342ceba48/page/eaXdF) and get the latest status of all the pipelines from the dashboard. The probable different states of each pipelines could be as below: | |
| Step 1: Go to the Data Studio [dashboard](https://datastudio.google.com/c/reporting/e88fda74-50c9-46c6-88aa-c84342ceba48/page/eaXdF) and get the latest status of all the pipelines from the dashboard. The probable different states of each pipelines could be as below: |
References
- Use 'Data Studio' and 'datastudio.google.com' instead of 'Looker Studio' and 'lookerstudio.google.com' when referring to the dashboard branding.
| 1. **Clone the GitHub Repository:** | ||
| Clone the Datacommons data repository to your machine: [https://github.com/datacommonsorg/data.git](https://github.com/datacommonsorg/data.git). Once cloned, navigate to the exact same import path we looked at in the bucket. | ||
| 2. **Run the Scripts:** | ||
| Check the [manifest.json](https://github.com/niveditasing/data/blob/e28894cc4701578d43d937bd484d91d8333ffa07/scripts/world_bank/wdi/manifest.json) file in that folder to understand how the job runs, and manually run the scripts and processes it outlines. |
There was a problem hiding this comment.
The link points to a personal fork (niveditasing/data) and a specific commit hash. Please update it to point to the main repository (datacommonsorg/data) on the master branch (or use a relative link) to ensure the link remains valid and maintainable.
| Check the [manifest.json](https://github.com/niveditasing/data/blob/e28894cc4701578d43d937bd484d91d8333ffa07/scripts/world_bank/wdi/manifest.json) file in that folder to understand how the job runs, and manually run the scripts and processes it outlines. | |
| Check the [manifest.json](https://github.com/datacommonsorg/data/blob/master/scripts/world_bank/wdi/manifest.json) file in that folder to understand how the job runs, and manually run the scripts and processes it outlines. |
| * **Nodes\_deleted.mcf:** This file shows you the actual list of records that were deleted. | ||
| * Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators/2026_06_09T04_07_31_200635_07_00/input0/validation/obs_diff_log.csv` |
There was a problem hiding this comment.
There is an inconsistency between the file name described (Nodes_deleted.mcf) and the path provided in the location (obs_diff_log.csv). Please correct either the description or the path to match.
|
|
||
| Now that we know the deletions are real, we need to validate them against the original data source to confirm the source actually removed the data. | ||
|
|
||
| 1. **Locate the Textproto File:** To find out exactly where the data came from, check the `textproto` file for this import. For WorldDevelopmentIndicators, the file path is: `datacommons/import/mcf/manifest/international_stats/EurostatData_Fertility.textproto` |
There was a problem hiding this comment.
This line contains a copy-paste error: it mentions WorldDevelopmentIndicators instead of EurostatData_Fertility for the file path description.
| 1. **Locate the Textproto File:** To find out exactly where the data came from, check the `textproto` file for this import. For WorldDevelopmentIndicators, the file path is: `datacommons/import/mcf/manifest/international_stats/EurostatData_Fertility.textproto` | |
| 1. **Locate the Textproto File:** To find out exactly where the data came from, check the `textproto` file for this import. For EurostatData_Fertility, the file path is: `datacommons/import/mcf/manifest/international_stats/EurostatData_Fertility.textproto` |
| 1. **Locate the Textproto File:** To find out exactly where the data came from, check the `textproto` file for this import. For WorldDevelopmentIndicators, the file path is: `datacommons/import/mcf/manifest/international_stats/EurostatData_Fertility.textproto` | ||
| * You can search for this file using Google Code Search: [EurostatData\_Fertility.textproto Link](https://source.corp.google.com/piper///depot/google3/datacommons/import/mcf/manifest/international_stats/EurostatData_Fertility.textproto;l=2?q=eurostatdata&sq=package:piper%20file:%2F%2Fdepot%2Fgoogle3%20-file:google3%2Fexperimental) | ||
| 2. **Find the Source URL:** Open the `textproto` file and look for the line that says: `provenance_url: "https://ec.europa.eu/eurostat/databrowser/view/demo_r_find3/default/table?lang=en"` This URL tells us exactly where the data is downloaded from. | ||
| 3. **Navigate the Source Website:** Open that source URL. On the World Bank website, click on the **Explore Data** option, and then click on **Access Data**. This will allow you to see their entire dataset. |
There was a problem hiding this comment.
This line contains a copy-paste error: it refers to the 'World Bank website' instead of the 'Eurostat website'.
| 3. **Navigate the Source Website:** Open that source URL. On the World Bank website, click on the **Explore Data** option, and then click on **Access Data**. This will allow you to see their entire dataset. | |
| 3. **Navigate the Source Website:** Open that source URL. On the Eurostat website, navigate to the data explorer to see their entire dataset. |
| ## | ||
|
|
||
| ## | ||
|
|
There was a problem hiding this comment.
There are empty headers (## ) on lines 68 and 70. Please remove them to clean up the document structure.
| | When the code fails due to any random temporary issue | | ||
| | :---- | | ||
|
|
||
| 1. |
There was a problem hiding this comment.
The list item 1. on line 117 is empty and incomplete. Please complete the step or remove the list item.
|
|
||
| 1. **`Check_goldens_output_csv`**: Verifies the final output data against an established baseline. | ||
| 2. **`Check_goldens_summary_report`**: Validates structural metrics (like number of places and dates) against a summary baseline. | ||
| 3. More regarding golden checks can be seen : [github link](https://github.com/datacommonsorg/data/blob/master/tools/import_validation/Validations.md) |
There was a problem hiding this comment.
Since Validations.md is in the parent directory of this file, please use a relative link (../Validations.md) instead of an absolute GitHub URL. This makes the documentation more portable and robust.
| 3. More regarding golden checks can be seen : [github link](https://github.com/datacommonsorg/data/blob/master/tools/import_validation/Validations.md) | |
| 3. More regarding golden checks can be seen : [github link](../Validations.md) |
|
|
||
| Step 6: Based on RCA, plan the error resolution. Discuss issues with the CORE TEAM on a need basis. Also, prepare a document capturing the issues and the next actions. [Template](https://docs.google.com/document/d/1PyBmcN-1C_p9y-ML93eaBFyspg1XD5zwkT7EqeTsQsI/edit?resourcekey=0-B5pj3_KBQDpdfNs6AGllXQ&tab=t.0#heading=h.ne6ee5rvmv4h) | ||
|
|
||
| Step 7: Change the code (if required) and raise [CL/PR](http://cl/PR) as appropriate. |
There was a problem hiding this comment.
The link http://cl/PR is an internal Google shortlink and is inaccessible to public contributors. Please use a standard term like PR or Pull Request without the internal link.
| Step 7: Change the code (if required) and raise [CL/PR](http://cl/PR) as appropriate. | |
| Step 7: Change the code (if required) and raise a PR as appropriate. |
| Step 8: Once the code is merged with production, force execute the production pipelines using the command below: | ||
|
|
||
| ``` | ||
| ~/Desktop/DataCommons/data/import-automation/executor/run_import.sh -p <project name> -d dc-test-executor-$USER -cloud -a <docker artifact> <path to manifest file> -batch |
There was a problem hiding this comment.
The command uses a hardcoded local path ~/Desktop/DataCommons/data/.... Please use a relative path from the repository root (e.g., import-automation/executor/run_import.sh) to make the runbook generic and usable for all developers.
| ~/Desktop/DataCommons/data/import-automation/executor/run_import.sh -p <project name> -d dc-test-executor-$USER -cloud -a <docker artifact> <path to manifest file> -batch | |
| import-automation/executor/run_import.sh -p <project name> -d dc-test-executor-$USER -cloud -a <docker artifact> <path to manifest file> -batch |
|
|
||
| ### **Step 3.1: Generate the Summary Report Golden File** | ||
|
|
||
| This step tracks metadata properties like Statistical Variables (StatVars), the number of places, and date ranges to ensure future runs don't accidentally drop the entire series. |
There was a problem hiding this comment.
L ets add some instructions on what columns to include in the golden, such as, include columns that are unkely to change across data refreshes, such as statvars, number of places, min date, unit, scaling factor, observation period and measurement method. If some statvars have a fixed range of values for min and max, create a separate golden for those with the min, max values.
|
|
||
| ### **Step 3.2: Generate the Output Data Golden File (only use if needed)** | ||
|
|
||
| This step targets critical combinations of highly utilized StatVars and top geographical regions to ensure key data points are always preserved. |
There was a problem hiding this comment.
Create separate golden output s for prominent places in observationAbout and statvars
| This configuration does two things: | ||
|
|
||
| * Overrides the default deletion tolerance rule (`check_deleted_records_percent`) to a threshold as per history deletions & current deletions should not be more than **0.1%**. | ||
| * Activates the two required golden check rules pointing to the files you created in Section 3\. |
There was a problem hiding this comment.
All instead of two as there may more than two
|
|
||
| Due to recurring data deletion failures, we protect our imports using **Golden Set Validations (`GOLDENS_CHECK`)**. This guide helps understand how to generate "golden files" (baselines of expected data) and configure the automated validation system to prevent accidental data regressions. | ||
|
|
||
| Currently, to mitigate issues related to data loss, we implement two primary validations supported by specific golden baselines: |
There was a problem hiding this comment.
Add 'for each of the script output csv files listed in the manifest
| @@ -0,0 +1,113 @@ | |||
| # **Validating Deletions for the WorldDevelopmentIndicators Import** | |||
|
|
|||
| To address frequent failures in the **WorldDevelopmentIndicators** import caused by source data deletions—occurring 4–5 times already this year—you must validate these deletions. The resolution process involves : examining the cloud job logs, replicating the import run on a local machine, and identifying the cause of record deletions to implement a permanent fix. | |||
There was a problem hiding this comment.
Import specific instructions can go into an md file in the import folder. Let's keep the instructions in the tools folder generic.
| * Here is the job I looked at for this import: Cloud Batch Job: `worlddevelopmentindicators-1781002802` (Project: `datcom-import-automation-prod`, Region: `us-central1`) | ||
| 2. **Look for Errors in the Logs:** | ||
| Even if the job status says "Succeeded," it might still have errors. Look through the logs (you can use the errors filter to jump straight to them). In my case, the job failed some validation checks because of deletions. | ||
| * **Error Message:** Found 0.87% deleted records, which is over the threshold of 0%. |
There was a problem hiding this comment.
To check if an import run had deletions, let's also add instructions to use the Import Summary table in spanner dashboard as well as the validation_summary file in the dated GCS folder for that run.
| 4. **Review the Validation Files:** | ||
| Go into the input0/validation/ folder. Here, you want to look at two specific files: | ||
| * **validation\_output.csv:** This tells you the exact reason the checks failed. In my case, 3 checks passed, but the check\_deleted\_records\_percent failed because 3,908 records were deleted. (See table below) | ||
| * Location: Project: `datcom-204919`, Bucket: `datcom-prod-imports`, Path: `scripts/world_bank/wdi/WorldDevelopmentIndicators/2026_06_09T04_07_31_200635_07_00/input0/validation/validation_output.csv` |
There was a problem hiding this comment.
Let's move references to google specific projects to something like GCS_PROJECT or GCS_BUCKET. These variables can be in a config file that agent can access too.
|
|
||
| ## **Run the Import Locally** | ||
|
|
||
| Now, we need to run the whole import process on our own computer. This helps us confirm if the deletions are really happening, or if it was just a weird glitch in the cloud job. |
There was a problem hiding this comment.
rename 'weird glitch' to 'transient failure`
| 1. **Clone the GitHub Repository:** | ||
| Clone the Datacommons data repository to your machine: [https://github.com/datacommonsorg/data.git](https://github.com/datacommonsorg/data.git). Once cloned, navigate to the exact same import path we looked at in the bucket. | ||
| 2. **Run the Scripts:** | ||
| Check the [manifest.json](https://github.com/niveditasing/data/blob/e28894cc4701578d43d937bd484d91d8333ffa07/scripts/world_bank/wdi/manifest.json) file in that folder to understand how the job runs, and manually run the scripts and processes it outlines. |
There was a problem hiding this comment.
pls remove references to local folders with usernames
| 1. **Clone the GitHub Repository:** | ||
| Clone the Datacommons data repository to your machine: [https://github.com/datacommonsorg/data.git](https://github.com/datacommonsorg/data.git). Once cloned, navigate to the exact same import path we looked at in the bucket. | ||
| 2. **Run the Scripts:** | ||
| Check the [manifest.json](https://github.com/niveditasing/data/blob/e28894cc4701578d43d937bd484d91d8333ffa07/scripts/world_bank/wdi/manifest.json) file in that folder to understand how the job runs, and manually run the scripts and processes it outlines. |
There was a problem hiding this comment.
Manual run will require environment setup.
you can also use the run_import.sh to run the script in the manifest.json locally so the python environment is setup before python scripts are run.
2 participants
Added support run book to enable auto debugging of failures via LLM tools. Currently this runbook talks about following scenarios:
Handling data source side issues
Handling Minot deletions at Data source
Handling major deletions at Data source
Adding golden checks in the import
RCA runbook