A real-world data wrangling project that gathers, assesses, cleans, and analyses two independent global datasets to examine how a country's development level relates to its reported mental health burden.
Completed as the Data Wrangling project for the Udacity Data Analyst Nanodegree.
Mental health is a major but underexamined component of human wellbeing, and its relationship to national development is not obvious. Does development reduce mental health burden — or simply make it more visible through better diagnosis and reporting?
This project merges country-level mental health burden estimates with the UN Human Development Index across 204 countries and 20 years (2000–2019), then measures where and how consistently the two diverge.
Research question: Which regions show the greatest mismatch between human development level and mental health burden — and has that mismatch changed between 2000 and 2019?
| # | Dataset | Source | Gathering method |
|---|---|---|---|
| 1 | Mental health DALYs (depressive, anxiety, and bipolar disorders) | IHME Global Burden of Disease 2023 | CSV downloaded via an approved data request to the GBD Results Tool |
| 2 | Human Development Index | UNDP, via Our World in Data | Programmatic download with urllib.request from a stable raw.githubusercontent.com URL, plus a manually downloaded supplement for 2018–2019 |
Note on Dataset 2: the OWID GitHub file only covers data through 2017. Every programmatic route for 2018–2019 (OWID's API, the UNDP portal, the World Bank API, GitHub mirrors) returned 403 errors, so those two years were downloaded manually from the OWID grapher page and concatenated with pd.concat(). The manual step is a documented workaround for one access limitation, not the primary gathering method.
DALYs (Disability-Adjusted Life Years) are used as the burden measure because they combine years of life lost to premature death with years lived with disability, and IHME applies a consistent modelling methodology across all 204 countries — making the rates comparable across regions.
Four issues were identified, each confirmed with both a visual and a programmatic check.
Quality issues
- Redundant ID columns. The IHME file ships 18 columns, seven of which are ID/name pairs (
cause_id/cause_name,measure_id/measure_name, etc.). Agroupby().nunique()check confirmed every ID maps one-to-one to its name column, carrying no additional information. Theupper/loweruncertainty bounds were also flagged, since the analysis uses point estimates only. - Incomplete HDI year coverage. Some countries are missing entire years — not as nulls, but as absent rows. A full country-year
MultiIndexgrid was built and differenced against the actual index to quantify the gaps. These reflect genuine limits in UNDP collection (e.g. South Sudan has no estimates before 2010, as it only became independent in 2011), so no imputation is appropriate.
Tidiness issues
- Long-format burden data. Each country-year occupies three rows, one per disorder. The observational unit is the country-year, so the three DALY rates belong in columns. A count check confirmed the three-row pattern holds for every country-year without exception, which makes a clean pivot possible with no introduced
NaNs. - Two tables, one observational unit. Burden and HDI describe the same country-year but live in separate tables. The obstacle to merging is inconsistent country naming — IHME uses formal UN names (
Bolivia (Plurinational State of),United States of America) while OWID uses common names — so a naive merge would silently drop mismatched rows.
- Dropped the seven ID columns and the two uncertainty-bound columns.
- Pivoted the burden data from long to wide with
pivot_table(), one row per country-year, then summed the three disorder rates into a singlemental_health_burdencolumn (GBD DALYs are additive across causes). - Standardised country names using an explicit 19-pair mapping dictionary rather than fuzzy matching, so every mapping decision stays transparent and no silent wrong matches occur.
- Filtered to 2000–2019 before merging, so year scoping is a deliberate step rather than a side effect of the join.
- Inner-joined the two tables on
country+Year. This drops 14 countries with no UNDP coverage (micro-territories and data-sparse states such as Somalia and North Korea); none belong to the seven World Bank regions used in the analysis, so regional averages are unaffected. - Added a World Bank region label to each country for grouped analysis.
The merge was validated for row count, duplicate country-year keys, and residual nulls.
Final dataset: country, Year, region, mental_health_burden, hdi — roughly 190 countries across 2000–2019.
The analysis runs in two parts.
Part 1 — 2019 baseline snapshot. Regional average HDI plotted against regional average burden.
North America sits at the top right with both the highest development and the highest reported burden. But the pattern breaks elsewhere: East Asia & Pacific combines relatively high development with the lowest reported burden globally, while Sub-Saharan Africa has the lowest HDI yet reports a higher burden than both South Asia and East Asia & Pacific. Development level alone does not predict reported burden.
Part 2 — Mismatch score, 2000–2019. Because HDI (0–1) and burden (thousands of DALYs) are on incomparable scales, both are converted to within-year percentile ranks:
mismatch = hdi_rank − (1 − burden_rank)
≈ 0— development rank and burden rank line up> 0— better mental health than the development level would predict< 0— heavier burden than the development level would predict
Finding: regions split cleanly into two groups — East Asia & Pacific, South Asia, and Sub-Saharan Africa hold persistently negative scores; North America, Europe & Central Asia, Latin America & Caribbean, and the Middle East & North Africa hold positive ones. Critically, none of the regional lines cross or converge across the two decades — they move in near-parallel. The mismatch has not changed. Development does not appear to shift this relationship over time; the gap looks structurally locked in, likely driven by long-standing differences in healthcare infrastructure, cultural reporting practices, and baseline stressors.
.
├── mental_health_hdi_project.ipynb # Main notebook: gather → assess → clean → analyse
├── datasets/
│ ├── IHME-GBD_2023_DATA-*.csv # Not committed — see "Data access" below
│ └── hdi_2018_2019.csv # Manual OWID supplement (CC BY)
├── data/
│ ├── raw/ # Untouched copies, saved before cleaning (not committed)
│ └── cleaned/ # Cleaned tables (not committed) and figures
├── .gitignore
└── README.md
data/raw/ and data/cleaned/ are created by the notebook on first run.
The IHME source file is not included in this repository. The IHME Free-of-Charge Non-Commercial User Agreement prohibits redistributing GBD data sets from user-hosted locations without written permission from the University of Washington; links to IHME's own download facilities are permitted instead. Intermediate and merged CSVs are excluded for the same reason, since they carry the underlying IHME estimates row-by-row. The figures are regional aggregates and are included.
To reproduce the analysis, request the same extract from the GBD Results Tool using these parameters:
| Field | Selection |
|---|---|
| Measure | DALYs (Disability-Adjusted Life Years) |
| Metric | Rate (per 100,000) |
| Cause | Depressive disorders, Anxiety disorders, Bipolar disorder |
| Location | All 204 countries and territories |
| Age | All ages |
| Sex | Both |
| Year | 2000–2019 |
Save the resulting CSV into datasets/ and update the filename in the first code cell if it differs. The HDI data is fetched programmatically at runtime, with the 2018–2019 supplement already included in the repo.
Commercial use of IHME data requires a separate licence — contact IHME for details.
Requirements: Python 3.8+
pip install pandas numpy matplotlib seaborn jupyterRun:
git clone https://github.com/malakbattat/Real-World-Data-Wrangling-with-Python-DAND.git
cd Real-World-Data-Wrangling-with-Python-DAND
jupyter notebook mental_health_hdi_project.ipynbRun the cells top to bottom. The HDI download runs over the network; the IHME file and the 2018–2019 supplement are read from datasets/.
- Detection bias. Higher reported burden in wealthy nations likely reflects better diagnostic infrastructure, not necessarily worse mental health. The mismatch score measures reported burden, not true prevalence.
- Aggregation tradeoff. Summing the three disorders into one score masks differences between conditions — bipolar and anxiety burdens may behave quite differently.
- Uneven regional sizes. Europe & Central Asia contains 40+ countries; North America contains 2. Regional averages therefore carry very different amounts of underlying variation, and smaller regions are more sensitive to which countries have complete data.
- Coverage loss. 14 countries with no UNDP coverage are excluded by the inner join.
- The unmapped
Otherregion category is excluded from all visualisations.
- Integrate mental health stigma datasets to test whether they statistically explain low reporting in high-HDI regions like East Asia.
- Add socio-economic and environmental stressor data to probe the unexpectedly high baseline burden in lower-HDI regions.
- Disaggregate the combined burden variable back into depression, anxiety, and bipolar disorder to check whether specific conditions drive the persistent regional mismatches differently.
- Institute for Health Metrics and Evaluation (IHME). Global Burden of Disease Study 2023. Seattle, WA: IHME, University of Washington.
- United Nations Development Programme. Human Development Index, via Our World in Data.