{ "cells": [ { "cell_type": "markdown", "id": "415d830e", "metadata": { "editable": true, "id": "415d830e", "slideshow": { "slide_type": "" }, "tags": [ "remove-cell" ] }, "source": [ "**Setup**\n", "\n", "**Colab**: run the next cell.\n", "**Binder**: nothing to do — the environment comes from `requirements.txt`.\n", "**Local**: `uv sync`.\n", "\n", "The `liberty` basemap needs no account. (Other leafmap styles ask for a MapTiler key.)" ] }, { "cell_type": "code", "execution_count": 2, "id": "7e6514bb", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "editable": true, "id": "7e6514bb", "outputId": "7f0cd5ef-8faa-4fa9-a95f-fa95d055152c", "slideshow": { "slide_type": "" }, "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# Colab needs these. Binder and a local `uv sync` already have them.\n", "import sys\n", "\n", "if \"google.colab\" in sys.modules:\n", " %pip install -q \"leafmap[maplibre]\" geopandas wbgapi mapclassify plotnine \"ipyvuetify<3\"" ] }, { "cell_type": "code", "execution_count": null, "id": "f4a1be74", "metadata": { "editable": true, "id": "c8d98073", "slideshow": { "slide_type": "" }, "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "import os\n", "from pathlib import Path\n", "\n", "import geopandas as gpd\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "import wbgapi as wb\n", "import leafmap.maplibregl as leafmap\n", "from plotnine import (ggplot, aes, geom_point, geom_line, geom_col, labs,\n", " coord_cartesian, facet_wrap, ggtitle, theme_bw, theme)\n", "from shapely.geometry import Polygon\n", "\n", "wb.db = 2 # World Development Indicators\n", "\n", "# Repo layout locally, plain folders on Colab.\n", "OUT = Path(\"../outputs\") if Path.cwd().name == \"scripts\" else Path(\"outputs\")\n", "(OUT / \"html\").mkdir(parents=True, exist_ok=True)\n", "(OUT / \"figures\").mkdir(parents=True, exist_ok=True)" ] }, { "cell_type": "markdown", "id": "b8e7ac05", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ ":::{warning}\n", "This chapter is a work in progress.\n", ":::" ] }, { "cell_type": "markdown", "id": "09a72a78", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "# Lets explore some spatial insight on sustainable urban development and mobility infrastructure across geographical scales\n", "\n", "* **Authors**: Sujit Sikder (IOER) & Sukanto Das (ZALF)\n", "* **Topics**: Urbanisation, Transport Emissions, World Bank Open Data, Interactive Maps, Reproducible Research\n", "* **Badges**: ![Interactive](https://img.shields.io/badge/Type-Interactive_Code-blue?style=flat-square) ![Colab](https://img.shields.io/badge/Colab-Tested-yellow?style=flat-square&logo=googlecolab&logoColor=white) ![Jupyter](https://img.shields.io/badge/Jupyter4NFDI-Ready-orange?style=flat-square&logo=jupyter)" ] }, { "cell_type": "markdown", "id": "c53a3a46", "metadata": { "editable": true, "slideshow": { "slide_type": "" } }, "source": [ "## Abstract\n", "We explore multi-level spatial insights into urban development and sustainable urban mobility across geographical scales. We adopt curated data from global open data sources and established indicator systems, applying comparative statistical analysis across continents to identify spatial patterns, disparities, and development trajectories. A data-harvesting workflow is demonstrated following reproducibility principles using R and Python, integrating statistical analysis, geospatial processing, and visualization. At the lowest possible spatial resolution, urban grid-level mobility indicators derived from open datasets are integrated to investigate the spatio-temporal dynamics of local public transit infrastructure and services across places in Germany. The workflow demonstrates how heterogeneous FAIR and open spatial data can be integrated and analysed to support evidence-based understanding of converging challenges related to climate change and decarbonization, while motivating pathways towards sustainability transformation through technological innovation, open science, and datafication. Further research remains open to exploring qualitative dimensions at different spatial scales and places.\n" ] }, { "cell_type": "markdown", "id": "5a2ead53", "metadata": { "editable": true, "id": "5a2ead53", "slideshow": { "slide_type": "" } }, "source": [ "## Introduction: \n", "\n", "Exploring on data source and scale, this chapter has two parts: \n", "\n", "\n", "### Part one: Global urban development and carbon emission from transport sector - in World Bank Indicators\n", "### Part two: Urban public transit service frequency in Germany - from ioerDATA repository " ] }, { "cell_type": "markdown", "id": "86be5c96", "metadata": {}, "source": [ "### Part one:\n", "\n", "we harvest indicators from the [World Bank API](https://data.worldbank.org/indicator) and look at them two ways: as a time series for Germany against the world regions, and as a globe for every country that reports.\n", "\n", "We will:\n", "\n", "- pull urbanisation and transport-CO₂ indicators live from an open API,\n", "- check that the indicator codes are still alive before we rely on them,\n", "- rebuild a retired indicator from its replacement,\n", "- compare Germany with the seven world regions since 1975,\n", "- map the latest reported value for around 170 countries on an interactive globe,\n", "- and export standalone maps that anyone can open in a browser.\n", "\n", "The aim is to show how an open API can be **queried, verified and reused** — with no manual downloads, and nothing that goes stale on disk." ] }, { "cell_type": "markdown", "id": "0eeb49c0", "metadata": { "id": "0eeb49c0" }, "source": [ "### Why urbanisation and transport together?\n", "\n", "More than half the world now lives in cities, and transport is one of the largest sources of urban CO₂. But the two do not move in step everywhere.\n", "\n", "A country can urbanise quickly while transport emissions stay small, or run a high transport share with barely any urban growth. What that relationship looks like depends on where you are — which is exactly what a global indicator can show.\n", "\n", "We use five World Bank indicators: three on urbanisation, two on emissions." ] }, { "cell_type": "markdown", "id": "6c9b409d", "metadata": { "id": "6c9b409d" }, "source": [ "### The indicators — and whether they still exist" ] }, { "cell_type": "code", "execution_count": null, "id": "47f0b837", "metadata": { "id": "47f0b837" }, "outputs": [], "source": [ "# Urban population (% of total) : SP.URB.TOTL.IN.ZS\n", "# Urban population growth (annual %) : SP.URB.GROW\n", "# Population in largest city (% of urban pop.) : EN.URB.LCTY.UR.ZS\n", "# CO2 from Transport (Mt CO2e) : EN.GHG.CO2.TR.MT.CE.AR5\n", "# CO2 total, excl. LULUCF (Mt CO2e) : EN.GHG.CO2.MT.CE.AR5\n", "#\n", "# EN.CO2.TRAN.ZS (transport CO2 as a %) is RETIRED. The live replacement is the\n", "# AR5 series in absolute Mt, so we pull the national total too and rebuild the\n", "# share ourselves.\n", "\n", "study_indicators = [\n", " \"SP.URB.TOTL.IN.ZS\",\n", " \"SP.URB.GROW\",\n", " \"EN.URB.LCTY.UR.ZS\",\n", " \"EN.GHG.CO2.TR.MT.CE.AR5\",\n", " \"EN.GHG.CO2.MT.CE.AR5\",\n", "]\n", "\n", "start_year, end_year = 1975, 2023" ] }, { "cell_type": "markdown", "id": "6df25bc2", "metadata": { "id": "6df25bc2" }, "source": [ "{warning} \n", "♻️ Check before you fetch\n", "\n", "Open data moves. The World Bank retires indicators, and a retired code does not\n", "fail politely: it takes the **whole request** down with it, including the codes\n", "that were fine.\n", "\n", "`EN.CO2.TRAN.ZS` — transport CO₂ as a share of fuel combustion, the obvious\n", "choice for this question — was retired exactly this way. The replacement reports\n", "absolute megatonnes, so further down we rebuild the share ourselves.\n", "\n", "Retired series leave the WDI database, so asking WDI what it still knows is the\n", "test. Better a clear error here than a confusing one later." ] }, { "cell_type": "code", "execution_count": null, "id": "c74788c2", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "c74788c2", "outputId": "58ffb285-1714-4312-a197-7df7eebdd2f9" }, "outputs": [ { "data": { "text/plain": [ "['EN.GHG.CO2.MT.CE.AR5',\n", " 'EN.GHG.CO2.TR.MT.CE.AR5',\n", " 'EN.URB.LCTY.UR.ZS',\n", " 'SP.URB.GROW',\n", " 'SP.URB.TOTL.IN.ZS']" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def check_indicators(codes, db=2):\n", " \"\"\"Fail loudly, naming any code WDI no longer serves.\"\"\"\n", " live = {s[\"id\"] for s in wb.series.list(codes, db=db)}\n", " dead = [c for c in codes if c not in live]\n", " if dead:\n", " raise ValueError(f\"Retired or unknown indicator(s) in db {db}: {dead}\")\n", " return sorted(live)\n", "\n", "\n", "check_indicators(study_indicators)" ] }, { "cell_type": "markdown", "id": "b0d8d495", "metadata": { "id": "b0d8d495" }, "source": [ "### World regions vs Germany\n", "\n", "We ask only for what the question needs: these five indicators, for Germany, the\n", "seven World Bank regions and the world, from 1975 onwards." ] }, { "cell_type": "code", "execution_count": null, "id": "6df9b40d", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 242 }, "id": "6df9b40d", "outputId": "cd4a6fea-9f0c-4e7a-d321-eb2aa4111dcb" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "9 geographies: ['DEU', 'EAS', 'ECS', 'LCN', 'MEA', 'NAC', 'SAS', 'SSF', 'WLD']\n", "441 rows\n" ] }, { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "summary": "{\n \"name\": \"panel\",\n \"rows\": 441,\n \"fields\": [\n {\n \"column\": \"economy\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 9,\n \"samples\": [\n \"EAS\",\n \"SSF\",\n \"LCN\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"country\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 9,\n \"samples\": [\n \"East Asia & Pacific\",\n \"Sub-Saharan Africa\",\n \"Latin America & Caribbean\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"date\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 14,\n \"min\": 1975,\n \"max\": 2023,\n \"num_unique_values\": 49,\n \"samples\": [\n 2010,\n 1978,\n 1976\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.GHG.CO2.MT.CE.AR5\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 8552.018587034643,\n \"min\": 259.7764,\n \"max\": 39112.6887901013,\n \"num_unique_values\": 441,\n \"samples\": [\n 484.2828,\n 1122.5887,\n 5731.2589\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.GHG.CO2.TR.MT.CE.AR5\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1769.3684168852296,\n \"min\": 49.3779,\n \"max\": 8287.0211535825,\n \"num_unique_values\": 441,\n \"samples\": [\n 70.7476,\n 107.5139,\n 1942.6302\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.URB.LCTY.UR.ZS\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 7.691489319655898,\n \"min\": 4.86797986186832,\n \"max\": 31.8312793660624,\n \"num_unique_values\": 441,\n \"samples\": [\n 26.7691527999798,\n 4.98085458766263,\n 8.47063546255311\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"SP.URB.GROW\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1.4091790958607335,\n \"min\": -1.81029751623121,\n \"max\": 5.25088897802315,\n \"num_unique_values\": 441,\n \"samples\": [\n 4.10888635709199,\n -0.398720997419651,\n 0.555963126996238\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"SP.URB.TOTL.IN.ZS\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 19.887353410010036,\n \"min\": 19.9267852576888,\n \"max\": 81.9034366692129,\n \"num_unique_values\": 441,\n \"samples\": [\n 30.2118748823815,\n 79.8416740841331,\n 80.4126136678733\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"co2_transport_share\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 7.016821731329616,\n \"min\": 9.577318923662782,\n \"max\": 35.47824835000047,\n \"num_unique_values\": 441,\n \"samples\": [\n 14.608736878534609,\n 9.577318923662782,\n 33.89534889097403\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", "type": "dataframe", "variable_name": "panel" }, "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
economycountrydateEN.GHG.CO2.MT.CE.AR5EN.GHG.CO2.TR.MT.CE.AR5EN.URB.LCTY.UR.ZSSP.URB.GROWSP.URB.TOTL.IN.ZSco2_transport_share
0WLDWorld202339112.6887908202.70907416.0264031.34534757.31169220.971990
1WLDWorld202238548.1189437964.54804515.9430451.32920357.07225420.661314
2WLDWorld202138219.9967677652.38192715.8651321.47777056.80922320.021932
3WLDWorld202036209.8616377097.07829915.8212461.52000256.44417619.599849
4WLDWorld201938008.4891318287.02115415.7794931.74214856.14098321.803080
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", "
\n" ], "text/plain": [ " economy country date EN.GHG.CO2.MT.CE.AR5 EN.GHG.CO2.TR.MT.CE.AR5 \\\n", "0 WLD World 2023 39112.688790 8202.709074 \n", "1 WLD World 2022 38548.118943 7964.548045 \n", "2 WLD World 2021 38219.996767 7652.381927 \n", "3 WLD World 2020 36209.861637 7097.078299 \n", "4 WLD World 2019 38008.489131 8287.021154 \n", "\n", " EN.URB.LCTY.UR.ZS SP.URB.GROW SP.URB.TOTL.IN.ZS co2_transport_share \n", "0 16.026403 1.345347 57.311692 20.971990 \n", "1 15.943045 1.329203 57.072254 20.661314 \n", "2 15.865132 1.477770 56.809223 20.021932 \n", "3 15.821246 1.520002 56.444176 19.599849 \n", "4 15.779493 1.742148 56.140983 21.803080 " ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# wbgapi has no \"regions_only\", so we take the seven WDI regional aggregates -\n", "# the entries with a numeric id - and add Germany and the world.\n", "regions = [r[\"code\"] for r in wb.region.list() if r[\"id\"]]\n", "economies = [\"DEU\"] + regions + [\"WLD\"]\n", "print(len(economies), \"geographies:\", economies)\n", "\n", "raw = wb.data.DataFrame(study_indicators, economies,\n", " time=range(start_year, end_year + 1),\n", " labels=True, columns=\"series\")\n", "\n", "panel = (raw.reset_index()\n", " .rename(columns={\"Country\": \"country\", \"Time\": \"date\"})\n", " .drop(columns=[\"time\"]))\n", "panel[\"date\"] = panel[\"date\"].astype(int)\n", "panel[\"co2_transport_share\"] = (\n", " 100 * panel[\"EN.GHG.CO2.TR.MT.CE.AR5\"] / panel[\"EN.GHG.CO2.MT.CE.AR5\"]\n", ")\n", "\n", "print(len(panel), \"rows\")\n", "panel.head()" ] }, { "cell_type": "markdown", "id": "4f5a7ae7", "metadata": { "id": "4f5a7ae7" }, "source": [ "### Let us plot multiple dimension at one plot\n", "Four indicators in one panel, repeated for each geography. Bars are urban\n", "population growth, the line is the population in the largest city, and the\n", "points are the transport share of national CO₂ — sized by how urban the\n", "country is." ] }, { "cell_type": "code", "execution_count": null, "id": "_Cdykwn3uFEi", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 717 }, "id": "aec318a0", "outputId": "a7f8be41-50f9-4683-facc-2d7f70e96717" }, "outputs": [], "source": [ "\n", "\n", "myplot = (\n", " ggplot()\n", " # Points: transport share, sized by urban population share.\n", " # colour and alpha sit OUTSIDE aes() - inside, they would be mapped, not set.\n", " + geom_point(panel,\n", " aes(x=\"date\", y=\"co2_transport_share\", size=\"SP.URB.TOTL.IN.ZS\"),\n", " colour=\"red\", alpha=0.3)\n", " # Line: population in the largest city\n", " + geom_line(panel, aes(x=\"date\", y=\"EN.URB.LCTY.UR.ZS\"),\n", " colour=\"red\", size=1.5)\n", " # Bars: urban population growth\n", " + geom_col(panel, aes(x=\"date\", y=\"SP.URB.GROW\"))\n", " + labs(x=\"YEAR\", y=\"IN PERCENTAGE (%)\")\n", " # coord_cartesian() zooms; xlim()/ylim() would delete the edge-year bars.\n", " + coord_cartesian(xlim=(start_year, end_year), ylim=(-5, 40))\n", " + theme_bw()\n", " + ggtitle(\"Bar: Urban population growth, \"\n", " \"Line: population in large cities, \"\n", " \"point: CO2 emissions from transport, \"\n", " \"point size: urban population\")\n", " + theme(legend_position=\"none\", figure_size=(12, 7))\n", " + facet_wrap(\"~country\")\n", ")\n", "\n", "myplot" ] }, { "cell_type": "code", "execution_count": null, "id": "61bf5208", "metadata": { "id": "_Cdykwn3uFEi" }, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "95c615a7", "metadata": { "id": "95c615a7" }, "source": [ "### What does the figure show?\n", "\n", "Germany urbanises slowly and its transport share of CO₂ climbs steadily. Sub-Saharan\n", "Africa is the opposite: fast urban growth, and a transport share that only starts\n", "rising recently.\n", "\n", "The regional aggregates hide as much as they show, though — a region is not a\n", "country. That is the reason for the second half of this chapter." ] }, { "cell_type": "markdown", "id": "14450a1d", "metadata": { "id": "14450a1d" }, "source": [ "### One value for every country\n", "\n", "For a map we need a single number per country. Countries do not all report in the\n", "same year, so instead of fixing a year and blanking out everyone who reports late,\n", "we take **the latest year each country actually reported**, and keep that year\n", "alongside the value." ] }, { "cell_type": "code", "execution_count": null, "id": "07bbe40c", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 224 }, "id": "07bbe40c", "outputId": "b41b449e-130e-433c-f6f7-47f145db18ee" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(217, 14)\n" ] }, { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "summary": "{\n \"name\": \"wide\",\n \"rows\": 217,\n \"fields\": [\n {\n \"column\": \"economy\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 217,\n \"samples\": [\n \"GBR\",\n \"YEM\",\n \"NPL\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Country\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 217,\n \"samples\": [\n \"United Kingdom\",\n \"Yemen, Rep.\",\n \"Nepal\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.GHG.CO2.MT.CE.AR5\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1011.5979979084033,\n \"min\": 0.0,\n \"max\": 13124.728,\n \"num_unique_values\": 199,\n \"samples\": [\n 12.2928,\n 0.7128,\n 1.7349\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.GHG.CO2.MT.CE.AR5:T\",\n \"properties\": {\n \"dtype\": \"object\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"2024\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.GHG.CO2.TR.MT.CE.AR5\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 143.44121836117338,\n \"min\": 0.0284,\n \"max\": 1678.5621,\n \"num_unique_values\": 194,\n \"samples\": [\n 6.7189\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.GHG.CO2.TR.MT.CE.AR5:T\",\n \"properties\": {\n \"dtype\": \"object\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"2024\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.URB.LCTY.UR.ZS\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 18.8763597365832,\n \"min\": 3.2665132513396,\n \"max\": 100.0,\n \"num_unique_values\": 151,\n \"samples\": [\n 23.817717227603\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"EN.URB.LCTY.UR.ZS:T\",\n \"properties\": {\n \"dtype\": \"object\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"2025\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"SP.URB.GROW\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1.6407681554343245,\n \"min\": -4.65327729336626,\n \"max\": 5.7990754437795,\n \"num_unique_values\": 217,\n \"samples\": [\n 0.393742423754665\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"SP.URB.GROW:T\",\n \"properties\": {\n \"dtype\": \"object\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"2025\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"SP.URB.TOTL.IN.ZS\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 23.081235162735723,\n \"min\": 14.6298998993293,\n \"max\": 100.0,\n \"num_unique_values\": 206,\n \"samples\": [\n 33.2395847043975\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"SP.URB.TOTL.IN.ZS:T\",\n \"properties\": {\n \"dtype\": \"object\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"2025\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"co2_transport_share\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 15.888192770095467,\n \"min\": 3.8068673655862266,\n \"max\": 83.7214808169165,\n \"num_unique_values\": 194,\n \"samples\": [\n 83.7214808169165\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"co2_transport_share:T\",\n \"properties\": {\n \"dtype\": \"object\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"2024\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", "type": "dataframe", "variable_name": "wide" }, "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
economyCountryEN.GHG.CO2.MT.CE.AR5EN.GHG.CO2.MT.CE.AR5:TEN.GHG.CO2.TR.MT.CE.AR5EN.GHG.CO2.TR.MT.CE.AR5:TEN.URB.LCTY.UR.ZSEN.URB.LCTY.UR.ZS:TSP.URB.GROWSP.URB.GROW:TSP.URB.TOTL.IN.ZSSP.URB.TOTL.IN.ZS:Tco2_transport_shareco2_transport_share:T
0AFGAfghanistan12.068420244.3071202442.88159020253.694491202525.940164202535.6890722024
1ALBAlbania4.243720241.5510202438.7282832025-0.611904202558.871474202536.5482952024
2DZAAlgeria186.3584202448.637620248.35296920252.031841202575.818736202526.0989582024
3ASMAmerican Samoa0.00012024NaNNaNNaNNaN-2.122542202580.5123512025NaNNaN
4ANDAndorraNaNNaNNaNNaNNaNNaN1.244439202588.9492672025NaNNaN
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", "
\n" ], "text/plain": [ " economy Country EN.GHG.CO2.MT.CE.AR5 EN.GHG.CO2.MT.CE.AR5:T \\\n", "0 AFG Afghanistan 12.0684 2024 \n", "1 ALB Albania 4.2437 2024 \n", "2 DZA Algeria 186.3584 2024 \n", "3 ASM American Samoa 0.0001 2024 \n", "4 AND Andorra NaN NaN \n", "\n", " EN.GHG.CO2.TR.MT.CE.AR5 EN.GHG.CO2.TR.MT.CE.AR5:T EN.URB.LCTY.UR.ZS \\\n", "0 4.3071 2024 42.881590 \n", "1 1.5510 2024 38.728283 \n", "2 48.6376 2024 8.352969 \n", "3 NaN NaN NaN \n", "4 NaN NaN NaN \n", "\n", " EN.URB.LCTY.UR.ZS:T SP.URB.GROW SP.URB.GROW:T SP.URB.TOTL.IN.ZS \\\n", "0 2025 3.694491 2025 25.940164 \n", "1 2025 -0.611904 2025 58.871474 \n", "2 2025 2.031841 2025 75.818736 \n", "3 NaN -2.122542 2025 80.512351 \n", "4 NaN 1.244439 2025 88.949267 \n", "\n", " SP.URB.TOTL.IN.ZS:T co2_transport_share co2_transport_share:T \n", "0 2025 35.689072 2024 \n", "1 2025 36.548295 2024 \n", "2 2025 26.098958 2024 \n", "3 2025 NaN NaN \n", "4 2025 NaN NaN " ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# mrnev=1 -> latest year each country actually reported\n", "# skipAggs -> countries only, no regional aggregates\n", "wide = wb.data.DataFrame(\n", " study_indicators, mrnev=1, skipAggs=True, labels=True,\n", " columns=\"series\", timeColumns=True,\n", ").reset_index()\n", "\n", "# The same derived share as Part 1, so both views show the identical quantity.\n", "wide[\"co2_transport_share\"] = (\n", " 100 * wide[\"EN.GHG.CO2.TR.MT.CE.AR5\"] / wide[\"EN.GHG.CO2.MT.CE.AR5\"]\n", ")\n", "wide[\"co2_transport_share:T\"] = wide[\"EN.GHG.CO2.TR.MT.CE.AR5:T\"]\n", "\n", "print(wide.shape)\n", "wide.head()" ] }, { "cell_type": "markdown", "id": "c0499e71", "metadata": { "id": "c0499e71" }, "source": [ "### Pick an indicator to map\n", "\n", "Change `INDICATOR` — that is the only line you need to touch." ] }, { "cell_type": "code", "execution_count": null, "id": "f786850f", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "f786850f", "outputId": "5aa56c01-64c2-4ffd-da92-cd708ca63486" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Transport share of national CO2 (%): 194 countries reporting, vintages 2024-2024\n" ] } ], "source": [ "INDICATORS = {\n", " # column : (legend title, single-hue colormap)\n", " \"co2_transport_share\": (\"Transport share of national CO2 (%)\", \"Reds\"),\n", " \"SP.URB.TOTL.IN.ZS\": (\"Urban population (% of total)\", \"Blues\"),\n", " \"SP.URB.GROW\": (\"Urban population growth (annual %)\", \"Purples\"),\n", " \"EN.URB.LCTY.UR.ZS\": (\"Population in largest city (% of urban)\", \"Greens\"),\n", " \"EN.GHG.CO2.TR.MT.CE.AR5\": (\"CO2 from transport (Mt CO2e)\", \"Reds\"),\n", "}\n", "\n", "INDICATOR = \"co2_transport_share\" # <- change this\n", "LEGEND_TITLE, CMAP = INDICATORS[INDICATOR]\n", "\n", "years = wide[f\"{INDICATOR}:T\"].dropna()\n", "print(f\"{LEGEND_TITLE}: {wide[INDICATOR].notna().sum()} countries reporting, \"\n", " f\"vintages {int(years.min())}-{int(years.max())}\")" ] }, { "cell_type": "markdown", "id": "6db3fd3f", "metadata": { "id": "6db3fd3f" }, "source": [ "### 6. Adding geometry\n", "\n", "Numbers join to country polygons. Which column you join on matters: Natural Earth\n", "ships several ISO-like codes and they are not interchangeable. `ISO_A3` carries\n", "`-99` for a handful of countries and would drop them without a word, so we use\n", "`ADM0_A3` and patch three known spelling differences by hand.\n", "\n", "The cell reports the two ways a country can end up grey — no match, or no value —\n", "because they look identical on the map and need opposite fixes." ] }, { "cell_type": "code", "execution_count": null, "id": "e24cde1f", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "e24cde1f", "outputId": "c868135e-7331-41e0-8322-8a6a757ba7f1" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "165 of 176 polygons carry a value\n", " no World Bank match (6): ['Falkland Is.', 'Fr. S. Antarctic Lands', 'N. Cyprus', 'Somaliland', 'Taiwan', 'W. Sahara']\n", " matched, but empty series (5): ['Kosovo', 'Montenegro', 'Palestine', 'S. Sudan', 'Serbia']\n" ] } ], "source": [ "COUNTRIES_URL = (\n", " \"https://github.com/opengeos/datasets/releases/download/vector/countries.geojson\"\n", ")\n", "world = gpd.read_file(COUNTRIES_URL)\n", "\n", "ALIASES = {\"KOS\": \"XKX\", \"SDS\": \"SSD\", \"PSX\": \"PSE\"} # Kosovo, S. Sudan, Palestine\n", "world[\"iso3\"] = world[\"ADM0_A3\"].replace(ALIASES)\n", "\n", "gdf = world.merge(wide, left_on=\"iso3\", right_on=\"economy\", how=\"left\")\n", "\n", "# Grey countries have two very different causes. Keep them apart:\n", "# no match -> the join broke (6 entities the World Bank does not report on)\n", "# no value -> the join worked, the series is just empty there\n", "no_match = sorted(gdf.loc[gdf[\"economy\"].isna(), \"NAME\"])\n", "no_value = sorted(gdf.loc[gdf[\"economy\"].notna() & gdf[INDICATOR].isna(), \"NAME\"])\n", "\n", "print(f\"{gdf[INDICATOR].notna().sum()} of {len(gdf)} polygons carry a value\")\n", "print(f\" no World Bank match ({len(no_match)}): {no_match}\")\n", "print(f\" matched, but empty series ({len(no_value)}): {no_value}\")" ] }, { "cell_type": "markdown", "id": "be8a1097", "metadata": { "id": "be8a1097" }, "source": [ "### The interactive globe\n", "\n", "Grey for every country first, then the colours on top, so **no data looks like no\n", "data** rather than like the bottom of the colour scale." ] }, { "cell_type": "code", "execution_count": null, "id": "de28c523", "metadata": { "id": "de28c523" }, "outputs": [], "source": [ "NO_DATA_COLOR = \"#d9d9d9\"\n", "\n", "\n", "def build_globe(data, indicator, legend_title, cmap, projection=\"globe\",\n", " for_export=False):\n", " \"\"\"The interactive globe. Built fresh for display and for export.\"\"\"\n", " m = leafmap.Map(\n", " center=[10, 25], zoom=1.4, style=\"liberty\",\n", " projection=projection, # \"mercator\" for a flat map\n", " height=\"700px\",\n", " use_message_queue=for_export, # on for to_html(), off for display\n", " add_sidebar=False, # the sidebar is an ipyvuetify widget\n", " add_floating_sidebar=False, # that VS Code cannot draw\n", " )\n", "\n", " # Put our fills under the basemap labels, so place names stay readable.\n", " first_symbol = m.find_first_symbol_layer()\n", " before = first_symbol[\"id\"] if first_symbol else None\n", "\n", " m.add_gdf(\n", " world, layer_type=\"fill\", name=\"No data\",\n", " paint={\"fill-color\": NO_DATA_COLOR, \"fill-outline-color\": \"#ffffff\"},\n", " fit_bounds=False, before_id=before,\n", " )\n", " m.add_data(\n", " data[data[indicator].notna()],\n", " column=indicator, cmap=cmap, scheme=\"Quantiles\", k=5,\n", " legend_title=legend_title, name=legend_title,\n", " fit_bounds=False, before_id=before,\n", " )\n", " m.add_layer_control()\n", " return m\n", "\n", "\n", "# Two ways to look at it:\n", "# plot_globe() - static, renders anywhere\n", "# build_globe() - interactive, and exported in Step 4" ] }, { "cell_type": "markdown", "id": "72e3efe9", "metadata": { "id": "72e3efe9" }, "source": [ "**Spin it.** Drag to rotate, scroll to zoom, buttons to switch layers or go flat." ] }, { "cell_type": "code", "execution_count": null, "id": "1013ac67", "metadata": { "id": "1013ac67" }, "outputs": [], "source": [ "build_globe(gdf, INDICATOR, LEGEND_TITLE, CMAP)" ] }, { "cell_type": "markdown", "id": "96c245d3", "metadata": {}, "source": [ "Here what you can expect to explore: \n", "\n", "![interactive-globe](/resources/interactive_globe_view_transport_indicator.gif)" ] }, { "cell_type": "markdown", "id": "23e95ab7", "metadata": { "id": "23e95ab7" }, "source": [ "### Try this\n", "\n", "- **Flat map:** `projection=\"mercator\"`, or click the globe button.\n", "- **One region:** `gdf[gdf.REGION_WB == \"Europe & Central Asia\"]`.\n", "- **3D:** add `extrude=True, scale_factor=20000` to `add_data()`. Looks great, reads badly." ] }, { "cell_type": "markdown", "id": "675daf2c", "metadata": { "id": "675daf2c" }, "source": [ "### What does the map show?\n", "\n", "The transport share of national CO₂ is highest across West and Central Africa —\n", "not because those countries drive the most, but because they emit little from\n", "anything else. Absolute transport emissions (try that indicator) put the same\n", "countries at the bottom.\n", "\n", "One indicator, two very different stories about the same places." ] }, { "cell_type": "markdown", "id": "7ff44ee0", "metadata": { "id": "7ff44ee0" }, "source": [ "## Try it yourself\n", "\n", "The data is live, so the notebook is a starting point rather than a result:\n", "\n", "- Swap `INDICATOR` for another entry — does the pattern move with it?\n", "- Add your own country next to Germany in Part 1.\n", "- Map a single region: `gdf[gdf.REGION_WB == \"Europe & Central Asia\"]`.\n", "- Find an indicator we did not use — the appendix below searches the catalogue.\n", "- Ask whether a *share* or an *absolute* value answers your question better.\n", "\n", "Reproducing a figure is the easy half. The interesting half is asking the data\n", "something it was not originally asked." ] }, { "cell_type": "markdown", "id": "3140e5e6", "metadata": { "editable": true, "slideshow": { "slide_type": "" } }, "source": [ "## Part two\n", "```\n", "# using ioer data repo api\n", "\n", "# country level point density visualization\n", "# https://data.fdz.ioer.de/file.xhtml?fileId=6268&version=1.2\n", "\n", "# city level stat\n", "# https://data.fdz.ioer.de/file.xhtml?fileId=6215&version=1.2\n", "\n", "# city level frequency eda\n", "# https://github.com/ssujit/public_transit_germany/blob/main/jupyter_notebook/1_exploratory_data_analysis.ipynb\n", "# https://github.com/ssujit/public_transit_germany/blob/main/jupyter_notebook/3_spatial_statistics_neighborhood.ipynb\n", "```" ] }, { "cell_type": "markdown", "id": "c1f6dfef", "metadata": { "editable": true, "id": "c1f6dfef", "slideshow": { "slide_type": "" } }, "source": [ "## Conclusion\n", "\n", "This chapter moves from:\n", "\n", "**open API → verified indicators → regional trend → global map → shareable artefact**\n", "\n", "Urbanisation and transport emissions are not one story but many, and which one you\n", "see depends on whether you look at a region or a country, a share or a total.\n", "\n", "Because everything here is fetched at run time and checked before use, the same\n", "notebook re-run next year gives next year's answer — including a loud failure if\n", "an indicator has been retired in the meantime." ] }, { "cell_type": "markdown", "id": "03e39b52", "metadata": { "editable": true, "id": "03e39b52", "slideshow": { "slide_type": "" } }, "source": [ "## Appendix: browse the catalogue\n", "\n", "Looking for a different indicator? `.list()` gives you dicts; wrap them in a\n", "DataFrame and filter. `wb.series.list(q=\"urban\")` searches." ] }, { "cell_type": "code", "execution_count": null, "id": "5c986acb", "metadata": { "id": "5c986acb" }, "outputs": [], "source": [ "countries = pd.DataFrame(wb.economy.list()) # ~ wb_countries()\n", "regions_df = pd.DataFrame(wb.region.list()) # ~ wb_regions()\n", "topics = pd.DataFrame(wb.topic.list()) # ~ wb_topics()\n", "series = pd.DataFrame(wb.series.list()) # ~ wb_indicators() (WDI only)\n", "\n", "print(f\"{len(countries)} economies, {len(regions_df)} regions, \"\n", " f\"{len(topics)} topics, {len(series)} indicators in WDI\")\n", "topics" ] }, { "cell_type": "markdown", "id": "7c5a63eb", "metadata": { "id": "7c5a63eb" }, "source": [ "## Notes\n", "\n", "- **Map blank?** Restart the runtime and run all." ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.4" } }, "nbformat": 4, "nbformat_minor": 5 }