WorldPop‐Malawi‐Fork Branch Comparison - datasciencecampus/WorldPop-Malawi-fork GitHub Wiki

Branches Compared - updated on 18/03/26

  • main
  • ons-compatability-updates

AI was used to create this content

Overview of File Changes

Status File Notes
Added .gitignore New in ons-compatability-updates
Modified 00_Data_Processing.R
Modified 00_Data_Processing2.R
Modified 01_Raster_Mosaicking_Buildings_2018.R
Modified 01_Raster_Mosaicking_Buildings_2024.R
Modified 01_Raster_Mosaicking_Workflow_2018.R
Modified 01_Raster_Mosaicking_Workflow_2024.R
Modified 02_Covariates_Extraction.R
Modified 03_HH_Model_Workflow_2024c.R
Modified 03_Pop_Model_Workflow_2018.R
Modified 04_Covs_Stack_Raster_cropping.R
Modified 04_Rasterize.R
Added load_required_libraries.R New utility script
Added src_covariates/covariate_download.log New log file
Added src_covariates/request_get_covariets.py New Python script
Added src_covariates/worldpop_covariets.txt New text file
Added src_covariates/worldpop_file_urls.md New markdown file
Added utils.R New utility script

00_Data_Processing.R

  • Updated file paths to use local data structure instead of network paths.
  • Changed path construction to use file.path() for cross-platform compatibility.
  • Updated shapefile references and improved spatial data handling.
  • Added logic to create and write spatial data if missing.
  • Fixed typos and improved comments.

00_Data_Processing2.R

  1. Path and Data Handling

    • All paths updated to use local data structure and file.path() for cross-platform compatibility.
    • Shapefile reference updated from "2018_MPHC_EAs_Final_for_Use_Corrected.shp" to "2018_MPHC_EAs_Final_for_Use.shp".

  2. Data Loading and Conditional Processing

    • Added conditional checks for the existence of currently missing input files* (DHS_Segmented_File.csv, zomba_rbind_data.csv, malemia_hh_without_IDs.csv).
    • If files are missing, processing is skipped and a warning is printed (see "4. Sumarisation and Output" for more information).
    • If files exist, data is loaded and processed as before.

  3. Spatial Data Processing

    • Improved logic for creating and saving spatial data (mphc_2018_sf) only if not already available.
    • Added steps to fix corrupt geometries and set spatial reference.
    • Used nearest neighbor assignment to link survey points to EA polygons.

  4. Summarisation and Output

    • Summarisation steps for each dataset (DHS, Zomba, Malemia) are now wrapped in conditional blocks.
    • If Zomba or Malemia data is missing, dummy columns filled with NAs are added to the output.
    • Output file writing updated to use file.path().

  5. Code Quality and Comments

    • Improved comments and documentation throughout the script.
    • Fixed typos and clarified logic.
    • Added notes about omitted data due to file availability.

  6. Data Integration

    • Data integration steps (left_join) now check for existence of summary tables before joining.
    • Ensures pipeline does not break if some input files are missing.

These changes make the script more robust, portable, and tolerant to missing input files, while improving spatial and data handling logic.

01_Raster_Mosaicking_Buildings_2018.R+ 01_Raster_Mosaicking_Buildings_2024.R

  • Updated paths to use local data and file.path().
  • Added source("utils.R") for utility functions.
  • Improved boundary loading logic with fallback to a generator function.
  • Changed folder order and ensured result directory creation.

01_Raster_Mosaicking_Workflow_2018.R + 01_Raster_Mosaicking_Workflow_2024.R

  • Updated paths to use local data and file.path().
  • Added source("utils.R").
  • Improved boundary loading logic.
  • Ensured result directory creation before writing rasters.

02_Covariates_Extraction.R

  • Updated paths to use local data and file.path().
  • Changed shapefile reference to a more specific file.
  • Improved comments and data loading.
  • skips "Extract non residential bcount" because file is made in 04_Rasterize.R which Ortis said was not needed - this is later adjusted for in several steps in the script
  • remove layer with duplicate name from raster_2024_covariates - not sure why this is happening - is this why we get fewer covariate unique names?
  • there are fewer covariates than intended in original script - there should be 64 but only 62 here. Code will now adapt to number of covariates present
  • when creating centroid:
Warning messages:
1: st_point_on_surface assumes attributes are constant over geometries 
2: In st_point_on_surface.sfc(st_geometry(x)) :
  st_point_on_surface may not give correct results for longitude/latitude data
  • removed non_res_extract from ea_2018 and ea_2024
  • skips "Extract non residential bcount_2018" because file is made in 04_Rasterize.R which Ortis said was not needed
  • same issue with unique covariates for 2018 as 2024 - latest run were 55 and 62 rather than 64 each

03_HH_Model_Workflow_2024c.R

  • Updated paths to use local data and file.path().

03_Pop_Model_Workflow_2018.R

  • Updated paths to use local data and file.path().

04_Covs_Stack_Raster_cropping.R

  • Updated paths to use local data and file.path().
  • Changed shapefile reference to a more specific file.

04_Rasterize.R

  • Updated paths to use local data and file.path().
  • Changed shapefile reference to a more specific file.

Data Directory Contents

data/Input_Data/

  • Mosaic_Buildings_2018/
  • Mosaic_Buildings_2024/
  • Mosaic_Covariates_2018/
  • Mosaic_Covariates_2024/

data/Malawi_Covs/

  • 2018_Buildings/
  • 2018_Covariates/
  • 2024_Buildings/
  • 2024_Covariates/

data/MNSO-Data/

  • FINAL MDHS LISTING DATA_Annon.dta
  • ICT Listing WorldPop.dta
  • IHS6 Listing WorldPop.dta
  • MDHS_2024_NoDZLK_anonymized.dta
  • mphc2018Data_AllRegions.dta
  • mphc2018Data_structures.dta
  • Naca Listing WorldPop.dta

data/Output_data/

  • hh_size_data.dbf
  • hh_size_data.gpkg
  • hh_size_data.prj
  • hh_size_data.shp
  • hh_size_data.shx
  • mphc_2018_sf_ea.gpkg
  • mphc_structures_points.gpkg
  • summarized_survey_data.csv

data/Shapefiles/

  • Extensive list of shapefiles (see above for details)

3. Getting covariate data

It was possible to download some of the covariate data needed to run the code in full getting the data from this website WorldPop :: Geospatial covariate data layers and using src_covariates scripts to source the data.

Some of the covariate data wasn't available on the website and had to be sourced from WorldPop.