PSRC Household Travel Survey Data Cleaning Project

Project Purpose and Goals

• Purpose: create tools to clean consultant delivered Household Travel Survey data
• Goals: get the HTS data to a certain level of quality for PSRC analysis and modeling purposes

Scope of work

1. Assessment of data quality

   • develop metrics (number of error flags, NAs) to assess if the data needs cleaning, how much cleaner the data is after each process

2. Rulesy

   • a set of scripts in SQL for automatic data cleaning process, identifying error flags and generating tables ready for Shiny-Fixie

3. Shiny-Fixie App: manual data cleaning

   Shiny-Fixie includes

   • a shiny user interface designed for manual data cleaning
   • a set of stored procedures (psrc/hhts_cleaning/hhts_cleaning /Stored Procedures) that update tables in database

4. Post-Fixie cleaning

   • update all derived variables from cleaned data

5. hhts_cleaning Database (hhts_cleaning repo)

   • a database that stores all data tables, views and stored procedures
   • temporal data tables in hhts_cleaning database that tracks all data edits (includes all previous records and when records are valid from and valid to, but not who made the edits)