The Survey Exercise

Before we started using the data from the opendata.rdw.nl, we had to do an exercise to process the data from the survey

Table of Contents

• The Survey Exercise
  • About the exercise
  • Getting the Data
    • First attempt
    • Next Step
      • Improvements
    • Final version
  • Filter The Data
    • Get data by column name
  • Clean the Data
    • Cleaning Dates
      • Birthdates
      • Time Travel Dates
    • Cleaning Dates Improvements
  • Manipulate The Data
    • Calculate Siblings

About the exercise

For this exersize we had to clean our own survey data. This data was collected at the start of the course, and was stored in a CSV file. This CSV file was cleaned and parsed into a JSON file by Jonah, Thanks for sharing!

Getting the Data

In my first attempt to log the data to the console, I tried to open the local folder in the browser and used the fetch() API method.

First attempt

fetch('../data/survey_id.json')
 .then((response) => {
  return response.json();
 })
 .then((obj) => {
  console.log(obj);
 })
 .catch((error) => {
  console.error('nope');
 });

This is the result i got from the console:

Cross origin requests are only supported for HTTP.
___

[Error] Fetch API cannot load [filepath] due to access control checks.

To fix this error I used the package live-server. For easy use I've added live-server to my start script.

The server will start by running the command: $ npm start in the terminal. The JSON data is now logged to the console.

Next Step

For better readability of the code i rewrote the code adding some callbacks

const data = '../data/survey_id.json';
getData = (data) => {
 fetch(data)
  .then(status)
  .then(json)
  .then((data) => {
   console.log('Request succeeded with JSON response', data);
  })
  .catch((error) => {
   console.error('nope' + error);
  });
};
getData(data);

Get the response status

status = (response) => {
 if (response.status >= 200 && response.status < 300) {
  console.log(response);
  return Promise.resolve(response);
 } else {
  return Promise.reject(new Error(response.statusText));
 }
};

Read the JSON response

json = (response) => {
 return response.json();
};

Improvements

I've rewrote some functions to improve reusability.
And changed function names to match the functionality.

const data = '../data/Survey_Information_Design_clean-parsed.json';

fetchJSON = (data) => {
 fetch(data)
  .then(responseStatus)
  .then(readResponseJSON)
  .then(logResult)
  .catch((error) => {
   console.error('nope' + error);
  });
};

Get Response Status

responseStatus = (response) => {
 if (response.status >= 200 && response.status < 300) {
  console.log(response);
  return Promise.resolve(response);
 } else {
  return Promise.reject(new Error(response.statusText));
 }
};

Read the response

readResponseJSON = (response) => {
 return response.json();
};

log the results to the console.

logResult = (result) => {
 console.log(result);
};

Final version

Filter The Data

Get data by column name

To filter the data and find answers by column name i wrote this function.

function getAnswersByColName(results, colName) {
 if (results.length < 1) {
  console.log('No Items in result');
  return;
 }
 let resultByColName = [];
 for (result of results) {
  resultByColName.push(result[colName]);
 }
 return resultByColName;
};

Clean the Data

Cleaning Dates

Date Strings: "geboortedatum" & "tijdreisJaar"

When I tried to calculate the age from a birthday string I found out that javascript couldn't read the dd-mm-yyyy date format.

Birthdates

To read these dates correctly the format had to be changed this function was my attempt to fix the problem:

  convertDateString = (date) => {
    let convertedDates = [];
    date.forEach(dateString => {
        let dateParts = dateString.split('-');
        let dateObj = new Date(+dateParts[2], dateParts[1] - 1, +dateParts[0]);
        convertedDates.push(dateObj.toString())
    });
    return convertedDates
}

Input: "13-01-1994" Output: Thu Jan 13 1994 00:00:00 GMT+0100 (CET)

The birthday dates where pretty clean compared to the timetracel dates. The timetravel dates where formatted even worse.

Time Travel Dates

function transformDates(results) {
 console.log('Transform date function');
 let getDates = getAnswersByColName(results, 'tijdreisJaar');
 console.log(getDates);
 let newDates = formatDateString(getDates);
 console.log(newDates);
 return results;
};

function formatDateString(date) {
 let convertedDates = [];
 date.forEach((dateString) => {
  if (dateString) {
   let dateParts = dateString
    .replaceAll('-', '/')
    .split('/');
   let dateObj = new Date(
    +dateParts[2],
    dateParts[1] - 1,
    +dateParts[0]
   );
   convertedDates.push(dateObj.toString());
  }
  
 });
 console.log(convertedDates);
 return convertedDates;
};

Cleaning Dates Improvements

Updates might follow

Manipulate The Data

Calculate Siblings

I tried to calculate the amount of siblings an object has by combining the amount of brothers & sisters.

calcSiblings = (result) => {
 let siblings = [];
 for (let i = 0; i < result.length; i++) {
  let getBrothers = result[i].hoeveelheidBroers;
  let getSisters = result[i].hoeveelheidZussen;
  siblings.push(getBrothers + getSisters);
 }
 console.log(siblings);
 return;
};

The output of this method wasn't correct because 2 strings of numbers being added together.

Example:

"2" + "1" = "21"

To Fix this problem I had to write a function to convert number properties inside a string to a number property.

stringToNumber = (results) => {
 console.log('str to nm');
 console.log(results);
 for (let i = 0; i < results.length; i++) {
  let obj = results[i];
  for (let prop in obj) {
   if (obj.hasOwnProperty(prop) && !isNaN(obj[prop])) {
    obj[prop] = +obj[prop];
   }
  }
 }
 return results;
};