Data Cleaning Exercise - GiovanniDw/frontend-applications GitHub Wiki
Before we started using the data from the opendata.rdw.nl, we had to do an exercise to process the data from the survey
Table of Contents
For this exersize we had to clean our own survey data. This data was collected at the start of the course, and was stored in a CSV file. This CSV file was cleaned and parsed into a JSON file by Jonah, Thanks for sharing!
In my first attempt to log the data to the console, I tried to open the local folder in the browser and used the fetch()
API method.
fetch('../data/survey_id.json')
.then((response) => {
return response.json();
})
.then((obj) => {
console.log(obj);
})
.catch((error) => {
console.error('nope');
});
This is the result i got from the console:
Cross origin requests are only supported for HTTP.
___
[Error] Fetch API cannot load [filepath] due to access control checks.
To fix this error I used the package live-server
.
For easy use I've added live-server
to my start script.
The server will start by running the command: $ npm start
in the terminal.
The JSON data is now logged to the console.
For better readability of the code i rewrote the code adding some callbacks
const data = '../data/survey_id.json';
getData = (data) => {
fetch(data)
.then(status)
.then(json)
.then((data) => {
console.log('Request succeeded with JSON response', data);
})
.catch((error) => {
console.error('nope' + error);
});
};
getData(data);
Get the response status
status = (response) => {
if (response.status >= 200 && response.status < 300) {
console.log(response);
return Promise.resolve(response);
} else {
return Promise.reject(new Error(response.statusText));
}
};
Read the JSON response
json = (response) => {
return response.json();
};
I've rewrote some functions to improve reusability.
And changed function names to match the functionality.
const data = '../data/Survey_Information_Design_clean-parsed.json';
fetchJSON = (data) => {
fetch(data)
.then(responseStatus)
.then(readResponseJSON)
.then(logResult)
.catch((error) => {
console.error('nope' + error);
});
};
Get Response Status
responseStatus = (response) => {
if (response.status >= 200 && response.status < 300) {
console.log(response);
return Promise.resolve(response);
} else {
return Promise.reject(new Error(response.statusText));
}
};
Read the response
readResponseJSON = (response) => {
return response.json();
};
log the results to the console.
logResult = (result) => {
console.log(result);
};
To filter the data and find answers by column name i wrote this function.
function getAnswersByColName(results, colName) {
if (results.length < 1) {
console.log('No Items in result');
return;
}
let resultByColName = [];
for (result of results) {
resultByColName.push(result[colName]);
}
return resultByColName;
}
Date Strings:
"geboortedatum"
&"tijdreisJaar"
When I tried to calculate the age from a birthday string
I found out that javascript couldn't read the dd-mm-yyyy
date format.
To read these dates correctly the format had to be changed this function was my attempt to fix the problem:
convertDateString = (date) => {
let convertedDates = [];
date.forEach((dateString) => {
let dateParts = dateString.split('-');
let dateObj = new Date(+dateParts[2], dateParts[1] - 1, +dateParts[0]);
convertedDates.push(dateObj.toString());
});
return convertedDates;
};
Input: "13-01-1994"
Output: Thu Jan 13 1994 00:00:00 GMT+0100 (CET)
The birthday dates where pretty clean compared to the timetracel dates. The timetravel dates where formatted even worse.
function transformDates(results) {
console.log('Transform date function');
let getDates = getAnswersByColName(results, 'tijdreisJaar');
console.log(getDates);
let newDates = formatDateString(getDates);
console.log(newDates);
return results;
}
function formatDateString(date) {
let convertedDates = [];
date.forEach((dateString) => {
if (dateString) {
let dateParts = dateString.replaceAll('-', '/').split('/');
let dateObj = new Date(
+dateParts[2],
dateParts[1] - 1,
+dateParts[0]
);
convertedDates.push(dateObj.toString());
}
});
console.log(convertedDates);
return convertedDates;
}
Updates might follow
I tried to calculate the amount of siblings an object has by combining the amount of brothers & sisters.
calcSiblings = (result) => {
let siblings = [];
for (let i = 0; i < result.length; i++) {
let getBrothers = result[i].hoeveelheidBroers;
let getSisters = result[i].hoeveelheidZussen;
siblings.push(getBrothers + getSisters);
}
console.log(siblings);
return;
};
The output of this method wasn't correct because 2 strings of numbers being added together.
Example:
'2' + '1' = '21';
To Fix this problem I had to write a function to convert number properties inside a string to a number property.
stringToNumber = (results) => {
console.log('str to nm');
console.log(results);
for (let i = 0; i < results.length; i++) {
let obj = results[i];
for (let prop in obj) {
if (obj.hasOwnProperty(prop) && !isNaN(obj[prop])) {
obj[prop] = +obj[prop];
}
}
}
return results;
};