Resit Functional Cleaning - RooyyDoe/functional-programming GitHub Wiki

At first I needed to clean my .then chain. So that my code would be more readable when somebody else is going through it. In the old version all my functions where standing in the .then chain and I had no more time to get them separated before the end of the assignment. Now that I am using ES6 Modules it is way easier for me to structure the code in separate files and be more consistent with this. This also makes my code look a lot cleaner than it did before.

Resit Assignments:

Cleaning the .then chain
Add ES6 modules into my project
Making the code more consistent (ES6)
Creating better function names
Style the visualization like the design

`.then` chain

In my old file I made every function inside the .then chain because I thought it was easier to read back what I did in my code and had no idea how to make it look better. After the assignment the teacher told me that it was better to use ES6 Modules to separate the code into more files. It indeed changed a lot if you compare my old and new .then chain. The chain is now also way more readable than it was when I had all my functions into it.

The old .then chain:

runQuery(url, ContinentQuery)
	.then((rawContinent) => cleanData(rawContinent))
	.then((filteringEmptyData) => filterEmptyResults(filteringEmptyData)) // optional
	.then((mainData) => mainData.map(uri => {
		return {uri: uri}
	}))
	.then(async (mainData) => {
		let catArray = await getCategories();

		return mainData.map((continent) => {
			continent.categories = catArray;
			return continent;
		});
	})
	.then((mainData)=> mainData.map(continent => {

		continent.categories = continent.categories.map(uri => {
			return {uri: uri};
		}); 
		return continent;
	}))
	// Calls anonymous async function
	.then(async (mainData) => { 
		// maps over continent of mainData and puts this in a variable
		let mainDataPromiseArray = mainData.map(async continent => { 
			// maps over continent.categories and puts this in a variable
			let categoriesPromiseArray = continent.categories.map(async categorie => { 
				// Invokes getCountOfCategory() and put all the results in count.
				let count = await getCountOfCategory(continent.uri, categorie.uri);
				// Returns two objects into continent.categories 
				// This will happen when the Promise.all has all the promises and
				// the override of continent.categories has happend.
				return {
					uri: categorie.uri,
					count: count
				};
	
			});
			// waits till all the promises are done and then puts them in a variable newCategories
			let newCategories = await Promise.all(categoriesPromiseArray);
			// Overrides continent.categories with the promises
			continent.categories = newCategories;
			// returns object
			return continent;
		});
		// waits till all the promises are done after the ones of categories and then puts them in newContinents
		let newContinents = await Promise.all(mainDataPromiseArray);
		// This is getting returned and there will be a list of arrays in the continent.array
		return newContinents;
		// After this I am going to make a object of the array.
	})
	.then((mainData) => mainData.map(continent => {
		continent.categories = continent.categories.map(category => {
			category.count = countCategoryResults(category.count);
			return category;
		}); 
		return continent;
	}))
	.then((mainData) => mainData.map(continent => {
		let sum = 0;
		for (let i = 0; i < continent.categories.length; i++) {
			sum = continent.categories[i].count + sum;

		}
		return {
			categories: continent.categories,
			uri: continent.uri,
			count: sum
		};
	}))
	.then((mainData) => mainData.map(continent => {
		for (let i = 0; i < continent.categories.length; i++) {
			continent.categories[i].percentage = continent.categories[i].count / continent.count;
		}	
		continent.categories = continent.categories.map(categories => {
			return {
				uri: categories.uri,
				count: categories.count,
				percentage: categories.percentage
			};
		
		});
		return continent;
	}))
	.then((mainData) => mainData.map(continent => {
		const newArray = [];
		continent.categories = continent.categories.map(categories => {
			let obj = {
				axis: categories.uri,
				value: categories.percentage
			};
			newArray.push(obj); 
		});
		return newArray;	
	}))
	// .then((mainData)=> console.log(mainData))
	.then((cleanData) => {
		// console.log('ik wil dit zien', cleanData)
		// const data = [cleanData];
		const data = cleanData;
		console.log('data', data)
		
		
		const margin = {top: 100, right: 100, bottom: 100, left: 100},
			width = Math.min(700, window.innerWidth - 10) - margin.left - margin.right,
			height = Math.min(width, window.innerHeight - margin.top - margin.bottom - 20);
		
		const color = d3.scale.ordinal()
			.range(['#EDC951','#CC333F','#00A0B0', '#002533', '#4D5B23']);
		
		const radarChartOptions = {
			w: width,
			h: height,
			margin: margin,
			maxValue: 0.5,
			levels: 7,
			color: color
		};
		//Call function to draw the Radar chart
		radarChart('.radarChart', data, radarChartOptions);
	});

New .then chain:

getData(ContinentQuery)
	.then((rawContinent) => clean.cleanData(rawContinent))
	.then((filteringEmptyData) => clean.filterEmptyResults(filteringEmptyData)) // optional
	.then((continentData) => clean.structureContinents(continentData))
	.then((allCategoryData) => clean.insertCategories(allCategoryData))
	.then((objectCount) => clean.amountOfObjects(objectCount))
	.then((objects) => clean.objectCount(objects))
	.then((totalObjects) => clean.totalObjects(totalObjects))
	.then((percentageOfCount) => clean.calcPercentage(percentageOfCount))
	.then((data) => clean.cleanedData(data))
	.then((cleanData) => {
		// console.log('ik wil dit zien', cleanData)
		// const data = [cleanData];
		const data = cleanData;
		console.log('data', data)
		
		
		const margin = {top: 100, right: 100, bottom: 100, left: 100},
			width = Math.min(700, window.innerWidth - 10) - margin.left - margin.right,
			height = Math.min(width, window.innerHeight - margin.top - margin.bottom - 20);
		
		const color = d3.scale.ordinal()
			.range(['#EDC951','#CC333F','#00A0B0', '#002533', '#4D5B23']);
		
		const radarChartOptions = {
			w: width,
			h: height,
			margin: margin,
			maxValue: 0.5,
			levels: 7,
			color: color
		};
		//Call function to draw the Radar chart
		radarChart('.radarChart', data, radarChartOptions);
	});

`.then` chain Explained

A lot of the chain works exactly the same as it did before so you will be able to read this here OLD CHAIN. All the changes that have been made into my functional cleaning will be explained in this page, everything that does not get explained has already been explained in the older pages.

ES6 Modules

There are different ways you can use ES6 Modules to structure your code. You can make separate files for every new function you make this would be useful when you have a big project and have more then one developer working on the code. I choose to structure my code into four files, one main file(.then chain), another file for fetching the API calls(getData.js), a file where I have all the functions(cleanfunction.js) and the last file for everything related to d3.js

Main.js

import getData from './modules/getData.js';
import clean from './modules/cleaningFunctions.js';
import radarChart from './modules/radarChart.js';

const ContinentQuery = `
		SELECT ?continents  WHERE {
				<https://hdl.handle.net/20.500.11840/termmaster2> skos:narrower ?continents .
        }`;

getData(ContinentQuery)
	.then((rawContinent) => clean.cleanData(rawContinent))
	.then((filteringEmptyData) => clean.filterEmptyResults(filteringEmptyData)) // optional
	.then((continentData) => clean.structureContinents(continentData))
	.then((allCategoryData) => clean.insertCategories(allCategoryData))
	.then((objectCount) => clean.amountOfObjects(objectCount))
	.then((objects) => clean.objectCount(objects))
	.then((totalObjects) => clean.totalObjects(totalObjects))
	.then((percentageOfCount) => clean.calcPercentage(percentageOfCount))
	.then((data) => clean.cleanedData(data))
	.then((cleanData) => {
		// console.log('ik wil dit zien', cleanData)
		// const data = [cleanData];
		const data = cleanData;
		console.log('data', data)
		
		
		const margin = {top: 100, right: 100, bottom: 100, left: 100},
			width = Math.min(700, window.innerWidth - 10) - margin.left - margin.right,
			height = Math.min(width, window.innerHeight - margin.top - margin.bottom - 20);
		
		const color = d3.scale.ordinal()
			.range(['#EDC951','#CC333F','#00A0B0', '#002533', '#4D5B23']);
		
		const radarChartOptions = {
			w: width,
			h: height,
			margin: margin,
			maxValue: 0.5,
			levels: 7,
			color: color
		};
		//Call function to draw the Radar chart
		radarChart('.radarChart', data, radarChartOptions);
	});
// .then((test) => console.log("test", test));

getData.js

const url = 'https://api.data.netwerkdigitaalerfgoed.nl/datasets/ivo/NMVW/services/NMVW-33/sparql';

const prefix = `PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
                PREFIX dc: <http://purl.org/dc/elements/1.1/>
                PREFIX dct: <http://purl.org/dc/terms/>
                PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
                PREFIX edm: <http://www.europeana.eu/schemas/edm/>
                PREFIX foaf: <http://xmlns.com/foaf/0.1/>`
        
export default async function getData(query){
	let response = await fetch(url+'?query='+ encodeURIComponent(prefix + query) +'&format=json');
	let json = await response.json();
	return json.results.bindings;
}

cleaningFunctions.js

import getData from './getData.js';

export default {
	cleanData,
	filterEmptyResults,
	structureContinents,
	insertCategories,
	amountOfObjects,
	objectCount,
	totalObjects,
	calcPercentage,
	cleanedData,

};

function structureContinents(mainData) {
	return mainData.map(uri => { return { uri }; } );
}

async function insertCategories(mainData) {
	let catArray = await getCategories();

	return mainData.map((continent) => {
		continent.categories = catArray;
		return continent;
	});
}

async function amountOfObjects(mainData) {
	// maps over continent of mainData and puts this in a variable
	let mainDataPromiseArray = mainData.map(async continent => { 
		// maps over continent.categories and puts this in a variable
		let categoriesPromiseArray = continent.categories.map(async categorie => { 
			// Invokes getCountOfCategory() and put all the results in count.
			let count = await getCountOfCategory(continent.uri, categorie.uri);
			// Returns two objects into continent.categories 
			// This will happen when the Promise.all has all the promises and
			// the override of continent.categories has happend.
			return {
				uri: categorie.uri,
				name: categorie.name,
				count: count
			};
	
		});
			// waits till all the promises are done and then puts them in a variable newCategories
		let newCategories = await Promise.all(categoriesPromiseArray);
		// Overrides continent.categories with the promises
		continent.categories = newCategories;
		// returns object
		return continent;
	});
		// waits till all the promises are done after the ones of categories and then puts them in newContinents
	let newContinents = await Promise.all(mainDataPromiseArray);
	// This is getting returned and there will be a list of arrays in the continent.array
	return newContinents;
	// After this I am going to make a object of the array.
}

function objectCount(mainData) {
	return mainData.map(continent => {
		continent.categories = continent.categories.map(category => {
			category.count = countCategoryResults(category.count);
			return category;
		}); 
		return continent;
	});
}

function totalObjects(mainData) {
	return mainData.map(continent => {
		let count = continent.categories.reduce((count, cur) => count += cur.count ,0);
		continent.count = count;
		return continent;
	});
}

function calcPercentage(mainData) {
	return mainData.map(continent => {
		continent.categories.forEach((category) => {
			category.percentage = category.count / continent.count; 
		});
		return continent;
	});
}

function cleanedData(mainData) {
	return mainData.map(continent => {
		const newArray = [];
		continent.categories = continent.categories.map(categories => {
			let obj = {
				axis: categories.name,
				value: categories.percentage
			};
			newArray.push(obj); 
		});
		return newArray;
	});
}

// Re-usable functions

function cleanCategoryData(rawResults) {
	return rawResults.reduce((cleanResults, rawResult) => {
		cleanResults.push({uri: rawResult.superCategory.value, name: rawResult.categoryName.value})
		return cleanResults;	
	},[]); 
	
}

function cleanData(rawResults) {
	return rawResults.reduce((cleanResults, rawResult) => {
		for(let key in rawResult) {
			if (rawResult[key].datatype === 'http://www.w3.org/2001/XMLSchema#integer') {
				let parsed = parseInt(rawResult[key].value, 10);
				cleanResults.push(parsed);
			} else cleanResults.push(rawResult[key].value);
		}
		return cleanResults;	
	},[]); 
	
}

function filterEmptyResults(emptyResults) {
	return emptyResults.slice(3,8);
}

function getCategories() {
	return new Promise(async(resolve) => {
		const categoryQuery = `
		SELECT ?superCategory ?categoryName WHERE {
				<https://hdl.handle.net/20.500.11840/termmaster2802> skos:narrower ?superCategory .
				?superCategory skos:prefLabel ?categoryName .
		}`;

		getData(categoryQuery)
			// .then((data) => console.log(data))
			.then((rawCategoryData) => cleanCategoryData(rawCategoryData))
			// .then((cleanCategoryData => combineContinentWithCategory(continentUriArray, cleanCategoryData)))
			.then((cleanResults) => resolve(cleanResults));
	});
}

function getCountOfCategory(continentUri, categoryUri) {
	return new Promise(async(resolve) => {
		let totalResult = `
			SELECT (COUNT(?category) AS ?categoryAmount) WHERE {
				
				<${continentUri}> skos:narrower* ?continent .
					?obj dct:spatial ?continent .
				<${categoryUri}> skos:narrower* ?category .
					?obj edm:isRelatedTo ?category .
					?category skos:prefLabel ?categoryName .
				
		} GROUP BY ?categoryName`;

		getData(totalResult)
			.then((rawCountData) => cleanData(rawCountData))
			// .then((countUpData) => countCategoryResults(countUpData))
			.then((countResults) => resolve(countResults));
		// .then((cleanCountData => combineCountWithCategory(continentUriArray, cleanCategoryData)))
		// .then((cleanResults) => resolve(cleanResults));
	});
}

function countCategoryResults(results) {
	return results.reduce((a, b) => a + b, 0);
}

radarChart.js

Refactoring dirty code

I needed to be more consistent with my code and try to keep a code standard. In my old code I made two functions that used ES5 javascript instead of ES6 javascript When I was working on the structure of my files I also have refactored these functions and made them work again.

.then((totalObjects) => clean.totalObjects(totalObjects))

In my old function I used a for loop to get a sum of all the category item counts of one continent. It would loop through every category and take the count of this and add them up and put the value into the variable sum. This would then be returned to the new object.

Old code

	.then((mainData) => mainData.map(continent => {
		let sum = 0;
		for (let i = 0; i < continent.categories.length; i++) {
			sum = continent.categories[i].count + sum;

		}
		return {
			categories: continent.categories,
			uri: continent.uri,
			count: sum
		};
	}))

In the new function I am making use of .reduce that gets all the count values and adds them up and puts the results into the variable count and then returns it to the new object.

New code

function totalObjects(mainData) {
	return mainData.map(continent => {
		let count = continent.categories.reduce((count, cur) => count += cur.count ,0);
		continent.count = count;
		return continent;
	});
}

.then((percentageOfCount) => clean.calcPercentage(percentageOfCount))

In this function I am doing exactly the same as I did before. I am making a for loop that goes through every category and then it will calculate the percentage of every categories.count and divide it by the total.count of that continent. This value will get returned in the new object

Old code

.then((mainData) => mainData.map(continent => {
		for (let i = 0; i < continent.categories.length; i++) {
			continent.categories[i].percentage = continent.categories[i].count / continent.count;
		}	
		continent.categories = continent.categories.map(categories => {
			return {
				uri: categories.uri,
				count: categories.count,
				percentage: categories.percentage
			};
		
		});
		return continent;
	}))

In the new function I am making use of .forEach that will go through all the categories and makes a calculation to get the percentage of every category.count and then returns this value in the new object

New code

function calcPercentage(mainData) {
	return mainData.map(continent => {
		continent.categories.forEach((category) => {
			category.percentage = category.count / continent.count; 
		});
		return continent;
	});
}

Names of categories

At my first version I only had the uri of the categories available to use in my radarChart but I also wanted the names so it would look better on the screen. First I made a new sparql query that would get the uri of the categories but also the names.

Sparql function

function getCategories() {
	return new Promise(async(resolve) => {
		const categoryQuery = `
		SELECT ?superCategory ?categoryName WHERE {
				<https://hdl.handle.net/20.500.11840/termmaster2802> skos:narrower ?superCategory .
				?superCategory skos:prefLabel ?categoryName .
		}`;

		getData(categoryQuery)
			// .then((data) => console.log(data))
			.then((rawCategoryData) => cleanCategoryData(rawCategoryData))
			// .then((cleanCategoryData => combineContinentWithCategory(continentUriArray, cleanCategoryData)))
			.then((cleanResults) => resolve(cleanResults));
	});
}

I needed also a new cleaning function that would clean the rawResults and then push the uri and the names into the new object so that the names can be used for the visualization in d3.js

I am using .reduce to get every category that there is in the rawResults and then push them into a new array. this array will then be re-used in a function of the .then chain.

cleanCategoryData()


function cleanCategoryData(rawResults) {
	return rawResults.reduce((cleanResults, rawResult) => {
		cleanResults.push({uri: rawResult.superCategory.value, name: rawResult.categoryName.value})
		return cleanResults;	
	},[]); 
	
}

Resit Functional Cleaning - RooyyDoe/functional-programming GitHub Wiki

.then chain

.then chain Explained