Building the Visualisation (part 1) - lennartdeknikker/frontend-data GitHub Wiki

Rendering the map

Getting the data

To start, I need to get the data I need from the endpoint. I started defining both the endpoint and the query and stored those in two const variables at the top of my script.:

const endpoint = "https://api.data.netwerkdigitaalerfgoed.nl/datasets/ivo/NMVW/services/NMVW-29/sparql";
const queryAncestorStatues = `
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX edm: <http://www.europeana.eu/schemas/edm/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX hdlh: <https://hdl.handle.net/20.500.11840/termmaster>
PREFIX wgs84: <http://www.w3.org/2003/01/geo/wgs84_pos#>
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX gn: <http://www.geonames.org/ontology#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT (SAMPLE(?identifier) AS ?identifierSample) ?title ?placeName ?imageLink ?extent ?lat ?long WHERE {
  <https://hdl.handle.net/20.500.11840/termmaster7745> skos:narrower* ?place .
  ?place skos:prefLabel ?placeName .
  VALUES ?type { "Voorouderbeelden" "Voorouderbeeld" "voorouderbeelden" "voorouderbeeld" }
  ?cho    dct:spatial   ?place ;
          dc:title      ?title ;
          dc:type       ?type ;
          dc:identifier ?identifier ;
          dct:extent    ?extent ;
          edm:isShownBy ?imageLink .
  ?place  skos:exactMatch/wgs84:lat   ?lat .
  ?place  skos:exactMatch/wgs84:long  ?long .
}
GROUP BY ?identifier ?title ?place ?placeName ?type ?imageLink ?lat ?long ?extent
`;

Later on, it seemed better to store those with some other settings in a settings object, to keep all settings in the same place and make the code easier to use for other objects or locations. For readability reasons, I kept the query separated like this:

// endpoint and query definitions
const queryAncestorStatues = `
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
...
`;

// application settings
const settings = {
  init: {
    endpoint: "https://api.data.netwerkdigitaalerfgoed.nl/datasets/ivo/NMVW/services/NMVW-29/sparql",
    query: queryAncestorStatues,
    mapJson: 'https://raw.githubusercontent.com/rifani/geojson-political-indonesia/master/IDN_adm_1_province.json'
  },
...

Adapting the example to show Indonesia

For the next steps I learned a lot from Ivan-ha's map of hongkong. I cloned his repository and started off trying to load a map of Indonesia instead of the map he used to show Hong Kong. First off I used a .geojson from Highmaps. After hours of changing settings to make this code work with this map, I figured it might be easier to look for another .geojson since this one appeared to already be hardcoded to a strange kind of projection which was hard to work with. I found another .geojson which worked like a charm after just changing a few projection settings:

const projection = d3
  .geoMercator()
  .center([120, -5])
  .scale(1600)
  .translate([window.innerWidth / 1.8, window.innerHeight/2.3]);

This projection is then initialized as a geoPath by const path = d3.geoPath().projection(projection); Now that I have a working map, I put the rendering in a different function called renderMap() like this:

function renderMap(geoJson) {
  g
  .append('g')
  .attr('class', 'g-map')
  .selectAll('path')
  .data(geoJson.features)
  .enter()
  .append('path')
  .attr('class', 'area')
  .attr('d', path)
  .attr('fill', 'white')
  .attr('stroke', '#c1eae8')
  .attr('stroke-width', 0.5)
  .on('mouseover', mouseOverHandler)
  .on('mouseout', mouseOutHandler)
  .on('click', areaClickHandler);
}

In contrast to the example I used, I needed to fetch the .geojson data from an external server. Putting this code in a function made it possible to first get the data needed and then render the map like this:

d3.json(geoJson)
.then(mapData => renderMap(mapData))
}

Adding zoom functionality

Next up, I wanted to add zoom functionality. I inserted the part of code responsible for that from the example. D3 has some great functions to do that, but I needed to tweak the ones used in the example. I changed the scaleExtent to use the variable given in the settings object, by rewriting the function like this:

const zoom = d3
  .zoom()
  .scaleExtent(settings.render.scaleExtent)
  .on('zoom', zoomHandler);

function zoomHandler() {
  g.attr('transform', d3.event.transform);
}

and adding scaleExtent as a property of a new object called render embedded in the settings object:

const settings = {
  init: {
    endpoint: "https://api.data.netwerkdigitaalerfgoed.nl/datasets/ivo/NMVW/services/NMVW-29/sparql",
    query: queryAncestorStatues,
    mapJson: 'https://raw.githubusercontent.com/rifani/geojson-political-indonesia/master/IDN_adm_1_province.json'
  },
  render: {
    scaleExtent: [.5, 20]
  }
}

Adding Mouse Over and Click Interaction

Next I rewrote the mouse over and mouse out functions to change the colors according to the Volkenkunde color scheme:

function mouseOverHandler(d, i) {
  let element = d3.select(this);
  if (element.attr('fill') !== '#00aaa0') {
    element.attr('fill', '#9aeae6')
  }
}
function mouseOutHandler(d, i) {
  let element = d3.select(this);
  if (element.attr('fill') !== '#00aaa0') {
    element.attr('fill', 'white')
  }
}

Now, I noticed clicking on different areas doesn't update the area name shown in the <h3>. I looked up the function that handles clicks, logged the data object and noticed that in the .geoJson I use, the area names are stored in different variables so I changed that in the function:

function areaClickHandler(d) {
  d3.select('#map_text').text(`You've selected ${d.properties.NAME_1}`)
  d3.selectAll('.area').attr('fill', 'white');
  d3.select(this).attr('fill', '#00aaa0')
}

Adding Zoom Buttons

Another functionality that's shown in the example is zooming by clicking on two buttons. At first I didn't see any improvement in adding that functionality, but it might come in handy when showing the application on different devices, so I added the buttons to my html and copied the code responsible for zooming like that from the example. I just needed to make some changes in the zoomSteps, but that was not too hard since the function was already written to have adjustable zoomsteps.:

// makes it possible to zoom on click with adjustable steps
d3.select('#btn-zoom--in').on('click', () => clickToZoom(2));
d3.select('#btn-zoom--out').on('click', () => clickToZoom(.5));

function clickToZoom(zoomStep) {
  svg
    .transition()
    .duration(500)
    .call(zoom.scaleBy, zoomStep);
}

Rendering the data points

Now I've exhausted the example, it's time to make the map reflect the Volkenkunde data. I started by loading the data into the application. After some research I found the d3.json function which makes it easier than ever to fetch the query results:

d3.json(endpoint + "?query=" + encodeURIComponent(query) + "&format=json")
.then(objects => { renderObjects(objects.results.bindings) })

Now I need to write the renderObjects() function. I started off copying the code to render the map and embedding that in the new function. Doing the Lynda tutorial on d3 I learnt how to draw circles using data. I used that to rewrite the function like this. I added attributes to reflect the coordinates and placenames in html data-attributes. That way it's easier what point corresponds with what data.

function renderObjects(objects) {
  g
  .append('g')
  .attr('class', 'g-datapoints')
  .selectAll('.datapoint')
  .data(objects)
  .enter()
  .append('circle')
  .attr('class', 'datapoint')
  .attr('data-place', d => d.placeName)
  .attr('data-long', d => d.long)
  .attr('data-lat', d => d.lat)
  .attr('cx', d => projection([d.long, d.lat])[0])
  .attr('cy', d => projection([d.long, d.lat])[1])
  .attr('fill', '#00827b')
  .attr('fill-opacity', .5)
  .on('click', objectClickHandler);
}

Transforming the data

Now that the objects are rendered in the map, I wanted to group the statues that are found at the same location and change the circle size to reflect the amount. After some struggling, Laurens showed us an example of using d3.nest() which does just that. I copied the code he wrote from the screen and adjusted id to group my data on locations and adding amount as a property like this:

function transformData(source) {
  let transformed = d3.nest()
  .key(function(d) { return d.placeName.value})
  .entries(source);
  transformed.forEach(element => {
    element.amount = element.values.length;
    element.placeName = element.values[0].placeName.value;
    element.long = element.values[0].long.value;
    element.lat = element.values[0].lat.value;
  });
  return transformed;
}

I changed the loadData function to transform the data before rendering the datapoints on the map:

function loadData(endpoint, query) {
d3.json(endpoint + "?query=" + encodeURIComponent(query) + "&format=json")
.then(objects => { renderObjects(transformData(objects.results.bindings)) })
}

Since it's working right away, I'm starting to see the benefits of functional programming.

adjusting datapoint size to zoom level

After testing my application and tweaking the maximal and minimal size of the data points, I decided to write a function to adjust the point sizes according to the zoom level:

function adjustCirclesToZoomLevel(zoomLevel) {
  const minRadius = (zoomLevel/3 < 2) ? 3 - (zoomLevel/3) : 1;
  const maxRadius = (zoomLevel*7 < 37.5) ? 40 - (zoomLevel*7) : 2.5;
  const maxZoomLevel = settings.render.scaleExtent[1];
  const factor = (maxRadius-minRadius) / (settings.render.maxValueInData-settings.render.minValueInData);

  g.selectAll('circle')
  .attr('r', d => {
    if (d.amount*factor < minRadius) {
      return minRadius;
    }
    else if (d.amount*factor > maxRadius) {
      return maxRadius;
    }
    else {
      return d.amount*factor;
    }
  })
  if (zoomLevel < maxZoomLevel/2) {
    g.selectAll('.datapoint')
    .attr('fill-opacity', (.3 + .7/zoomLevel))
  } else {
    g.selectAll('.datapoint')
    .attr('fill-opacity', 1)
  }
}

As you see, the function needs the data range to adjust the size accordingly. For now, I stored those variables in the settings object, but later on, I want to have the function adapt automatically to the rendered data source.

const settings = {
  init: {
    endpoint: "https://api.data.netwerkdigitaalerfgoed.nl/datasets/ivo/NMVW/services/NMVW-29/sparql",
    query: queryAncestorStatues,
    mapJson: 'https://raw.githubusercontent.com/rifani/geojson-political-indonesia/master/IDN_adm_1_province.json'
  },
  render: {
    scaleExtent: [.5, 20],
    minValueInData: 3,
    maxValueInData: 200
  }
}

This function needs to be executed every time the user zooms in or out, so I called it in the zoomHandler function that's already there to transform the viewport:

function zoomHandler() {
  g.attr('transform', d3.event.transform);
  adjustCirclesToZoomLevel(d3.event.transform.k);
}

After some experimenting, I found out the function also needs to be called on initialisation of the map, so I added it to the renderObjects() function with 1 as parameter, since the map always starts on that level. I first wrote it obtaining the zoomlevel to start with from the settings, but that seemed a bit redundant, so I removed that.

function renderObjects(objects) {
  g
  .append('g')
  .attr('class', 'g-datapoints')
  ...
  adjustCirclesToZoomLevel(1);
}

Showing a list of objects

Now there's just one more functionality I want to add, because I think it's not too hard and we're running out of time. I want the user to be able to click the data points and see the corresponding statues listed below. To do that, I added a click handler to each drawn data point:

function renderObjects(objects) {
  g
  .append('g')
  ...
  .on('click', objectClickHandler);
}

The objectClickHandler generates html for all the data for a specific clicked data point like this:

function objectClickHandler(d, i) {
  let newHtml =`
  <h3>List of statues found at ${d.placeName}</h3>
  <ol class="item-list">
  `;

  d.values.forEach( value => {
    newHtml += `
      <li>${value.title.value}
        <ul class="sub-item-list">
          <li>Identifier: ${value.identifierSample.value}</li>
          <li>Extent: ${value.extent.value}</li>
          <li>Image Link: <a href="${value.imageLink.value}">${value.imageLink.value}</a></li>
          <li>Origin location: ${value.placeName.value} (${value.lat.value}, ${value.long.value})</li>
        </ul>
        <img src="${value.imageLink.value}" class="object-image" /> 
      </li>
    `;
  });

  newHtml += "</ol>";
  document.querySelector('.info').innerHTML = newHtml;
}

I added a new <div class="info"> to the html document to have that ready to be populated with the generated HTML. I know this can be written nicer and show the objects in a better way, but I don't have any more time for that.

Adding Item search

After splitting up the code, it became easier to see how to implement a search bar to change the map to display different kind of items. I now have a settings object that just needs to get a different query when sent to the addDataVisualisation function to establish that.

First, I added a searchbar to my html: <div id="search_bar">Search for items: <input type="text" id="search_input"></div>

Then I wrote a function to store different variables in the SPARQL query like this

function loadMapWithItemType(searchWord) {
  document.querySelector('#map_container').innerHTML = "";
  // endpoint and query definitions
  const query = `
  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
  PREFIX dc: <http://purl.org/dc/elements/1.1/>
  PREFIX dct: <http://purl.org/dc/terms/>
  PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
  PREFIX edm: <http://www.europeana.eu/schemas/edm/>
  PREFIX foaf: <http://xmlns.com/foaf/0.1/>
  PREFIX hdlh: <https://hdl.handle.net/20.500.11840/termmaster>
  PREFIX wgs84: <http://www.w3.org/2003/01/geo/wgs84_pos#>
  PREFIX geo: <http://www.opengis.net/ont/geosparql#>
  PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
  PREFIX gn: <http://www.geonames.org/ontology#>
  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
  PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
  SELECT (SAMPLE(?identifier) AS ?identifierSample) ?title ?placeName ?imageLink ?extent ?lat ?long WHERE {
    <https://hdl.handle.net/20.500.11840/termmaster7745> skos:narrower* ?place .
    ?place skos:prefLabel ?placeName .
    VALUES ?type {"${searchWord}"}
    ?cho    dct:spatial   ?place ;
            dc:title      ?title ;
            dc:type       ?type ;
            dc:identifier ?identifier ;
            dct:extent    ?extent ;
            edm:isShownBy ?imageLink .
    ?place  skos:exactMatch/wgs84:lat   ?lat .
    ?place  skos:exactMatch/wgs84:long  ?long .
  }
  GROUP BY ?identifier ?title ?place ?placeName ?type ?imageLink ?lat ?long ?extent
  `;

  console.log(query);

  // application settings
  const settings = {
    init: {
      targetDiv: '#map_container',
      endpoint: "https://api.data.netwerkdigitaalerfgoed.nl/datasets/ivo/NMVW/services/NMVW-29/sparql",
      query: query,
      mapJson: 'https://raw.githubusercontent.com/rifani/geojson-political-indonesia/master/IDN_adm_1_province.json',
      svgSize: ['100%', '100%']
    },
    render: {
      scaleExtent: [.5, 20],
      minValueInData: 3,
      maxValueInData: 200
    },
    projection: {
      center: [120,-5],
      scale: 1600,
      translation: [window.innerWidth / 1.8, window.innerHeight/2.3]
    }
  }

  AddDataVisualisation(settings);
}

Then I added an eventlistener to the <input id="search_input"> like this:

document.querySelector('#search_input').addEventListener('keypress', function (e) {
  let searchWord;
  let key = e.which || e.keyCode;
  if (key === 13) {
    searchWord = this.value;
    loadMapWithItemType(searchWord);
  }
});

Check out the application here.

⚠️ **GitHub.com Fallback** ⚠️