Heng Log - VIDA-NYU/reprozip GitHub Wiki

On Fridays, please devote at least a half hour to reflect on the work you've done over the previous week. Please use this format:

Date Log of what you've done during the week (can be a bulleted list or as detailed as you want).

Week 1

Monday May 22, 2017

  • Went over the initial given ReproZip Data and got the idea of how to map data with visualization
  • Drafted and documented the ideas & use cases of how the visualizations and interactions can be for our application.
  • Came up a consolidated list of questions and reviewed them with Remi
  • Researched on visualization inspirations: datasketch , Andy Shora
  • Picked the initial candidates of visualizations to work on for Tuesday.

Tuesday May 23, 2017

  • Tried out the online D3 examples of Tree, Network and Arc Diagrams. Realized, the existing common diagram structure can not fully represent what we need to present from ReproZip trace.
  • Taking the consideration of Remi's suggestion on the timeline design for performance comparison for future development. Since this approach does add value for visualizing the provenance graph, I started to draft a markup that is suitable for ReproZip trace
  • Drafted the following markups by recreated the bash-count and digits-io diagram with the new design: Digits-io comparison

Digits-io comparison with the previous design. Bash-count comparison Bash-count comparison with the previous design.

Friday May 23, 2017

  • Reviewed and compared the new JSON file and old JSON files that Remi sent me.
  • Composted a list of questions and suggestions to go over with Remi on Tuesday.
  • Inprogress, enhancing the existing design to reflect the new JSON file format.
  • The goal for next week: Finalized the design and start coding for the first iteration.

Week 2

Tuesday May 30, 2017

  • Reviewed and updated the visual markups. Created multiple markups to figure out what can be done and what should not.
  • Discovered few bugs from reprounzip(ex. Python regression that prevent reprounzip graph to generate the provenance graph in Python 3.6).
  • Reproduced and generated the provenance graph for multiple experiments from the Reprozip Examples Page to get the better understanding of different varieties of provenance graph.

Wednesday May 31, 2017

  • Reviewed D3 tree diagram data structure and coded the functions to map the data from the json files to the D3 specified data structure.
  • Started coding on a working prototype: Screenshot of the working prototype

Friday June 2, 2017

  • Updated and enhanced the working prototype and uploaded to my personal Github for hosting. The gh-pages can be found here, please take a look.

Here is the demo for stacked-up-website

  • Reviewed the new json files with timestamps. I will start to look into how to incorporate the timestamp into the working prototype next week.
  • For next week, I will need to revisit the markups with the new updates. And I will start working on the how to visualize the files and incorporate the time stamp into this working prototype

Week 3

Monday June 5th, 2017

  • Updated our visualization with mouse over to show the "description" details and enhanced our code to handle double click / single click for future interaction development
  • Created the initial draft for our RaproZip Logo: Repro Zip Logo 1st Draft:

Thursday June 8th, 2017

  • Designed the markedup for how to visualize files in the provenance graphs. After some research, I decided to achieve the visualization by using the force layout from D3. Design and markedup

  • Studying force layout from D3.

  • Started coding on on the force layout: Trying to use force layout to represent files in the provenance graph

Friday June 9th, 2017

  • Continued to configure the D3 specified data structure to use the given json data.

  • To ensure we can accomplish the task for this week for listing files, I took what Remi and Vicky suggested in the morning meeting to list the reads from and writes to files by clicking on the process. Here is screenshot for visulizing the reads from and writes to files

  • Updated and enhanced the working prototype and uploaded to my personal Github for hosting. The gh-pages can be found here, please take a look.

  • And I also managed to create few logo markups Here are the few logos markups I designed

Week 4

Monday June 12th, 2017

  • Updated our visualization to show the time on the timeline by using the timestamps on the the json file. Time line to show the duration between 2 proccess:

Thursday June 15th, 2017

  • Designed the markedup for side panel by studying the best practices from The Webby Awards. After some research, I decided to achieve the visualization by using the navigation panel from The Intercept_. (https://theintercept.com) The initial version fo the side panel

Friday June 16th, 2017

  • Tried to fix the bug of the time line not align with the tree diagram when collapsed and expanded the diagram from the nodes that is 1 level down from the very last node that just expanded. This bug should be able to fix by next Monday.
  • Updating side panel to let a range of files to group by the shortest common path. I designed he algorithm and implementing it on the coming Monday.
  • I have designed another set of ReproZip logos markups with the team's feedbacks. Please take a look: Here are the few logos markups I designed

Week 5

Monday June 19th, 2017

  • Implemented the data-structure group files by the shortest common path Files are grouped by shortest common path:

Tuesday June 20th, 2017

  • Implemented the data-structure for files. We now can use the file objects that contain the meta data Data Structure to show the meta data for each file:

Thursday June 22th, 2017

  • Update and enhanced the side bar to incorporate with the data-structure of both files and group of files.
  • Added /visualize the commands that the selected process use to the side panel

Friday June 23th, 2017

  • Continue working on the file search function
  • Work on shorten the file path within the group of listing files
  • Working on double click to select the entire file with their proper path incase if user wants to copy and paste the file path for other usage

Week 6

Monday June 26th, 2017

  • Modified the timeline to show the total time and the duration for the current selected process.
  • The length of duration of process on the timeline is now based on the ratio to the total time took for the entire experiment instead of the length of the expended process that show on the network graph. Updated Time Line:

Tuesday June 27th, 2017

  • Got stuck on figuring out why a function was not executing from the expected order. It was due to "d3.json method is asynchronous, which means that after you call d3.json(), the code afterwards will execute first before the function inside the d3.json method has finished."
  • The issue from above was fixed by the end of the day. And which helps to make the progress to fix a road block in the data structure for the visualization.

Thursday June 29th, 2017

  • Added "Associated Packages" in the sidebar when a process is selected
  • Visualizing the packages and able to make the visulizing the connections between packages and processes. Packages & Processes relationship visualization 1:

Packages & Processes relationship visualization 2:

Friday June 30th, 2017

  • Trying to figure out why the links between connecting forces nodes were not displaying. Will continue to figure this out on Monday

Week 7

Monday July 3rd, 2017

  • Morning: Tried to visualize the links to connected the packages to the process. But the position is still a bit off not able to drag the package around yet.
  • Afternoon: Set up the back-end environment for the dictionary of files

Wednesday July 5th, 2017

  • Morning: Completed the updates for the sidebar to include list of packages and files within. list of packages and files with in:
  • Afternoon: Still tried to connect the links to force nodes created links to connect force nodes:

Thursday July 6th, 2017

  • Morning: Continue to work on connection the links Packages & Processes relationship visualization 2:
  • Afternoon: Enhance the side bar's search function and automatically adjust the visualization's width when open or close the sidebar Updated sidebar:

Friday July 7th, 2017

  • Refactoring the code and had code review with Remi

Week 8

July 10th - July 14th

  • Able to link packages with associated processes and display the links. But when move the packages, the links did not follow Linked packages with associated processes:
  • Finally, wired the packages with the processes. When moving the packages, the links that connected to the process will move alongside. Linked packages with associated processes:

Week 9

July 17th - July 21st

  • Tried to put packages on the right hand side. But the result was not looking as expected Put packages on the right hand side:
  • Keep packages on the top and added an margin on top so the network graph will not cross over to the packages. Margin on top:
  • Researching on how to make large scope changes via user interaction (ex. check boxes). Researched on using react js, jQuery and bootstrap

Week 10

Monday July 24th, 2017

  • Managed to accomplish "check boxes for specific packages to show/hide" alongside with all the neccessary setups check boxes for specific packages to show/hide:

Wednesday July 26th, 2017

  • Morning: Completed the updates for the sidebar to include list of packages and files within.

  • Afternoon: Still tried to connect the links to force nodes created links to connect force nodes:

Thursday July 17th, 2017

  • Evaluate the value and effort to build between "When a parent process is selected, show pkgs used by children" and "show all associated process from selected package"

Friday July 7th, 2017

  • Build up the back-end structure to connect all associated process from selected package and attempted to visualize it.

Week 11

Monday July 31st, 2017

  • Managed to accomplish "Export as PNG & SVG" with NYTimes svg-crowbar
  • Managed to accomplish "When you click on a package, have it connect to the processes"
    When you click on a package, have it connect to the processes:

Wednesday August 2nd, 2017

  • Managed to accomplish "Use new "sections" from JSON to color packages"
    Use new "sections" from JSON to color packages:

Thursday August 3rd, 2017

  • Continue to refactor the code and update the comments
  • Further updated the sidebar for better look and feel Further updated the sidebar for better look and feel:

Week 12

Monday August 7th, 2017

*Fixed the issues for both "expandable views in the sidebar: first entry doesn't have the hover effect" and "probably shouldn't be a hover on associated packages? (there are not links and don't have tooltips)"

  • Managed to accomplish "When you click on a package, have it connect to the processes"
    Fixed the issues:

Wednesday August 9th, 2017

  • Changed the in focus toggle to take immediate effect when focus toggle is on
    Focus effect on immediately:

Thursday August 10th, 2017

  • Updated the visualization according to Fernando's comments
    Focus effect on immediately:
  • Continue to work on code refactoring according to Remi's comments

Friday August 11th, 2017

  • Continue to work on code refactoring according to Remi's comments

Week 13

Monday August 14th, 2017

*Fixed:

  • Remove '/' at beginning of non-top-level paths
  • Side bar show/hide icons are reversed
  • Add button to show all package edges
  • Resizing Window using $(window).resize() — Still need to review with Remi
  • Differentiate double-click & single-click functions Fixed the issues:

Tuesday August 15th, 2017

  • Code review with Remi

Wednesday August 15th, 2017

  • Tried to use append the nodes into another 'g' elements so they will be appearing on top of the edges, however, it is not able to achieve what we wanted, so I will take further investigation on it.
  • Setting the length of the timeline according to the max longest path for each tree diagram.
    Focus effect on immediately: Focus effect on immediately:

Friday August 18th, 2017

  • Tried out Gantt chart with current timestamps

Week 14

Monday August 21st, 2017

*Updates:

  • Added the run time when a process is clicked
  • Updated the mini timeline on the bottom to visualize the progress. So when this selected process is clicked, it will indicate where the entire experiment is up to when this process is completed Fixed the issues: Fixed the issues:

Tuesday August 22nd, 2017

  • Updates & Fixes:
  • On sidebar, when a process is clicked, add total time it took to run
  • If there is no timestamp, timeline will not show
  • Fixed Timeline label cutoff due to screen size differences
  • Enhancements and bugs fixing

Thursday August 24th, 2017

  • Continue to work on make visualization works for digits json with multiple runs

Friday August 18th, 2017

  • Display timeline when “Show Timeline” Button is click