Course Schedule, Reading and Assignments - randalburns/atidic-fall18 GitHub Wiki

(4 September 2018) Practical Deep Learning: How a combination of algorithms and GPUs rescued this tortured approach

• Reading:
  • Hinton et al., A Fast Learning Algorithm for Deep Belief Nets, Neural Computation 18, 1527–1554 (2006)
• Lecture: Lecture 1: Intro and How to Read a Paper
• Exercise: Configure Gigantum for GPU accelerated learning.

(6 September 2018) Practical Deep Learning: How a combination of algorithms and GPUs rescued this tortured approach

• Reading:
  • Hinton et al., ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012.
• Project 1 Deep Learning on GPUs

(11 September 2018) Roofline

• Reading: We will discuss the first paper. The second can be read more casually and for reference.
  • Williams et al. Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures, CACM
  • Lorenzo et al. Using an extended Roofline Model to understand data and thread affinities on NUMA systems, Annals for GPU and Mutlicore Programming, 1(1), 2014.

(13 September 2018) Tensor Processing Unit

• Reading: Amazing. Please spend some extra time looking at the history of RISC/CISC processors and be prepared to discuss how a return to a CISC approach is reasonable for a TPU.
  • Jouppi et al. In-Datacenter Performance Analysis of a Tensor Processing Unit. ISCA, 2017.

(18 September 2018) EIE

• Reading: We will discuss the first paper. The second can be read more casually and for reference.
  • S. Han et al. EIE: Efficient Inference Engine on Compressed Deep Neural Network, ISCA, 2016.
  • S. Han et al. Learning both weights and connections for efficient neural networks. NIPS, 2015.

(20 September 2018) NN+LSH

• Reading:
  • R Spring and A. Shrivastava. Scalable and Sustainable Deep Learning via Randomized Hashing. KDD, 2016
• Background Reading: not for discussion, but for context
  • Sections 3.3. and 3.4. Leskovec, Rajaraman, and Ullman, Mining of Massive Datasets, Cambridge Unviersity Press, 2014.

(25 September 2018) Presentations

(27 September 2018) Getting Big. HogWild!

• Reading:
  • F. Niu et al. Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. NIPS, 2011.

(2 October 2018)

• Reading:
  • Chen et al. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. arXiv:1512.01274v1, 3 Dec 2015
• Project 2 Reverse Engineering MLaaS

(4 October 2018)

• Reading:
  • M. Li et al. Scaling Distributed Machine Learning with the Parameter Server. OSDI, 2014.

(9 October 2018) Work Day

• No paper. Gather as teams.

(11 October 2018) Project 2 Presentations

• NOTE Class is in Remsen 101 . We have been moved by the trustees.

(16 October 2018) Class Cancelled

Sorry, I am ill and not able to make it to class today.

(18 October 2018) Forests I and Presentations II

Let's read this paper quite carefully. This is a new concept spacing so you will have to do some background reading on topics like bagging, gradient boosting and CART to really understand this paper. This is the first section about looking at the computational properties of trees. Keep an eye on parallelism as you read this paper.

• T. Chen and C. Guestrin. XGBoost: A Scalable Tree Boosting System. KDD 2016.

(23 October 2018) Forests II

• Reading:
  • Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. "LightGBM: A Highly Efficient Gradient Boosting Decision Tree". In Advances in Neural Information Processing Systems (NIPS), pp. 3149-3157. 2017.
• Final Project: One Page Summary due Friday 26 October 2018. Submit by email to Randal.

(25 October 2018) Tree-based approaches to NN

This paper relates to both readings on random forest (for comparable techniques) and LSH (for nearest neighbors).

• Reading:
  • M. Muja and D. Lowe. Scalable Nearest Neighbor Algorithms for High Dimensional Data. IEEE Trans. PAMI. 36(11), 2014.

(30 October 2018) Graph approaches to NN (and benchmarking)

Please read both papers. Benchmarking first. The Smal)l World Graphs paper is the current leader on the benchmarking site.

• Reading:
  • M. Aumiller, E. Bernhardsson, and A. Faithfull. ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms. SISAP, 2017.
  • Y. A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs. arXiv:1603.09320 [cs.DS]

(6 November 2018) PANENE (by Seth)

• Reading:
  • J. Jo, J. Seo, and J. D. Fekete. PANENE: A Progressive Algorithm for Indexing and Querying Approximate k-Nearest Neighbors. IEEE TVCG 2015.

(8 November 2018) Dense Linear Algebra

• Reading: CUBLAS. Let's get down in the weeds with Linear Algebra.
  • Dongarra et al. Accelerating Numerical Dense Linear Algebra Calculations with GPUs. Chapter of Numerical Computations with GPUs. Springer, Cham. 2014

(13 November 2018) Sparse Linear Algebra

• Reading:
  • A. Buluc and J. R. Gilbert. Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments. SIAM J. Sci. Comput., 34(4), 170 - 191, 2012

(15 November 2018) Project Pitches . (Delayed indefinitely)

• Talks: Prepare a 1 (or 2 slide) presentation that will last 4 minutes that expresses:
  • What research question does your project attempt to answer?
  • What experiment are you going to run to answer that question?
  • What is the expected outcome of the experiment?
  • What conclusion are you going to draw?
• Please add your slide(s) to https://docs.google.com/presentation/d/1TP2HhkKFaaWST4XTPgs1LmNm56qUm-EmSKEwH-692iE/edit?usp=sharing
  • Include a title slide (example provided)
  • Append your slides to the end of the deck (we will go in order, don't cut in line)
  • If you want to present in PPT/Keynote/Markdown or don't feel like Google slides is suitable, you can present from your own laptop.

(27 and 29 November) Clustering Week

Randal is at a conference and out this week. I'll need two volunteers to lead discussions on the following topics and papers.

• Tues: KMeans++: B. Bahmani et al. Scalable K-Means++. VLDB 2012.
• Thurs: Spectral Clustering: W. Y. Chen et al. Parallel Spectral Clustering in Distributed Systems. IEEE Trans PAMI, 33(3), 2011.