Neural Networks & Deep Learning - BKJackson/BKJackson_Wiki GitHub Wiki

Autoencoders for dynamical physical models

Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders - Lee and Carlberg, 2019
Time-series machine-learning error models for approximate solutions to parameterized dynamical systems - Parish and Carlberg, 2019

Articles, Blogs, etc

A Recipe for Training Neural Networks - Andrej Karpathy, 4/25/2019. "Once you make it here you’ll have all the ingredients for success: You have a deep understanding of the technology, the dataset and the problem, you’ve set up the entire training/evaluation infrastructure and achieved high confidence in its accuracy, and you’ve explored increasingly more complex models, gaining performance improvements in ways you’ve predicted each step of the way. You’re now ready to read a lot of papers, try a large number of experiments, and get your SOTA results. Good luck!"
How to build a simple Neural Network from scratch with Python - without using a framework
MIT Technology Review of Deep Learning (2013) One of Ten Breakthrough Technologies.
Jeff Dean On Large-Scale Deep Learning At Google
Drum Sample Variational Autoencoder t-SNE Experiment (Video) This shows around 28k "drum samples", mostly one-shot hits, all less than 6 seconds, only the first second is analyzed. They come from a few different sample packs. I run them through two different 512-512 layer variational autoencoders, one with a 2d (position) bottleneck and another with 3d (color) bottleneck. I used librosa to extract the constant-q transform of each sound with 84 bins and 6 time steps making for a 504 element vector per sample. See also kylemcdonald.net.
Deep Learning and Convolutional Neural Networks for Image Recognition Adam Geitgey
Neural Network Architectures Eugenio Culurciello
Navigating the Unsupervised Learning Landscape Eugenio Culcurciello
OpenAI - Generative Models
OpenAI Gym A toolkit for developing and comparing reinforcement learning algorithms.
DeepMind Blog
DeepLearning.net
Deep Learning and Long-Term Investing
Summaries and notes on Deep Learning research papers Danny Britz

Types of Neural Networks

LSTM

Time Series Prediction with LSTM Recurrent Neural Networks in Python with Keras Jason Brownlee, July 21, 2016
Backpropagating an LSTM: A Numerical Example Aiden Gomez, April 17, 2016
RNN and LSTM Resources Handy list of resources and tutorials.
NTM-Lasagne A github library to create Neural Turing Machines (NTMs) in Theano using the Lasagne library
NTM-Lasagne: A Library for Neural Turing Machines in Lasagne Blog post by Tristan Deleu.
Awesome Recurrent Neural Networks A curated list of resources dedicated to recurrent neural networks
How to teach logic to your neural networks Preetham
Attention and Augmented Recurrent Neural Networks Chris Olah and Shan Carter, Google Brain, Sept. 8, 2016.
A Beginner’s Guide to Recurrent Networks and LSTMs By deeplearning4j

Visualization of Neural Networks

Visual Analysis for RNNs and LSTMs

TensorFlow

TensorFlow Official Home
[TensorFlow:Large-Scale Machine Learning on Heterogeneous Distributed Systems] (http://download.tensorflow.org/paper/whitepaper2015.pdf) Abadi et al. (2015) white paper by the Google TensorFlow team. Read this first
CS224D Lecture 7 - Introduction to TensorFlow Video. Presented & published on 19th Apr 2016.
TensorFlow Scan Examples
TensorFlow Playground Fun and games with TensorFlow.
Tensor Flow Tutorials Includes basic classifiers and NN.
TensorFlow with Rajat Monga Video. Rajat Monga, TensorFlow Technical Lead & Manager, Google
First Contact with TensorFlow Intro book by Jordi Torres.
TensorFlow Scan Examples Python notebook by Rob DiPietro.
The Good, Bad, & Ugly of TensorFlow A survey of six months rapid evolution (+ tips/hacks and code to fix the ugly stuff)
[Announcing SyntaxNet: The World's Most Accurate Parser (and Parsey McParseface)] The Ultimate List of TensorFlow Resources: Books, Tutorials, Libraries and More From HackerLists.
Recurrent Highway Networks for Tensorflow and Torch
TF Learn TF Learn is a simplified interface for TensorFlow, to get people started on predictive analytics and data mining. The library covers a variety of needs: from linear models to Deep Learning applications like text and image understanding.
Edward: A library for probabilistic modeling, inference, and criticism Builds on top of TensorFlow.

SyntaxNet & Parsey McParseface

SyntaxNet:an open-source neural network framework implemented in TensorFlow that provides a foundation for Natural Language Understanding (NLU) systems Parsey McParseface: an English parser trained by Google that you can use to analyze English text SyntaxNet - Github
Announcing SyntaxNet: The World’s Most Accurate Parser Google Research Blog, May 12, 2016, announcing SyntaxNet & Parsey McParseface. Bossy girls, Parser McParseface, and why deep learning is not just another fad Blog article by Pete Warden.

Torch & Lua

Torch Cheatsheet
Getting Started with Torch
Learn Lua
Torch7 Google Group For discussing bugs, etc.

Luarocks package management:

$ luarocks list
$ luarocks install image

Torch:

$ th
th> torch.Tensor{1,2,3}
th> dofile "file.lua"
th> ?
th> ? funcname
th> os.exit()  
$ th file.lua

Torch Tensor class
Torch Tensor slicing and indexing
Torch Storage
Touch Math Functions

Keras: Deep Learning library for Theano and TensorFlow

Keras docs Official docs.
Deep Learning and Keras with François Chollet Article and podcast.
Deep Learning with Keras EuroScipy 2016 From Lerio Maggio.
Introduction to Deep Learning with Keras - How to use the Keras Deep Learning library, Jan 9, 2019

Theano: Python library for fast multidimensional array math

Theano docs
GPU Computing with Theano James Bergstra talk at SciPy 2010. Good 20 minute intro to Theano.

ConvNetJS: Deep Learning in your browser

ConvNetJS Home Andrew Karpathy

Books

Deep Learning Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016)
Advanced Deep Learning with Python

Courses

Dive Into Deep Learning
Spring 2019 Full Stack Deep Learning Bootcamp - Hands-on program for developers familiar with the basics of deep learning
Introduction to Deep Learning - Prof. Matthias Nießner, TUM, 2019/2020
Stanford CS224d: Deep Learning for Natural Language Processing
Stanford CS231n: Convolutional Neural Networks for Visual Recognition
Oxford: Machine Learning & Neural Networks Course with Github practicals CSC321 Winter 2015: Introduction to Neural Networks Roger Grosse & Nitish Srivastava, University of Toronto. Based largely on lectures produced by Geoff Hinton.
Neural Networks Hugo Larochelle
2016 Deep Learning Summer School With lecture slides.

Research Groups & People

Montreal Institute for Learning Algorithms Yoshua Bengio's group at U. Montreal.
Alex Graves Pubs at U Toronto RNN, LSTM Through 2013 or 2014