EECS282 Advanced Topics in Machine Learning (Fall semester 2017)

Instructor

Miguel Á. Carreira-Perpiñán
Professor
Electrical Engineering and Computer Science
School of Engineering
University of California, Merced
mcarreira-perpinan-[at]-ucmerced.edu; 209-2284545
Office: 217, Science & Engineering Building 2

Office hours: Tuesdays 5:30-7:30pm (SE2-217).

Lectures: Mondays/Wednesdays 12-1:15pm (COB2-274).

Lab class: Tuesdays 7:30-10:20pm (Linux Lab, SE1-100).

Course web page: http://faculty.ucmerced.edu/mcarreira-perpinan/teaching/EECS282

Course description

The course reviews advanced topics in machine learning. Machine learning is the study of models and algorithms that learn information from data. Machine learning ideas underlie many algorithms in computer vision, speech processing, bioinformatics, robotics, computer graphics and other areas.

Prerequisites: prior knowledge about machine learning at the level of an introductory course (e.g. having taken CSE176 at UC Merced, or equivalent), or instructor approval.

Syllabus

Textbook

Required textbook:

• C. Bishop: Pattern Recognition and Machine Learning. Springer, 2006.
  The companion site for the book has additional materials (lecture slides, errata, etc.).
• T. J. Hastie, R. J. Tibshirani and J. H. Friedman: The Elements of Statistical Learning: Data Mining, Inference and Prediction, 2nd ed. Springer, 2009. This book is accessible online from within UC Merced.
  The companion site for the book has additional materials (lecture slides, errata, etc.).

Practical project

Implementation of the "learning-compression" (LC) algorithm for deep net quantization described here:

• Carreira-Perpiñán and Idelbayev: Model compression as constrained optimization, with application to neural nets. Part II: quantization, arXiv 2017

Schedule of presentations

• Aug 29: overview of machine learning (presenter: Miguel Á. Carreira-Perpiñán): lecture notes
• Aug 30: deep learning, applications in computer vision
  • Lai et al: Deep Laplacian pyramid networks for fast and accurate super-resolution, CVPR 2017 (presenter: Wei-Sheng Lai)
  • Li et al: Deep joint image filtering, ECCV 2016 (presenter: Yujun Li)
• Sep 5: spectral and nonlinear embedding methods (presenters: Ramin Raziperchikolaei, Yerlan Idelbayev): slides
• Sep 6: nonnegative matrix factorisation and tensors in machine learning (presenter: Suryabhan Singh Hada)
  • Cichocki et al: Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation, Wiley 2009
  • Kolda and Bader: Tensor decompositions and applications, SIAM Rev 2009
• Sep 11: model assessment 1 (presenter: Pooya Tavallali)
  • Chapter 7 in ESL, slides 1 and slides 2
• Sep 12: model assessment 2 (presenter: Pooya Tavallali); splines 1 (presenter: Arman Zharmagambetov)
  • Chapter 5 in ESL, slides
• Sep 13: splines 2 (presenter: Arman Zharmagambetov)
  • Chapter 5 in ESL, slides
• Sep 18: deep learning overview 1 (presenter: Yerlan Idelbayev): slides
• Sep 19: deep learning toolboxes (slides) and installation in the MERCED cluster
  • Theano (presenter: Yerlan Idelbayev)
  • Tensorflow (presenter: Yerlan Idelbayev, Arman Zharmagambetov)
  • MatConvNet (presenter: Suryabhan Singh Hada)
• Sep 20: deep learning overview 2 (presenter: Yerlan Idelbayev): slides
• Sep 25: neural net compression (presenter: Yerlan Idelbayev)
  • Carreira-Perpiñán: Model compression as constrained optimization, with application to neural nets. Part I: general framework, arXiv 2017
  • Carreira-Perpiñán and Idelbayev: Model compression as constrained optimization, with application to neural nets. Part II: quantization, arXiv 2017
• Sep 26: deep learning lab: neural net compression (presenter: Yerlan Idelbayev)
• Sep 27: neural net compression 2 (presenter: Yerlan Idelbayev)
  • Carreira-Perpiñán and Idelbayev: Model compression as constrained optimization, with application to neural nets. Part III: Pruning, arXiv 2017
• Oct 2: spatial data structures and nearest neighbour search 1 (presenter: Ramin Raziperchikolaei)
  • Samet: Foundations of multidimensional and metric data structures, Morgan Kaufmann 2006; slides
• Oct 4: visualisation of deep nets (presenter: Suryabhan Singh Hada)
  • Zeiler et al: Deconvolutional networks, CVPR 2010
  • Zeiler and Fergus: Visualizing and understanding convolutional networks, ECCV 2014
  • Simonyan et al: Deep inside convolutional networks: visualising image classification models and saliency maps, ICLR 2014 workshop
  • Mahendran and Vedaldi: Understanding deep image representations by inverting them, CVPR 2015
  • Dosovitskiy and Brox: Inverting visual representations with convolutional networks, CVPR 2016
  • Nguyen et al: Synthesizing the preferred inputs for neurons in neural networks via deep generator networks, NIPS 2016
  • Nguyen et al: Plug & play generative networks: conditional iterative generation of images in latent space, CVPR 2017
• Oct 9: deep net optimisation
  • Kingma and Ba: Adam: A Method for Stochastic Optimization, ICLR 2015 (presenter: Pooya Tavallali)
  • Srivastava et al: Dropout: a simple way to prevent neural networks from overfitting, JMLR 2014 (presenter: Arman Zharmagambetov)
• Oct 10: deep net architectures
  • Krizhevsky et al: ImageNet classification with deep convolutional neural networks, Comm. ACM 2017 (presenter: Arman Zharmagambetov)
  • Szegedy et al: Going deeper with convolutions, CVPR 2015 (presenter: Xueting Li)
  • He et al: Deep residual learning for image recognition, CVPR 2016 (presenter: Xueting Li)
• Oct 11: deep net optimisation (presenter: Jacob Rafati Heravi)
  • Ioffe and Szegedy: Batch normalization: accelerating deep network training by reducing internal covariate shift, ICML 2015
• Oct 16: deep nets and unsupervised learning (presenter: Xueqing Deng)
  • Weston et al: Deep learning via semi-supervised embedding, ICML 2008
  • Zhao et al: Stacked what-where auto-encoders, ICLR workshop 2016
  • Zhang et al: Augmenting supervised neural networks with unsupervised objectives for large-scale image classification, ICML 2016
• Oct 17: spatial data structures and nearest neighbour search 2 (presenter: Ramin Raziperchikolaei):
  • Samet: Foundations of multidimensional and metric data structures, Morgan Kaufmann 2006; slides
• Oct 18: deep nets:
  • Yang et al: Bridging collaborative filtering and semi-supervised learning: a neural approach for POI recommendation, KDD 2017 (presenter: Xueqing Deng)
  • Zhang et al: Deep spatio-temporal residual networks for citywide crowd flows prediction, AAAI 2017 (presenter: Xueqing Deng)
  • Huang et al: Densely connected convolutional networks, CVPR 2017 (presenter: Jacob Rafati Heravi)
• Oct 23: autoencoders (presenter: Pooya Tavallali)
  • Goodfellow et al: Deep learning, MIT Press 2016, chapter 14: Autoencoders
• Oct 24: recurrent neural nets (RNNs and LSTMs) (presenter: Shrishail Baligar)
  • Goodfellow et al: Deep learning, MIT Press 2016, chapter 10 Sequence modeling: recurrent and recursive nets
  • Blog entries: 1 and 2
• Oct 25: overview of reinforcement learning (presenter: Jacob Rafati Heravi)
  • Chapters 3, 4, 6, 8, 9 in Reinforcement learning: an introduction, 2nd ed., MIT Press 2017
  • Deep reinforcement learning: slides 1 and slides 2
• Oct 30: deep nets and applications to computer vision (presenter: Xueting Li)
  • Girshick et al: Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR 2014
  • Girshick: Fast R-CNN, ICCV 2015
  • Ren et al: Faster R-CNN: towards real-time object detection with region proposal networks, NIPS 2015
  • Gatys et al: Image style transfer using convolutional neural networks, CVPR 2016
  • Johnson et al: Perceptual losses for real-time style transfer and super-resolution, ECCV 2016
• Oct 31: deep nets and reinforcement learning (presenter: Jacob Rafati Heravi)
  • Mnih et al: Playing Atari with deep reinforcement learning, NIPS Deep Learning Workshop 2013
  • Mnih et al: Human-level control through deep reinforcement learning, Nature 2015
  • Silver et al: Mastering the game of go with deep neural networks and tree search, Nature 2016
• Nov 1: CPU, GPU and FPGA computation (presenter: Dong Li): slides
• Nov 6: deep nets and reinforcement learning (presenter: Shrishail Baligar)
  • Graves et al: Neural Turing Machines, arXiv 2014
  • Graves et al: Hybrid computing using a neural network with dynamic external memory, Nature 2016
• Nov 7: word embeddings (presenter: Arman Zharmagambetov)
  • Turney and Pantel: From frequency to meaning: vector space models of semantics, JAIR 2010
  • Bengio et al: A neural probabilistic language model, JMLR 2003
  • Mikolov et al: Efficient estimation of word representations in vector space, ICLR 2013
  • Mikolov et al: Distributed representations of words and phrases and their compositionality, NIPS 2013
  • Pennington et al: GloVe: global vectors for word representation, ACL 2014
  • Levy and Goldberg: Neural word embeddings as implicit matrix factorization, NIPS 2014
• Nov 8: recurrent neural nets (presenter: Shrishail Baligar)
  • Lukoševičius and Jaeger: Reservoir computing approaches to recurrent neural network training, Computer Science Review 2009
• Nov 13: deep nets and applications to computer vision (presenter: Xueqing Deng)
  • Chen et al: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs, IEEE PAMI 2017
  • Shelhamer et al: Fully convolutional networks for semantic segmentation, IEEE PAMI 2017
  • Farabet et al: Learning hierarchical features for scene labeling, IEEE PAMI 2013
• Nov 14: optimising nested systems using auxiliary coordinates (presenter: Pooya Tavallali): slides
  • Carreira-Perpiñán and Wang: Distributed optimization of deeply nested systems, AISTATS 2014 and arXiv 2012
  • Wang and Carreira-Perpiñán: The role of dimensionality reduction in classification, AAAI 2014
  • Carreira-Perpiñán and Raziperchikolaei: Hashing with binary autoencoders, CVPR 2015
  • Carreira-Perpiñán and Vladymyrov: A fast, universal algorithm to learn parametric nonlinear embeddings, NIPS 2015
  • Raziperchikolaei and Carreira-Perpiñán: Optimizing affinity-based binary hashing using auxiliary coordinates, NIPS 2016
  • Carreira-Perpiñán and Alizadeh: ParMAC: distributed optimisation of nested functions, with application to learning binary autoencoders, arXiv 2016
• Nov 15: recurrent neural nets (presenter: Shrishail Baligar)
  • Graves: Adaptive computation time for recurrent neural networks, arXiv 2017
• Nov 20: deep nets (presenter: Xueqing Deng)
  • Goyal et al: Accurate, large minibatch SGD: training ImageNet in 1 hour, arXiv 2017
  • Zhang et al: Understanding deep learning requires rethinking generalization, ICLR 2017
• Nov 21: Generative Adversarial Nets and applications (GANs)
• Tutorials: NIPS 2016 (and survey), CVPR 2017 (video), ICCV 2017 (presenter: Suryabhan Singh Hada and Xueting Li)
• Goodfellow et al: Generative Adversarial Nets, NIPS 2014 (presenter: Suryabhan Singh Hada)
• Salimans et al: Improved techniques for training GANs, NIPS 2016 (presenter: Suryabhan Singh Hada)
• Isola et al: Image-to-image translation with conditional adversarial nets, CVPR 2017 (presenter: Xueting Li)
• Nov 27: Generative Adversarial Nets and applications (GANs)
• Zhang et al: StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks, ICCV 2017 (presenter: Xueting Li)
• Zhu et al: Unpaired image-to-image translation using cycle-consistent adversarial networks, ICCV 2017 (presenter: Xueting Li)
• Dosovitskiy and Brox: Generating images with perceptual similarity metrics based on deep networks, NIPS 2016 (presenter: Suryabhan Singh Hada)
• Nov 28: neural net compression project presentations
• Nov 29: deep nets (presenter: Xueqing Deng)
• van den Oord et al: Conditional image generation with PixelCNN decoders, NIPS 2016
• Nagamine and Mesgarani: Understanding the representation and computation of multilayer perceptrons: a case study in speech recognition, ICML 2017

UC Merced | EECS | MACP's Home Page