• Speaker: Helmut Bolcskei, ETH Zurich
• Title: Fundamental limits of learning in deep neural networks; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 08/20/2020
• Time: 2:30 PM - 3:30 PM
• Place: 

(Part of One World MINDS seminar: https://sites.google.com/view/minds-seminar/home) \[ \] We develop a theory that allows to characterize the fundamental limits of learning in deep neural networks. Concretely, we consider Kolmogorov-optimal approximation through deep neural networks with the guiding theme being a relation between the epsilon-entropy of the hypothesis class to be learned and the complexity of the approximating network in terms of connectivity and memory requirements for storing the network topology and the quantized weights and biases. The theory we develop educes remarkable universality properties of deep networks. Specifically, deep networks can Kolmogorov-optimally learn essentially any hypothesis class. In addition, we find that deep networks provide exponential approximation accuracy—i.e., the approximation error decays exponentially in the number of non-zero weights in the network—of widely different functions including the multiplication operation, polynomials, sinusoidal functions, general smooth functions, and even one-dimensional oscillatory textures and fractal functions such as the Weierstrass function, both of which do not have any known methods achieving exponential approximation accuracy. We also show that in the approximation of sufficiently smooth functions finite-width deep networks require strictly smaller connectivity than finite-depth wide networks. We conclude with an outlook on the further role our theory could play.

• Speaker: Daniel Potts, TU Chemnitz
• Title: High dimensional approximation with trigonometric polynomials; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 09/03/2020
• Time: 2:30 PM - 3:30 PM
• Place: 

(Part of One World MINDS seminar: https://sites.google.com/view/minds-seminar/home) \[ \] In this talk, we present fast Fourier based methods for the approximation of multivariate functions. Our aim is to learn the support of the Fourier coefficients in the frequency domain of high-dimensional functions. We are interested in two different approximation scenarios. The first case is black-box approximation where the user is allowed to sample the unknown function at any point and in the second case we are working with fixed scattered data. For black-box approximation we employ quasi Monte-Carlo methods on rank-1 lattice points. The fast algorithms are then based on one-dimensional fast Fourier transforms (FFT). In the second case, which is much more difficult, we will couple truncated ANOVA (analysis of variance) decompositions with the fast Fourier transform on nonequispaced data (NFFT). In both cases, we present error estimates and numerical results. The presented methods can be understood as sparse high dimensional FFT’s. This talk based on joint work with Lutz Kämmerer, Michael Schmischke, Manfred Tasche, and Toni Volkmer.

• Speaker: Mauro Maggioni, Johns Hopkins University
• Title: Learning Interaction laws in particle- and agent-based systems; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 09/17/2020
• Time: 2:30 PM - 3:30 PM
• Place: 

Interacting agent-based systems are ubiquitous in science, from modeling of particles in Physics to prey-predator and colony models in Biology, to opinion dynamics in economics and social sciences. Oftentimes the laws of interactions between the agents are quite simple, for example they depend only on pairwise interactions, and only on pairwise distance in each interaction. We consider the following inference problem for a system of interacting particles or agents: given only observed trajectories of the agents in the system, can we learn what the laws of interactions are? We would like to do this without assuming any particular form for the interaction laws, i.e. they might be “any” function of pairwise distances. We consider this problem both the mean-field limit (i.e. the number of particles going to infinity) and in the case of a finite number of agents, with an increasing number of observations, albeit in this talk we will mostly focus on the latter case. We cast this as an inverse problem, and study it in the case where the interaction is governed by an (unknown) function of pairwise distances. We discuss when this problem is well-posed, and we construct estimators for the interaction kernels with provably good statistically and computational properties. We measure their performance on various examples, that include extensions to agent systems with different types of agents, second-order systems, and families of systems with parametric interaction kernels. We also conduct numerical experiments to test the large time behavior of these systems, especially in the cases where they exhibit emergent behavior. This is joint work with F. Lu, J.Miller, S. Tang and M. Zhong.

• Speaker: Rene Vidal, Johns Hopkins University
• Title: On the Regularization Properties of Structured Dropout; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 10/01/2020
• Time: 2:30 PM - 3:30 PM
• Place: Online (virtual meeting)

Dropout and its extensions (e.g. DropBlock and DropConnect) are popular heuristics for training neural networks, which have been shown to improve generalization performance in practice. However, a theoretical understanding of their optimization and regularization properties remains elusive. This talk will present a theoretical analysis of several dropout-like regularization strategies, all of which can be understood as stochastic gradient descent methods for minimizing a certain regularized loss. In the case of single hidden-layer linear networks, we will show that Dropout and DropBlock induce nuclear norm and spectral k-support norm regularization, respectively, which promote solutions that are low-rank and balanced (i.e. have factors with equal norm). We will also show that the global minimizer for Dropout and DropBlock can be computed in closed form, and that DropConnect is equivalent to Dropout. We will then show that some of these results can be extended to a general class of Dropout-strategies, and, with some assumptions, to deep non-linear networks when Dropout is applied to the last layer.

• Speaker: Reinhard Heckel, Technical University of Munich
• Title: Provable Image Recovery with Untrained Convolutional Neural Networks; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 10/15/2020
• Time: 2:30 PM - 3:30 PM
• Place: Online (virtual meeting)
• Contact: Olga Turanova (turanova@msu.edu)

Convolutional Neural Networks are highly successful tools for image recovery and restoration. A major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images—so strong that they enable image recovery without any training data. A surprising observation that highlights those prior assumptions is that one can remove noise from a corrupted natural image by simply fitting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to the noisy image. In this talk, we discuss a simple un-trained convolutional network, called the deep decoder, that provably enables image denoising and regularization of inverse problems such as compressive sensing with excellent performance. We formally characterize the dynamics of fitting this convolutional network to a noisy signal and to an under-sampled signal, and show that in both cases early-stopped gradient descent provably recovers the clean signal. Finally, we discuss our own numerical results and numerical results from another group demonstrating that un-trained convolutional networks enable magnetic resonance imaging from highly under-sampled measurements, achieving results surprisingly close to trained networks, and outperforming classical untrained methods.

• Speaker: Yang Wang, Hong Kong University of Science and Technology
• Title: Improving Generative Adversarial Nets (GAN) Using VGrow; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 10/29/2020
• Time: 4:30 AM - 5:30 AM
• Place:  (Virtual Meeting Link)
• Contact: Olga Turanova (turanova@msu.edu)

(Note the unusual time: 4:30pm Shanghai, 10:30am Paris.) \[ \] Generative Adversarial Nets (GAN) have been one of the most exciting developments in machine learning and AI. But training of GAN is highly nontrivial. In this talk I will give an introduction to GAN, and propose a framework to learn deep generative models via Variational Gradient Flow (Vgrow) on probability measure spaces. Connections of our proposed VGrow method with other popular methods, such as VAE, GAN and flow-based methods, have been established in this framework, gaining new insights of deep generative learning.

• Speaker: Yonina Eldar, Weizmann Institute of Science
• Title: Deep Analog-to-Digital Compression: Tasks, Structures, and Models; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 11/19/2020
• Time: 2:30 PM - 3:30 PM
• Place:  (Virtual Meeting Link)
• Contact: Olga Turanova (turanova@msu.edu)

The famous Shannon-Nyquist theorem has become a landmark in the development of digital signal and image processing. However, in many modern applications, the signal bandwidths have increased tremendously, while the acquisition capabilities have not scaled sufficiently fast. Consequently, conversion to digital has become a serious bottleneck. Furthermore, the resulting digital data requires storage, communication and processing at very high rates which is computationally expensive and requires large amounts of power. In the context of medical imaging sampling at high rates often translates to high radiation dosages, increased scanning times, bulky medical devices, and limited resolution. In this talk, we present a framework for sampling and processing a large class of wideband analog signals at rates far below Nyquist in space, time and frequency, which allows to dramatically reduce the number of antennas, sampling rates and band occupancy. Our framework relies on exploiting signal structure and the processing task. We consider applications of these concepts to a variety of problems in communications, radar and ultrasound imaging and show several demos of real-time sub-Nyquist prototypes including a wireless ultrasound probe, sub-Nyquist MIMO radar, super-resolution in microscopy and ultrasound, cognitive radio, and joint radar and communication systems. We then discuss how the ideas of exploiting the task, structure and model can be used to develop interpretable model-based deep learning methods that can adapt to existing structure and are trained from small amounts of data. These networks achieve a more favorable trade-off between increase in parameters and data and improvement in performance, while remaining interpretable.

• Speaker: Ding-Xua Zhou, City University of Hong Kong
• Title: Theory of Deep Convolutional Neural Networks; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 12/03/2020
• Time: 3:30 AM - 4:30 AM
• Place: Online (virtual meeting) (Virtual Meeting Link)
• Contact: Olga Turanova (turanova@msu.edu)

(Note the unusual time: 4:30pm Shanghai, 10:30am Paris.) Deep learning has been widely applied and brought breakthroughs in speech recognition, computer vision, natural language processing, and many other domains. The involved deep neural network architectures and computational issues have been well studied in machine learning. But there lacks a theoretical foundation for understanding the modelling, approximation or generalization ability of deep learning models with network architectures. Here we are interested in deep convolutional neural networks (CNNs) with convolutional structures. The convolutional architecture gives essential differences between deep CNNs and fully-connected neural networks, and the classical approximation theory for fully-connected networks developed around 30 years ago does not apply. This talk describes an approximation theory of deep CNNs associated with the rectified linear unit (ReLU) activation function. In particular, we prove the universality of deep CNNs, meaning that a deep CNN can be used to approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough. We also show that deep CNNs perform at least as well as fully-connected neural networks for approximating general functions, and much better for approximating radial functions in high dimensions.

• Speaker: David P. Woodruff , Carnegie Mellon University
• Title: TBA; zoom link @ https://sites.google.com/view/minds-seminar/home
• Date: 12/17/2020
• Time: 2:30 PM - 3:30 PM
• Place:  (Virtual Meeting Link)
• Contact: Olga Turanova (turanova@msu.edu)

No abstract available.