Numerical optimization is an indispensable tool of modern data analysis, and there are many optimization problems where it is difficult or impossible to compute the full gradient of the objective function. The field of derivative free optimization (DFO) addresses these cases by using only function evaluations, and has wide-ranging applications from hyper-parameter tuning in machine learning to PDE-constrained optimization.

We present two projects that attempt to scale DFO techniques to higher dimensions.  The first method converges slowly but works in very high dimensions, while the second method converges quickly but doesn't scale quite as well with dimension.  The first-method is a family of algorithms called "stochastic subspace descent" that uses a few directional derivatives at every step (i.e. projections of the gradient onto a random subspace). In special cases it is related to Spall's SPSA, Gaussian smoothing of Nesterov, and block-coordinate descent. We provide convergence analysis and discuss Johnson-Lindenstrauss style concentration.  The second method uses conventional interpolation-based trust region methods which require large ill-conditioned linear algebra operations.  We use randomized linear algebra techniques to ameliorate the issues and scale to larger dimensions; we also use a matrix-free approach that reduces memory issues.  These projects are in collaboration with David Kozak, Luis Tenorio, Alireza Doostan, Kevin Doherty and Katya Scheinberg.

I will discuss the use of partitioned schemes for neural networks. This work is in the tradition of multrate numerical ODE methods in which different components of system are evolved using different numerical methods or with different timesteps. The setting is the training tasks in deep learning in which parameters of a hierarchical model must be found to describe a given data set. By choosing appropriate partitionings of the parameters some redundant computation can be avoided and we can obtain substantial computational speed-up. I will demonstrate the use of the procedure in transfer learning applications from image analysis and natural language processing, showing a reduction of around 50% in training time, without impairing the generalization performance of the resulting models. This talk describes joint work with Tiffany Vlaar.

A common observation that wider (in the number of hidden units/channels/attention heads) neural networks perform better motivates studying them in the infinite-width limit.

Remarkably, infinitely wide networks can be easily described in closed form as Gaussian processes (GPs), at initialization, during, and after training—be it gradient-based, or fully Bayesian training. This provides closed-form test set predictions and uncertainties from an infinitely wide network without ever instantiating it (!).

These infinitely wide networks have become powerful models in their own right, establishing several SOTA results, and are used in applications including hyper-parameter selection, neural architecture search, meta learning, active learning, and dataset distillation.

The talk will provide a high-level overview of our work at Google Brain on infinite-width networks. In the first part I will derive core results, providing intuition for why infinite-width networks are GPs. In the second part I will discuss challenges and solutions to implementing and scaling up these GPs. In the third part, I will conclude with example applications made possible with infinite width networks.

The talk does not assume familiarity with the topic beyond general ML background.

In this talk we will present some work in progress generalising Lubin-Tate formal group laws to higher dimensions. There have been some other generalisations, but ours is different in that the ring over which the formal group law is defined changes as the dimension increases. We will state some conjectures about these formal group laws, including their relationship to the Drinfeld tower over the p-adic upper half plane, and provide supporting evidence for these conjectures.

Let F be a p-adic field, and k an algebraically closed field of characteristic l different from p. In this talk, we will first give a category decomposition of Rep_k(SL_n(F)), the category of smooth k-representations of SL_n(F), with respect to the GL_n(F)-equivalent supercuspidal classes of SL_n(F), which is not always a block decomposition in general. We then give a block decomposition of the supercuspidal subcategory, by introducing a partition on each GL_n(F)-equivalent supercuspidal class through type theory, and we interpret this partition by the sense of l-blocks of finite groups. We give an example where a block of Rep_k(SL_2(F)) is defined with several SL_2(F)-equivalent supercuspidal classes, which is different from the case where l is zero. We end this talk by giving a prediction on the block decomposition of Rep_k(A) for a general p-adic group A.

Fluids sculpt many of the shapes we see in the world around us. We present a new mathematical model describing the shape evolution of a body that dissolves or melts under gravitationally stable buoyancy-driven convection, driven by thermal or solutal transfer at the solid-fluid interface. For high Schmidt number, the system is reduced to a single integro-differential equation for the shape evolution. Focusing on the particular case of a cone, we derive complete predictions for the underlying self-similar shapes, intrinsic scales and descent rates. We will present the results of new laboratory experiments, which show an excellent match to the theory. By analysing all initial power-law shapes, we uncover a surprising result that the tips of melting or dissolving bodies can either sharpen or blunt with time subject to a critical condition.