Past DataSig Seminar

We study several matrix diffusion processes constructed from a unitary Brownian motion. In particular, we use the Stiefel fibration to lift the Brownian motion of the complex Grass- mannian to the complex Stiefel manifold and deduce a skew-product decomposition of the Stiefel Brownian motion. As an application, we prove asymptotic laws for the determinants of the block entries of the unitary Brownian motion.

Our understanding and ability to compute the solutions to nonlinear partial differential equations has been strongly curtailed by our inability to effectively parameterize the inertial manifold of their solutions.  I will discuss our ongoing efforts for using machine learning to advance the state of the art, both for developing a qualitative understanding of "turbulent" solutions and for efficient computational approaches.  We aim to learn parameterizations of the solutions that give more insight into the dynamics and/or increase computational efficiency. I will discuss our recent work using machine learning to develop models of the small scale behavior of spatio-temporal complex solutions, with the goal of maintaining accuracy albeit at a highly reduced computational cost relative to a full simulation.  References: https://www.pnas.org/content/116/31/15344 and https://arxiv.org/pdf/2102.01010.pdf 

A new explanation of geometric nature of the reservoir computing phenomenon is presented. Reservoir computing is understood in the literature as the possibility of approximating input/output systems with randomly chosen recurrent neural systems and a trained linear readout layer. Light is shed on this phenomenon by constructing what is called strongly universal reservoir systems as random projections of a family of state-space systems that generate Volterra series expansions. This procedure yields a state-affine reservoir system with randomly generated coefficients in a dimension that is logarithmically reduced with respect to the original system. This reservoir system is able to approximate any element in the fading memory filters class just by training a different linear readout for each different filter. Explicit expressions for the probability distributions needed in the generation of the projected reservoir system are stated and bounds for the committed approximation error are provided.

It was implicitly conjectured by Hambly-Lyons in 2010, which was made explicit by Chang-Lyons-Ni in 2018, that the length of a tree-reduced path with bounded variation can be recovered from its signature asymptotics. Apart from its intrinsic elegance, understanding such a phenomenon is also important for the study of signature lower bounds and may shed light on more general signature inversion properties. In this talk, we discuss how the idea of path development onto suitably chosen Lie groups can be used to study this problem as well as its rough path analogue.

In this talk, we present simple ideas to combine nonparametric approaches based on positive definite kernels with deep learning models. There are many good reasons for bridging these two worlds. On the one hand, we want to provide regularization mechanisms and a geometric interpretation to deep learning models, as well as a functional space that allows to study their theoretical properties (eg invariance and stability). On the other hand, we want to bring more adaptivity and scalability to traditional kernel methods, which are crucially lacking. We will start this presentation by introducing models to represent graph data, then move to biological sequences, and images, showing that our hybrid models can achieves state-of-the-art results for many predictive tasks, especially when large amounts of annotated data are not available. This presentation is based on joint works with Alberto Bietti, Dexiong Chen, and Laurent Jacob.

Deep convolutional networks have spectacular performances that remain mostly not understood. Numerical experiments show that they classify by progressively concentrating each class in separate regions of a low-dimensional space. To explain these properties, we introduce a concentration and separation mechanism with multiscale tight frame contractions. Applications are shown for image classification and statistical physics models of cosmological structures and turbulent fluids.

In this talk, I will study the properties of parametric estimators based on the Maximum Mean Discrepancy (MMD) defined by Briol et al. (2019). In a first time, I will show that these estimators are universal in the i.i.d setting: even in case of misspecification, they converge to the best approximation of the distribution of the data in the model, without ANY assumption on this model. This leads to very strong robustness properties. In a second time, I will show that these results remain valid when the data is not independent, but satisfy instead a weak-dependence condition. This condition is based on a new dependence coefficient, which is itself defined thanks to the MMD. I will show through examples that this new notion of dependence is actually quite general. This talk is based on published works, and works in progress, with Badr-Eddine Chérief Abdellatif (ENSAE Paris), Mathieu Gerber (University of Bristol), Jean-David Fermanian (ENSAE Paris) and Alexis Derumigny (University of Twente):

http://arxiv.org/abs/1912.05737

http://proceedings.mlr.press/v118/cherief-abdellatif20a.html

http://arxiv.org/abs/2006.00840

https://arxiv.org/abs/2010.00408

https://cran.r-project.org/web/packages/MMDCopula/

Any binary classifier (or score-function) can be used to define a dissimilarity between two distributions of points with positive and negative labels. Actually, many well-known distribution-dissimilarities are classifier-based dissimilarities: the total variation, the KL- or JS-divergence, the Hellinger distance, etc. And many recent popular generative modelling algorithms compute or approximate these distribution-dissimilarities by explicitly training a classifier: eg GANs and their variants. After a brief introduction to these classifier-based dissimilarities, I will focus on the influence of the classifier's capacity. I will start with some theoretical considerations illustrated on maximum mean discrepancies --a weak form of total variation that has grown popular in machine learning-- and then focus on deep feed-forward networks and their vulnerability to adversarial examples. We will see that this vulnerability is already rooted in the design and capacity of our current networks, and will discuss ideas to tackle this vulnerability in future.

Driven by the need for principled extraction of features from time series, we introduce the iterated-sums signature over any commutative semiring. The case of the tropical semiring is a central, and our motivating, example, as it leads to features of (real-valued) time series that are not easily available using existing signature-type objects.

This is joint work with Kurusch Ebrahimi-Fard (NTNU Trondheim) and Nikolas Tapia (WIAS Berlin).

We study the minimum Wasserstein distance from the empirical measure to a space of probability measures satisfying linear constraints. This statistic can naturally be used in a wide range of applications, for example, optimally choosing uncertainty sizes in distributionally robust optimization, optimal regularization, testing fairness, martingality, among many other statistical properties. We will discuss duality results which recover the celebrated Kantorovich-Rubinstein duality when the manifold is sufficiently rich and associated test statistics as the sample size increases. We illustrate how this relaxation can beat the statistical curse of dimensionality often associated to empirical Wasserstein distances.

The talk builds on joint work with S. Ghosh, Y. Kang, K. Murthy, M. Squillante, and N. Si.

Topological features as computed via persistent homology offer a non-parametric approach to robustly capture multi-scale connectivity information of complex datasets. This has started to gain attention in various machine learning applications. Conventionally, in topological data analysis, this method has been employed as an immutable feature descriptor in order to characterize topological properties of datasets. In this talk, however, I will explore how topological features can be directly integrated into deep learning architectures. This allows us to impose differentiable topological constraints for preserving the global structure of the data space when learning low-dimensional representations.

The signature of a path in Euclidean space resides in the tensor algebra of that space; it is obtained by systematic iterated integration of the components of the given path against one another. This straightforward definition conceals a host of deep theoretical properties and impressive practical consequences. In this talk I will describe the homotopical origins of path signatures, their subsequent application to stochastic analysis, and how they facilitate efficient machine learning in topological data analysis. This last bit is joint work with Ilya Chevyrev and Harald Oberhauser.

The random initialisation of Artificial Neural Networks (ANN) allows one to describe, in the functional space, the limit of the evolution of ANN when their width tends towards infinity. Within this limit, an ANN is initially a Gaussian process and follows, during learning, a gradient descent convoluted by a kernel called the Neural Tangent Kernel.

Connecting neural networks to the well-established theory of kernel methods allows us to understand the dynamics of neural networks, their generalization capability. In practice, it helps to select appropriate architectural features of the network to be trained. In addition, it provides new tools to address the finite size setting.

The concept of path signatures has been widely used in several areas of pure mathematics including in applications to data science. However, we remain unable to answer even the most basic questions about it. For instance, how to fully characterise the set of (untruncated) signatures of bounded variation paths? Can certain norms on signatures be related to the length of a path, like in Fourier isometry? In this talk, we will review some known results, explain the open problems and discuss their difficulties.

We present a method for obtaining approximate solutions to the problem of optimal execution, based on a signature method. The framework is general, only requiring that the price process is a geometric rough path and the price impact function is a continuous function of the trading speed. Following an approximation of the optimisation problem, we are able to calculate an optimal solution for the trading speed in the space of linear functions on a truncation of the signature of the price process. We provide strong numerical evidence illustrating the accuracy and flexibility of the approach. Our numerical investigation both examines cases where exact solutions are known, demonstrating that the method accurately approximates these solutions, and models where exact solutions are not known. In the latter case, we obtain favourable comparisons with standard execution strategies.

The heart of modern machine learning (ML) is the approximation of high dimensional functions. Traditional approaches, such as approximation by piecewise polynomials, wavelets, or other linear combinations of fixed basis functions, suffer from the curse of dimensionality (CoD). We will present a mathematical perspective of ML, focusing on the issue of CoD. We will discuss three major issues: approximation theory and error analysis of modern ML models, dynamics and qualitative behavior of gradient descent algorithms, and ML from a continuous viewpoint. We will see that at the continuous level, ML can be formulated as a series of reasonably nice variational and PDE-like problems. Modern ML models/algorithms, such as the random feature and two-layer and residual neural network models, can all be viewed as special discretizations of such continuous problems. We will also present a framework that is suited for analyzing ML models and algorithms in high dimension, and present results that are free of CoD. Finally, we will discuss the fundamental reasons that are responsible for the success of modern ML, as well as the subtleties and mysteries that still remain to be understood.

Abstract: Gradient descent is known to converge quickly for convex objective functions, but it can be trapped at local minimums. On the other hand, Langevin dynamic can explore the state space and find global minimums, but in order to give accurate estimates, it needs to run with small discretization step size and weak stochastic force, which in general slows down its convergence. This work shows that these two algorithms can “collaborate” through a simple exchange mechanism, in which they swap their current positions if Langevin dynamic yields a lower objective function. This idea can be seen as the singular limit of the replica-exchange technique from the sampling literature. We show that this new algorithm converges to the global minimum linearly with high probability, assuming the objective function is strongly convex in a neighbourhood of the unique global minimum. By replacing gradients with stochastic gradients, and adding a proper threshold to the exchange mechanism, our algorithm can also be used in online settings. This is joint work with Xin Tong at National University of Singapore.

We introduce constrained variational principles for fluid dynamics on rough paths. The advection of the fluid is constrained to be the sum of a vector field which represents coarse-scale motion and a rough (in time) vector field which parametrizes fine-scale motion. The rough vector field is regarded as fixed and the rough partial differential equation for the coarse-scale velocity is derived as a consequence of being a critical point of the action functional.

The action functional is perturbative in the sense that if the rough vector f ield is set to zero, then the corresponding variational principle agrees with the reduced (to the vector fields) Euler-Poincare variational principle introduced in Holm, Marsden and Ratiu (1998). More precisely, the Lagrangian in the action functional encodes the physics of the fluid and is a function of only the coarse-scale velocity. 

By parametrizing the fine-scales of fluid motion with a rough vector field, we preserve the pathwise nature of deterministic fluid dynamics and establish a flexible framework for stochastic parametrization schemes. The main benefit afforded by our approach is that the system of rough partial differential equations we derive satisfy essential conservation laws, including Kelvin’s circulation theorem. This talk is based on recent joint work with Dan Crisan, Darryl Holm, and Torstein Nilssen.

Abstract: The signature of a path is a sequence of iterated coordinate integrals along the path. We aim at reconstructing a path from its signature. In the special case of lattice paths, one can obtain exact recovery based on a simple algebraic observation. For general continuously differentiable curves, we develop an explicit procedure that allows to reconstruct the path via piecewise linear approximations. The errors in the approximation can be quantified in terms of the level of signature used and modulus of continuity of the derivative of the path. The main idea is philosophically close to that for the lattice paths, and this procedure could be viewed as a significant generalisation. A key ingredient is the use of a symmetrisation procedure that separates the behaviour of the path at small and large scales.We will also discuss possible simplifications and improvements that may be potentially significant. Based on joint works with Terry Lyons, and also with Jiawei Chang, Nick Duffield and Hao Ni.

Abstract: Sequential and temporal data arise in many fields of research, such as quantitative finance, medicine, or computer vision. We will be concerned with a novel approach for sequential learning, called the signature method, and rooted in rough path theory. Its basic principle is to represent multidimensional paths by a graded feature set of their iterated integrals, called the signature. On the one hand, this approach relies critically on an embedding principle, which consists in representing discretely sampled data as paths, i.e., functions from [0,1] to R^d. We investigate the influence of embeddings on prediction accuracy with an in-depth study of three recent and challenging datasets. We show that a specific embedding, called lead-lag, is systematically better, whatever the dataset or algorithm used. On the other hand, in order to combine signatures with machine learning algorithms, it is necessary to truncate these infinite series. Therefore, we define an estimator of the truncation order and prove its convergence in the expected signature model.