Processing math: 100%
\usepackagecolor\usepackage[matrix]xy\usepackageskak Skip to content →

Tag: Belfiore

Deep learning and toposes

Judging from this and that paper, deep learning is the string theory of the 2020s for geometers and representation theorists.

If you want to know quickly what neural networks really are, I can recommend the post demystifying deep learning.

The typical layout of a deep neural network has an input layer L0 allowing you to feed N0 numbers to the system (a vector v0RN0), an output layer Lp spitting Np numbers back (a vector vpRNp), and p1 hidden layers L1,,Lp1 where all the magic happens. The hidden layer Li has Ni virtual neurons, their states giving a vector viRNi.



Picture taken from Logical informations cells I

For simplicity let’s assume all neurons in layer Li are wired to every neuron in layer Li+1, the relevance of these connections given by a matrix of weights WiMNi+1×Ni(R).

If at any given moment the ‘state’ of the neural network is described by the state-vectors v1,,vp1 and the weight-matrices W0,,Wp, then an input v0 will typically result in new states of the neurons in layer L1 given by

v1=c0(W0.v0+v1)

which will then give new states in layer L2

v2=c1(W1.v1+v2)

and so on, rippling through the network, until we get as the output

vp=cp1(Wp1.vp1)

where all the ci are fixed smooth activation functions ci:RNi+1RNi+1.

This is just the dynamic, or forward working of the network.

The learning happens by comparing the computed output with the expected output, and working backwards through the network to alter slightly the state-vectors in all layers, and the weight-matrices between them. This process is called back-propagation, and involves the gradient descent procedure.

Even from this (over)simplified picture it seems doubtful that set valued (!) toposes are suitable to describe deep neural networks, as the Paris-Huawei-topos-team claims in their recent paper Topos and Stacks of Deep Neural Networks.

Still, there is a vast generalisation of neural networks: learners, developed by Brendan Fong, David Spivak and Remy Tuyeras in their paper Backprop as Functor: A compositional perspective on supervised learning (which btw is an excellent introduction for mathematicians to neural networks).

For any two sets A and B, a learner AB is a tuple (P,I,U,R) where

  • P is a set, a parameter space of some functions from A to B.
  • I is the interpretation map I:P×AB describing the functions in P.
  • U is the update map U:P×A×BP, part of the learning procedure. The idea is that U(p,a,b) is a map which sends a closer to b than the map p did.
  • R is the request map R:P×A×BA, the other part of the learning procedure. The idea is that the new element R(p,a,b)=a in A is such that p(a) will be closer to b than p(a) was.

The request map is also crucial is defining the composition of two learners AB and BC. Learn is the (symmetric, monoidal) category with objects all sets and morphisms equivalence classes of learners (defined in the natural way).

In this way we can view a deep neural network with p layers as before to be the composition of p learners
RN0RN1RN2RNp
where the learner describing the transition from the i-th to the i+1-th layer is given by the equivalence class of data (Ai,Bi,Pi,Ii,Ui,Ri) with
Ai=RNi, Bi=RNi+1, Pi=MNi+1×Ni(R)×RNi+1
and interpretation map for p=(Wi,vi+1)Pi
Ii(p,vi)=ci(Wi.vi+vi+1)
The update and request maps (encoding back-propagation and gradient-descent in this case) are explicitly given in theorem III.2 of the paper, and they behave functorial (whence the title of the paper).

More generally, we will now associate objects of a topos (actually just sheaves over a simple topological space) to a network op p learners
A0A1A2Ap
inspired by section I.2 of Topos and Stacks of Deep Neural Networks.

The underlying category will be the poset-category (the opposite of the ordering of the layers)
012p
The presheaf on a poset is a locale and in this case even the topos of sheaves on the topological space with p+1 nested open sets.
X=U0U1U2Up=
If the learner AiAi+1 is (the equivalence class) of the tuple (Ai,Ai+1,Pi,Ii,Ui,Ri) we will now describe two sheaves W and X on the topological space X.

W has as sections Γ(W,Ui)=p1j=iPi and the obvious projection maps as the restriction maps.

X has as sections Γ(X,Ui)=Ai×Γ(W,Ui) and restriction map to the next smaller open
ρii+1 : Γ(X,Ui)Γ(X,Ui+1)(ai,(pi,p))(pi(ai),p)
and other retriction maps by composition.

A major result in Topos and Stacks of Deep Neural Networks is that back-propagation is a natural transformation, that is, a sheaf-morphism XX.

In this general setting of layered learners we can always define a map on the sections of X (for every open Ui), Γ(X,Ui)Γ(X,Ui)
(a,(pi,p))(R(pi,ai,pi(ai)),(U(pi,ai,pi(ai)),p)
But, in order for this to define a sheaf-morphism, compatible with the restrictions, we will have to impose restrictions on the update and restriction maps of the learners, in general.

Still, in the special case of deep neural networks, this compatibility follows from the functoriality property of Backprop as Functor: A compositional perspective on supervised learning.

To be continued.

One Comment

Huawei and topos theory

Apart from the initiatives I mentioned last time, Huawei set up a long term collaboration with the IHES, the Huawei Young Talents Program.

“Every year, the Huawei Young Talents Program will fund on average 7 postdoctoral fellowships that will be awarded by the Institute’s Scientific Council, only on the basis of scientific excellence. The fellows will collaborate with the Institute’s permanent professors and work on topics of their interest.”

Over the next ten years, Huawei will invest 5 million euros in this program, and an additional 1 million euros goes into the creation of the ‘Huawei Chair in Algebraic Geometry’. It comes as no particular surprise that the first chairholder is Laurent Lafforgue.

At the launch of this Young Talents Program in November 2020, Lafforgue gave a talk on The creative power of categories: History and some new perspectives.

The latter part of the talk (starting at 47:50) clarifies somewhat Huawei’s interest in topos theory, and what Lafforgue (and others) hope to get out of their collaboration with the telecom company.

Clearly, Huawei is interested in deep neural networks, and if you can convince them your expertise is useful in that area, perhaps they’ll trow some money at you.

Jean-Claude Belfiore, another mathematician turned Huaweian, is convinced topos theory is the correct tool to study DNNs. Here’s his Huawei-clip from which it is clear he was originally hired to improve Huawei’s polar code.

At the 2018 IHES-Topos conference he gave the talk Toposes for Wireless Networks: An idea whose time has come, and recently he arXived the paper Topos and Stacks of Deep Neural Networks, written jointly with Daniel Bennequin. Probably, I’ll come back to this paper another time, for now, the nForum has this page on it.

Towards the end of his talk, Lafforgue suggests the idea of creating an institute devoted to toposes and their applications, endorsed by IHES and supported by Huawei. Surely he knows that the Topos Institute already exists.

And, if you wonder why Huawei trows money at IHES rather than your university, I leave you with Lafforgue’s parting words:

“IHES professors are able to think and evaluate for themselves, whereas most mathematicians just follow ‘group thinking'”

Ouch!

One Comment