ICLR14 Workshop: A Szlam: Unsupervised Feature Learning by Deep Sparse Coding

What this episode is about

The presentation discusses an ambitious project on unsupervised feature learning through deep sparse coding, focusing on developing a model for images that captures translation and deformation invariance while enabling composition of objects. Despite challenges in execution, the concept aims to create a parameterization of the object space and a composition operator to generate new objects from existing parts, drawing inspiration from various prior works in both image and natural language processing.

Key takeaways

• Ambitious plans often give way to pedestrian realities, reminding us that even dreams require practical steps.
• Modeling images as compositions identifies a challenge of combinatorial explosion—physics of creativity meets the limits of computation.
• Embedding objects in a shared space reveals the scale-free nature of images, where every macro is a micro waiting to emerge.
• Recurrent networks can transform into single-space mappings, elegantly merging inputs and outputs into a seamless creative flow.
• Pooling as trainable dimension reduction showcases AI's adaptability, cleverly addressing complexity without sacrificing the essence of the original.

Who speaks in this episode?

• Arthur (Host) · 13 min speaking

What is discussed, and when

1. 0:00 – 3:00: Leo Ju, Ronald Kolobov, Bill Bowman, Jensen Huang
2. 9:00 – 12:00: Naomi Lynn Lim

Notable quotes from "ICLR14 Workshop: A Szlam: Unsupervised Feature Learning by Deep Sparse Coding"

• There is number two's paper which will be very which we drew a lot of inspiration from and ronald kolobar's natural language from scratch and in fact there's even several things in this conference for example.

  about Leo Ju · at 2:11

• What was the dimension reduction so again this is the key thing we used essentially dr lim and you had to very carefully engineer what are the positive things that are supposed to be close together and what are the things that.

  about Naomi Lynn Lim · at 9:20

Who and what is mentioned in this episode?

People

• Naomi Lynn Lim — Person · Referenced method/author (technique used) · 2 mentions · first at 9:23
  Referenced in connection with a carefully engineered contrastive/neighbor-based objective (“dr lim ...”) used to make the dimension reduction work without collapse.
• Leo Ju — Person · Referenced researcher (prior work influence) · 1 mention · first at 2:08
  Mentioned as having a relevant paper on object/image composition; used as a key inspiration for the approach discussed.
• Ronald Kolobov — Person · Referenced author (prior work influence) · 1 mention · first at 2:17
  Cited as a proximal influence via a work described as “natural language from scratch,” used as inspiration for compositional/semantic modeling.
  Be very which we drew a lot of inspiration from and ronald kolobar's natural language from scratch and in fact there's even several things in this conference for example bowman and also huang in this.
• Bill Bowman — Person · Referenced conference presenter (related work) · 1 mention · first at 2:27
  Cited as someone at the same conference using compositional methods to model semantics, aligning with the talk’s theme.
  Fact there's even several things in this conference for example bowman and also huang in this conference where they use compositional you know you want to do some sort of semantics via composition by the way and of course the history.
• Jensen Huang — Person · Referenced conference presenter (related work) · 1 mention · first at 2:27
  Mentioned alongside Bowman as another conference participant applying compositional approaches to semantics.
  Fact there's even several things in this conference for example bowman and also huang in this conference where they use compositional you know you want to do some sort of semantics via composition by the way and of course the history.

Full transcript with timestamps

0:15 bit okay so um first the the the main author is who couldn't come so i'm giving this um i'm arthur um okay so uh this is unfortunately one of these one of these uh things where we wanted to do something big and it didn't quite work out as well as we want um so i think there's one really nice idea even though it didn't work it is still really nice and i hope that in the future it will work so we want to do something ambitious what actually happened was less ambitious so most of the time i'm going to talk about the dream rather than the the pedestrian reality so okay so what do we want we wanted a model of images that is translation and information deformation variant but composition and variant and so this is kind of a i guess a pretty standard thing in natural language models it's i mean seems to be gaining some steam in for images and for objects as well but what you want is a parameterization of

1:17 the space of objects or some sort of embedding of the space of objects and a composition operator so that you can take you know a piece of an object a piece of an object a piece of an object piece of object apply the composition operator each of these pieces is embedded in the object space you arrange them in the right way you hit it with the condition operator you get another object in the space um so mature the outputs of the parts of an object arrange spatially fed into fed into the composition operator give back another object and of course you have a combinatorial problem here because you know you know each time you talk about a composition operator you you mass you've multiplied the size of your space by the number of things you're composing um so again this is i mean this is this is not a particularly new idea for objects there's leo juice paper um there is number two's paper which will

2:14 be very which which we drew a lot of inspiration from and ronald kolobar's natural language from scratch and in fact there's even several things in this conference um for example um bowman and also huang in this conference where they use compositional you know you want to do some sort of semantics via composition um by the way and of course the history is much longer than that i just i'm naming some proximal um influences okay so so what are we talking about composition we work we're talking about the you know if we're going to talk about images we have a 2d grid and we assume that we have some sort of object space which we'll call o so because we're used to it we'll map o and rd and what we're going to want is an embedding of all objects into our d that's o o has the entire space of all the objects and you have a mapping from uh o to four or the nine depending on how your grid is arranged or some other product of

3:18 o back into o and again mapping from a product of a space back into space is a concatenation operator and the way we think about it is you have a dimension reduction on each of the say four pieces let's call it g and then you re-concatenate or if you want a slightly more powerful model you have but you can kennedy first and then you follow with the dimension reduction so schematically it looks like this you have i can't is there a certain point well anyway so i'll kind of look over here uh so you have you have your four featured vectors each of these b's is an element of o and they're arranged um they're arranged on the grid let's say you do a dimension reduction via g that gives you four new vectors that are arranged on the grid and then you concatenate them and that big vector you get back um is the new guy in o and hopefully uh the the swish dimension is the big dimension over ford in this case

4:19 so you map right back into the same space okay so again this is just a recurrent network right because the the inputs are just fed right back into the outputs just at one scale up um another way of saying it and we'll talk maybe a little bit more about this in a couple of slides is that uh you think of cooling as a trainable dimension reduction so it's not entirely obvious from this it is there but it's really messy when when we talk about the way we actually build a model in a few slides then they'll be more there but you can think of pooling as a trainable dimension reduction and the point of the pooling is to you know staunch the combinatorial explosion um again what we have is especially recurrent lowering compact um so this now we're getting to the one idea which i think is a new idea and is actually pretty interesting um okay sorry so let's see okay so one idea which is uh which i think is

5:23 actually pretty interesting which is that if you have a recurrent map like this if you have a some sort of recurrence i guess instead of thinking about it as a recurrent thing you can think about as a mapping from o four to o there's just there's no recurrence there's no inputs and outputs there's just one space right at the same space that you're taking your inputs from the same space you're doing you're getting outputs from the only thing you need to learn is an embedding and the composition map mapping so so you kind of uh you squish all the recurrence into one piece um so this is really sensible in my mind for images images are in some sense scale free if you look at any scale you can if you talk about objects at any scale you can think of them as subject of sub objects at a smaller scale this isn't entirely always obvious for natural language things you may not this may not be a reasonable uh

6:16 intuition but for images it seems to be a very reasonable intuition like if you take four objects you mush them together into a super object well that also is an object and you have essentially a scale so this idea that there's no recurrence you just have one space and everything that lives in the same space every object every scale that's in the same space seems seems to make sense um and you could have some sort of optimization where you alternate between updates of the optimal targets and updates of them of the mapping f um the problem is and the reason why we failed to a large extent is it's really hard to choose a loss function or an optimization that just doesn't collect the system so you trade when you look at this as a recurrent system it's hard to push the gradients through if it's if it's deep um and in this case there's no there's no free lunch the way the reason you pay is because you're

7:11 trading the f and the embedding simultaneously the system just wants to collapse it's really happy to just uh to just collapse off and so the key is training the dimension reduction of finding the correct contrast term and what you find when you try and play this game is that kind of standard graph based uh dimension reduction techniques even kind of unconstrained ones like you know plus snagit max and lle have a way too weak contrastive term so what does uh laplacian maps are l they do roughly they say you take a point its neighbors need to stay its neighbors in the embedding space um well the simplest way to make neighbors say the neighbors is to map everything to zero how do you make things knock knocked to zero you force the coordinates to be orthogonal that's actually a very very weak uh contrastive term and what you see if you try and play this game is that very different objects get sent to the same

8:06 location and vegetables and actually this is true in a lot of things in text you can use the predict the middle patch trick if you want to kind of get feature vectors in other words your game you try and say if my if my features are this one this one this one this one in object space i can try and have a a loss function that pays when i can't predict the middle patch the problem is with image patches and even at kind of higher levels of things in images it's too easy because you have smoothness predicting this under patch you just you just assume things are smooth and you can really easily predict the center patch and so again you get a collapse okay so let me briefly talk about what we actually did which was significantly less ambitious so we didn't go straight from patches which we really wanted to do we use sim vectors as input the middle layers are compositional and trained unsupervised

9:10 in the way that we wanted to and essentially you've got a variation on a sif sparse coding pyramid match pyramid fullscreen what was the dimension reduction so again this is the key thing we used um essentially uh dr lim and you had to very carefully engineer what are the positive things that are supposed to be close together and what are the things that are supposed to be far apart um so again always when you look at things in this way the hard part is to make things not uh if you if you use a really strong contrastive term it's very hard to train and this is always true if you justify means keeping far away things far away if you have a reasonably weak contrastive term it's easier to train but the thing collapses and so here the way we did it is you say you have your neighbor you have your original you look at original space and you find your neighbors and you say okay my first neighbors

10:12 we'll try and keep them close the second neighbors not necessarily the number two but some carefully engineered number of neighbors away will make sure those are far away and again there's there's some unpleasant engineering there but you can get something that that roughly works um and so for the total mapping f it's parameterized as uh sparse coding to get a very high dimensional vector and then a linear dimension reduction that's uh trained with the dr lim walls to get the object space and what this this really looks like a lower lowering convolutional net if you use a lista type encoder for sparse coding this is precisely a lower conversional map you have uh a bunch of feature maps that map you into very high dimensional space and then you have a a low dimensional linear embedding so it's just um filter some non-linearity filter non-linearity and then a filter bank that's very very tight in

11:11 feature size so this is the uh this is the final architecture that we used and again this was this is essentially a variation on a pyramidal system so an image comes in you take the sip vector you code it using sparse code you feed that into a spatial pyramid pool into a classifier and then you you play this game where you uh crush it with the um with the learned linear mapping so this is the dimension reduction part um you take the sparse code of you you do the concatenation and you take the uh sparse code of the concatenated vector of the density vector vectors you code those and you repeat you pull with a linear train cooling and so on and so forth um so we we didn't actually test this on any kind of real image data sets we just used the baseline from it was a pyramid pooling thing so we show um reasonable improvement on how the physics and so on and so forth it's really unclear the the problem with that as you all

12:20 know is that a a motivated graduate student can always get improvement on celtic 256 so it's unclear how much is due to the model um but in any case you do actually get an improvement on baseline from doing this sort of game where you um concatenate well you you expand concatenate and then dimension reduce expand concatenate and dimension reduce and all those transitions okay i'll stop so you don't search for which parts go together no that's the whole idea is that the machine should figure this out itself so the whole idea is to make a parts model that is uh i mean exactly in the natural language thing the uh the concatenation um operator and the embedding together tell you which parts go with which objects so again i'm saying this you know in the in the dream world this is what should happen whether it actually happens or not and the way we've trained it it's not critical so i had a question on the training do

13:29 you train about by virus and joint so here it's i think it was i think it was apparent i think it was one uh yeah exactly so it was trained in the way that we wanted to do it but just not going to okay thought thank god for again

14:14 uh bye okay so the last talk is going to be

Questions about this episode

Who speaks in "ICLR14 Workshop: A Szlam: Unsupervised Feature Learning by Deep Sparse Coding"?

Arthur (Host).

What is "ICLR14 Workshop: A Szlam: Unsupervised Feature Learning by Deep Sparse Coding" about?

The presentation discusses an ambitious project on unsupervised feature learning through deep sparse coding, focusing on developing a model for images that captures translation and deformation invariance while enabling composition of objects. Despite challenges in execution, the concept aims to create a parameterization of the object space and a composition operator to generate new objects from…

What is mentioned in this ICLR episode?

5 people, works and ideas are mentioned, including Naomi Lynn Lim, Leo Ju, Ronald Kolobov, Bill Bowman, Jensen Huang. Each one on this page links to its own page with every episode that mentions it.

How long is "ICLR14 Workshop: A Szlam: Unsupervised Feature Learning by Deep Sparse Coding"?

The episode runs 15 min; the transcript is about 2,338 words in 15 timestamped sections.

More from ICLR

• All analyzed episodes
• People mentioned on ICLR
• Topics mentioned on ICLR