ICLR14: G Dulac-Arnold: Sequentially Generated Instance-Dependent Image Representations...

What this episode is about

The presentation discusses the innovative approach of sequentially generating instance-dependent image representations to enhance classification accuracy. By allowing classifiers to autonomously identify and acquire relevant information from images—starting with fewer details and progressively refining their focus—the method aims to improve decision-making while minimizing unnecessary data processing.

Key takeaways

• Image classification just got a plot twist: classifiers learn to ask for more info, like a curious detective.
• Efficiency wins: by focusing on key regions, AI dodges noise and speeds up processing – think 'less is more'.
• Why gaze at the whole picture? Sequential learning hones in on crucial details, proving that context is king.
• Budget constraints aren't just for shoppers: they teach classifiers to be smart, making every data point count.
• In a world of data overload, adaptive classifiers show us that sometimes, the best view is a selective one.

Who speaks in this episode?

• Gabriel Dulac-Arnold (Host) · 15 min speaking

What is discussed, and when

1. 0:00 – 3:00: Dylan Gabriel, Nikola Patrick
2. 9:00 – 12:00: PPMI
3. 15:00 – 16:41: Dylan Gabriel, MNIST

Notable quotes from "ICLR14: G Dulac-Arnold: Sequentially Generated Instance-Dependent Image Representations..."

• There we go you very much so my name is trying to work with nikola patrick from our live in paris and today's talk like i said is sequentially generated instance-defined image representations for classification.

  about Nikola Patrick · at 0:14

• Theoristic or some sort of a one-step policy and so we ran this on the two data sets just to kind of show how it can work so we have the ppmi data set which is people playing with musical instruments so rather power playing or arm playing.

  about PPMI · at 11:07

• Classifiers and generalization of the states and so it's right so actually my thesis is all about this source of problems and all sorts of data sets and so we have all sorts of…

  about Dylan Gabriel · at 15:36

Who and what is mentioned in this episode?

People

• Dylan Gabriel — Person · Main speaker · 2 mentions · first at 0:00
  Presents the talk on sequentially generated, instance-defined image representations for classification and describes the method, training procedure, and experiments.
• Nikola Patrick — Person · Collaborator (co-author / co-presenter referenced) · 1 mention · first at 0:18
  Named as someone Gabriel is working with; likely a collaborator on the presented work, but provides no direct speech in the excerpt.

Concepts & topics

• PPMI — Concept · Referenced dataset · 1 mention · first at 11:16
  Used as one of the datasets to demonstrate the effectiveness of the sequential classification model, specifically for binary classification problems involving musical instruments.
• MNIST — Concept · Referenced dataset · 1 mention · first at 16:15
  Mentioned as an example dataset where the sequential classification approach was applied, particularly for distinguishing digits with varying complexity.
  Get should i think deeper sentences or should i just stop right away and actually do it on mnist with cost and you can see that simple digits such as one and two are quickly…

Full transcript with timestamps

0:00 all right so good morning my name is gabriel how does this might work with uh right there no it has to be pointing out otherwise there we go you very much so my name is trying to work with nikola patrick from our live in paris and today's uh talk like i said is sequentially generated instance-defined image representations for classification so it's kind of helpful uh so we'll go through stirred up questions that's available so the point here is to sequentially generate image representations that are well adapted for classification and so this is sort of a different vision of representations than what you get in terms of related representation i'm just going to hold this and so the goal here is that you're going to be able to have a classifier that not only learns to classify images but learns to acquire the information on its own and so here you have an example with a cow and a dog and so what you have at first the

1:03 classifier has no prior information and so it's learned so the images are pre-cut into 16 zones and all this all these examples and experiments and so first it has no information and it's learned that information is most likely present in the bottom right central square so it asks uh to see that information once your season information is able to leverage it to make a second decision since the first piece of information isn't really all that discriminating it goes down tries to get more information and once this additional information is there it's able to kind of branch off and follow what it believes is going to be the best pass to acquire the information that's going to help it classify correctly and so this is just a simulated example it's not about example except but we can see that the idea is that it would be follow the cow's body to see that there's a cow and follow the dog's pose to see if it's a dog without

1:52 actually having to consider all the information present in the image and so the information acquisition here is not only sequential which means it goes step by step but it's also adaptive so each new image being inferred upon is going to have a different path of the classifier follows that corresponds to the best path forward to be able to acquire information and so what we call the final representation here isn't any sort of latent representation but it's actually just what information was used by the classifier to classify and so this is part of a bigger framework that we've worked on during my thesis that we call sequential classification where we model classification as a sequential process and the goal here is that not only do we learn how to classify we also learn how to acquire the information necessary to classify and oh 10 of the two tricks is that as we acquire the information we can leverage

2:39 all this additional information so the end of our class the end of our acquisition trace for our acquisition test gets better and better we have more and more context to know what we should be looking at additionally since this classification process is explicitly existing we can constrain it during tasks to maybe implement some sort of budget or other constraint so in this particular test we actually have a fixed budget and we don't constrain that position at all but it's definitely possible so there's a couple advantages to this sort of approach for images first of all it's an efficient use of information so if we're not allowed to look at all the information we're going to use it efficiently additionally we'll see in some cases we have an ability to ignore noisy regions so we're going to be able to actually classify better on some data sets because we're not going to be looking at the whole image but only with the

3:22 important regions for classification and of course if we have some sort of a very heavy processing stage we're going to be able to do faster insurance so in terms of applications uh one nice one is if you have a robot navigation task and so you have a small opening for your camera it's not a very wide wheel field of view and you want to have a localization uh for example run so you need to look around you need to know which direction you're putting your camera to better localize you don't want to you know uh just go over your whole your whole possible set of directions to point the camera because that takes way too long additionally if you have a very large image stream and you want to be doing object detection inside this uh you're not going to want to convolve over your whole image through logic detection you're going to want to basically try to find out where most likely objects will

4:03 appear and additionally if you've seen first objects kind of see where other objects might appear nearby and then finally the test we do here is just a more naive image classification test where if you have a bunch of regions in your image and you want to be able to classify without using all the regions of the image then you can classify it but quicker and more efficiently using this kind of model and so there's kind of two conceptual steps for this model the first part is looking at information so you're iteratively starting off with new information acquiring some of the information leveraging that information to require more information and you do this you know as many times as necessary so in our case you have a fixed budget so that means you won't do it three times you can also learn to do it more or less and once you've required enough information either because of an outsider constraint

4:46 or because you've run out that's enough information but you can choose to classify it so these two steps can actually be learned together in one total process described by a market decision process and solve by agreeing with learning or at least finding some sort of great policy for your personal time so there's kind of two big ways to represent the state and so the state here is uh the set of regions represented by the local signatures so each region of this so let's say we have 16 regions each region has a local sift descriptor which is bag of words and our state fly of s is this matrix x which for each row is the bag of words for that region so we have k regions we have k rows in our matrix and each row is the bag of words sip descriptor for that row for that region and so as the process uh advances in the acquisition test these rows are going to one by one get filled out by the regions

5:42 required but at first the major distance to the empty and so our state representation is just this matrix flattened out into a vector and our available actions are k image acquisition actions so if we have care regions we start off with k possible image acquisition actions and as we acquire regions we're going to decrease the number of that image application actions possible and then we have our classification actions and so in certain tasks we could allow the classifier to classify at any point so we could actually learn when to stop acquiring and choose to classify it but in this particular paper we chose to just fix the budgets using the kind of expanded concept and so these classification actions appear only once required a certain number of regions and at that point we no longer require regions that are forced to testify and so here's the policy of house defined so the classifier for us is a policy

6:32 there's no like policy and then the final classifier the classifier is the policy so the policy is responsible not only for acquisition but also the classification of the our labeling and so it's a parameterized policy that takes in the state s so like we saw we have a state descriptor maker and it outputs action a transaction a is either an acquisition action or a labeling action and so the goal of this policy is to maximize the overall reward and this is important because we don't have the short-sighted uh heuristic in terms of choosing the regions but a long-term uh goal of getting covered classification forward and so especially in cases if we have some sort of cost associated to regions the classifier is able to make up or learn the trade-off between immediate penalty in terms of crime regions and ultimate positive reward in terms of the correct investment and so in our case our reward is

7:18 actually quite simple we just project a zero one loss onto the reinforcement running problem and so we have zero in the case of proper classification which is the zero penalty zero one loss correct label and then negative one reward which corresponds to a penalty of one on this investigation you can kind of show that this basically means that solving this reported learning test is equivalent to solving this request classification test and so in our particular case we have a fixed budget if we choose to use a slightly more funky policy so we actually have uh these sub policies so for each stage in the in the selection process we have a new sub policy so like that these cell policies can specialize and since they're linear they'll be able to work better on their specific stage in the classification process and the final sub policy we just call it f to point out that it's kind of the classifier of this process but in

8:06 general you should consider the whole policy is both the classifier and the acquisition process and in a lot of tasks we actually don't differentiate these so like i said each sub policy is a classifier in another cell and it takes in this feature vector 5sa which is just our 5s which was our matrix of advanced apparent features and places in some place in this big action vector so it's a very sparse vector and the state is just offset by which actually we're looking at i'm going to do an art max to find what the best action is for a specific context so like this we have a simple linear classifier parameterized by gamma b that basically gives us what the best action is to do a specific contrast and so here's an example of how it can work so we take in that let's say we're on the second step for classification we've seen these two regions and so that this classifier tells us oh then i should look at the central region of

8:57 this image get the center of our wonderful tower and then we see we have fixed the budget three so now that we have these three regions then the classifier is forced to classify and so it uses the information that it's acquired itself and then gets a classification label by full tower in this case and so this is the training i'm going to go over really quickly because otherwise on the run at a time the basic idea is that we just kind of learned the last part of our sub policies and propagate backwards so we are in our final classifier by getting in a bunch of random regions so let's say our budget's attempt we're going to give a bunch of 10 of 10 region samples and we're going to give it a supervision same code it tells them to classify given a random choice with 10 regions then we're going to learn our second to last concept for sub policy that's going to basically take nine regions and find

9:40 the best complementary tenth region given the nine linear regions we back this up one by one and each classifier is going to be able to take uh t regions and find the best t plus first region that's going to allow us to testify the best and so we do this all the way to the beginning and that gives us the middle initial classifier our initial sub policy which will be able to find out what's the best initial reason region that i can choose it will allow the rest of my fixed policy chains to classify it about the most performing manner so it's a cast of a policy search by dynamic programming which is in 2003 we kind of reinvented this like that so if you're interested in the algorithm i would suggest looking at the paper um but it runs really quick so we only have instead of having one linear classifier we just want to deal with the custom viruses so training isn't that bad and so here's kind of an example we have

10:33 our second to last cell policy we give it a random subset of two minus one regions and then it's going to try to find what's the possible complementary p region and from one to the ph region so region three that's gonna allow the final step to classify correctly but for regions five and thirteen the final step can't classify correctly so it's going to learn that for this specific subset of d minus one region the best complementary region is making three and so like i said because this is reinforcing learning uh because of your input reinforcement learning this is basically allowing us to choose the best region over the long term and not in terms of some sort of immediate theoristic or some sort of a one-step uh policy and so we ran this on the two data sets just to kind of show how it can work so we have the ppmi data set which is people playing with musical instruments so rather power playing or arm playing

11:21 and there's 12 binary problems for 12 instruments and the 15 scenes problem which is just a simple classification problem and we split everything into 16 regions this is a mono resolution we also had some multi-resolution but it didn't review the article it's always nice and for each region we just have a synthetic rewards feature that was calculated like standard industry processing and like i said multiple resolutions are possible but we just do that here and so here's kind of a nice example of how to work obviously this is one of the nicer ones sometimes it's never worse but sometimes it's just closer but what you can see here is on the bottom we have percentage of regions so in terms of 16 so half would be of course and the accuracy on the left and so as we sweep the percentage of regions we've changed the fixed budget our classifier we're going to increase slowly the performance of random

12:07 subsampling but we see that our learned sub sampling immediately is able to gain more or less acceptable performance and at the end of course we converge and at the beginning we convert it to the standard baseline so so you can see basically this pocket where we have very little information uh learning it is really important because you're going to sample correctly and as you get more information the advantage of learning a policy isn't necessarily all that much we can also see that in the case of ppmi over the entire data sets it's actually more more interesting to not look at your entire image because when we have very little information or all the information we convert to random selection but in the middle we're actually able to ignore noise and so we verify this by just making a static policy we look at the four middle regions of your of your image and since this data probably has a bias or

12:50 centered on the information looking at the four middle regions and ignoring the outside of the shell actually gives you the higher accuracy so if that's the static policy that you define the system is actually able to find it on its own and so here's also a 15 team so as you can see it's always slightly better to learn it's not always going to give you that huge performance yet it really depends on the distribution of what your data sounds like and also you know how much how much expressive do you want to bring your policies these are all linear but you can also find more complex here's an example of regions required with a budget of four on the fluke data set trying to see if somebody's playing or not instrumental so you can see they're always kind of looking at the same reach in the bottom left and there's this fourth region that kind of is different based on the image and so you can't always interpret what it's

13:31 doing but you can at least make a histogram so left is b equals four and on the right is equal to eight and so we can see that if we only using four regions these bottom two left regions are really really used a lot but if you allow the system to use eight regions actually that far left region isn't considered as important because it's able to spread out the information more and it chooses to look further so this is you know kind of some intuition you can pick up out of the system uh of course it's always hard to understand exactly what's going on but um that's how it works so there's a little bit more work on multi-resolution every budget uh especially getting the budget to learn more quick because if you want to learn to stop you can do it you've done it but you're returning advertising something gets a lot more complex because you have to not only learn what to choose but also when they classify

14:17 and some stuff just talking with people yesterday that seemed interesting would be either object detection and also video classification so not only can you move in space to look you can also move in time try to find keyframes that might have the most information available so uh hopefully this can help some people it definitely is like a nice way to avoid evolving over all your information and you're definitely going to find at least a better than random policy and sometimes that's quite a good policy to look for information not only in images but actually any sort of tutorial so thank you for your time and if you have any questions

15:11 i have a few questions actually um so you're you're using a fixed budget and you're treating it as an mvp i guess one way to think about what you're doing is really to train it as a pumping beat right because you're you're deciding where to look or think of them as information gathering actions so have you thought about so did you never really use the company that much we're much more into the idea of having all the hidden state the pure classifiers and generalization of the states and so it's

15:52 right so actually my thesis is all about this source of problems and all sorts of data sets and so we have all sorts of constraints and probably the most simple to understand cost-based data would be a medical diagnosis data where you have cost associated with different features and you're actually going to find a sort of uh doctor policy which is how to ask questions and based on the answers you get should i think deeper sentences or should i just stop right away and actually do it on mnist with cost and you can see that uh simple digits such as one and two are quickly classified with their proper best but more complicated to distinguished digits of the side of eight will actually be more deeply respectful and

16:35 all right um there's no other questions

Questions about this episode

Who speaks in "ICLR14: G Dulac-Arnold: Sequentially Generated Instance-Dependent Image Representations..."?

Gabriel Dulac-Arnold (Host).

What is "ICLR14: G Dulac-Arnold: Sequentially Generated Instance-Dependent Image Representations..." about?

The presentation discusses the innovative approach of sequentially generating instance-dependent image representations to enhance classification accuracy. By allowing classifiers to autonomously identify and acquire relevant information from images—starting with fewer details and progressively refining their focus—the method aims to improve decision-making while minimizing unnecessary data…

What is mentioned in this ICLR episode?

4 people, works and ideas are mentioned, including Dylan Gabriel, PPMI, MNIST, Nikola Patrick. Each one on this page links to its own page with every episode that mentions it.

How long is "ICLR14: G Dulac-Arnold: Sequentially Generated Instance-Dependent Image Representations..."?

The episode runs 17 min; the transcript is about 3,413 words in 22 timestamped sections.

More from ICLR

• All analyzed episodes
• People mentioned on ICLR
• Topics mentioned on ICLR