ICLR14: P Sprechmann (for Qiang Qiu): Learning Transformations for Classification Forests

What this episode is about

The main topic revolves around enhancing classification forests by learning transformation representations specific to splitting nodes within binary decision trees. This approach aims to improve the accuracy and efficiency of weak learners by discovering low-dimensional representations of high-dimensional data, thus optimizing classification performance.

Key takeaways

• Transformations in classification aren't just about compression; they're about crafting clarity between classes.
• Low-dimensional representations can be powerful—if managed cleverly, they avoid the muddle of mixed classes.
• Randomization in decision trees isn't chaos; it’s a strategic move to keep classifying clean and effective.
• Even with face detection, complexities arise—real-world data rarely fits the tidy 'low-dimensional subspace' ideal.
• Orthogonal subspaces yield shared clarity; juxtaposed, they reveal patterns hidden in high-dimensional noise.

Who speaks in this episode?

• P Sprechmann (Host) · 13 min speaking
• Chang (Guest)
• Rob (Guest)

What is discussed, and when

1. 0:00 – 3:00: Andrew Chang
2. 6:00 – 9:00: LDA, Connect Data, Extended DLP Face Detection, NIST
3. 9:00 – 12:00: NIST

Notable quotes from "ICLR14: P Sprechmann (for Qiang Qiu): Learning Transformations for Classification Forests"

• So well as rob was saying unfortunately chang couldn't make it here but he explicitly told me that if you have any questions or comments you can please.

  about Andrew Chang · at 0:00

• Transformation will collapse the classes and make them orthogonal whereas lda makes them separable linearly and the lower level dimensionless phase but does not enjoy this property another case would be the planes that i.

  about LDA · at 6:12

• Synthetic data and also on some real applications they used a mist that is not shown here they also use the extended dlp face detection the 15 themes and the connect data which has the depth images from.

  about Extended DLP Face Detection · at 8:35

Who and what is mentioned in this episode?

People

• Andrew Chang — Person · Absent referenced speaker/author · 1 mention · first at 0:03
  Mentioned as someone who was expected to be present but could not attend; the presenter relays that Chang invited questions via email.

Companies & orgs

• NIST — Company · Case study · 2 mentions · first at 8:38
  Mentioned as one of the datasets used to test the performance of the proposed learning transformation method.
  Synthetic data and also on some real applications they used a mist that is not shown here they also use the extended dlp face detection the 15 themes and the connect data which has the depth images from.

Products & tech

• Extended DLP Face Detection — Product · Case study · 1 mention · first at 8:40
  Used as a dataset to demonstrate the effectiveness of the learning transformation method in face detection tasks.
• Connect Data — Product · Case study · 1 mention · first at 8:45
  Mentioned as a dataset used for testing the learning transformation method, specifically for detecting different body parts in depth images.
  Not shown here they also use the extended dlp face detection the 15 themes and the connect data which has the depth images from human bodies in several positions and the idea to try to detect the different.

Concepts & topics

• LDA — Concept · Framework · 1 mention · first at 6:18
  Discussed as a comparative method for dimensionality reduction and classification in the context of learning transformations.

Full transcript with timestamps

0:00 so well uh as rob was saying unfortunately chang couldn't make it here but he explicitly told me that if you have any questions or comments you can please email me so okay the title of the topic is learning transformation from justification forests so i will begin the talk by giving a little bit of the context of what a classification with forest is so the basic idea is that the classifier is built as an ensemble of many binary trees which are composed by two type of nodes some splitting nodes and some leaf nodes the idea is that each of these splitting nodes is linked to one weak classifier that when a point the data point reaches this splitting point it makes a binary decision to send it to the left or to the right according to what the weak learner says in the lymph nodes what they have is a statistical collection of the different classes of the training points that landed in that specific leap note

1:12 in the training process so when testing the idea is that you get a new point you run it through all of all the different trees and then you do the you get for each tree's posterior class probabilities and you average them in order to get the classification so the main goal of this work is to try to learn a representation that is a splitting note specific in order to make the weak learner be to help them with learners to make accurate and fast classifications so in order to see what type of transformations they learn let's go to some motivation so it was recently used that some high dimensional data can be well represented with low dimensional substances it's well known this case where images of the same individual and their varying lighting conditions are more or less fall into a low dimensional cell this has been exploited by many methods in the literature so what they would like is to say okay

2:17 let's learn a transformation that really makes the representation go run so in this case this is simple because we know that the data already lives in a low dimensional subspace however if we consider this slightly more difficult case where the pose can vary then even if we do phase detection and cropping we are not going to be able to represent this as a very low rank dimension so what they do is they learn a transformation that brings a representation that is low run so after phase detection and cropping this is what they would obtain so if we look at this in the context of classification that learning a low dimensional subspace for the classes is good but it would be better to also add some discriminability so the idea is to collapse these subspaces but also separate them from the rest and this can be done in a supervised setting so let's look at two examples to illustrate this idea so the

3:16 first thing is let's consider this a three-dimensional case in which the red and the yellow lines are one class and the blue and the light blue are the other class so the transformation there they are hoping to learn collapses the two the red and yellow and the two blue into a lower dimensional subspace and makes them more prominent which is the more separated they can be as linear substances another example would be two planes that are forming a small angle and the idea is to learn a representation that would do something like this would collapse them to a lower dimensional subspace and make them more smaller so how do they achieve this task the idea is that okay let's take them solve this optimization problem in which t is the transformation and y plus and y minus are matrices which have in their columns different data parts so the three terms here are the nuclear norm which is the sum of the singular

4:20 values of the matrix here the matrices are the transformed data points and is known that the nuclear non is a convex relaxation of the run so if we look at the first two terms in this objective what they are trying to do is they are trying to minimize the run of the representation on each class so um but that would be not enough because it would be a very good solution to collapse everything to one single subspace and then the classes would be mixed so they add this other term that says that when you transform all data together the overall rank of that cannot be small and then the yellow constraint on t to avoid the 3d solutions so in the paper they show some properties of this cost the first one is that is non-negative and that it reaches zero when the subspaces associated to each class are orthogonal and it is an interesting comment that this would not be true if we were minimizing the rank as i was

5:25 using to explain the idea so the the question now is what's the size of the transformation t and an observation is that the target space they can be designed and you can obtain like some dimensionality reduction by choosing a small smaller space in the target dimension so these ideas that they that they are promoting resembles lda and they have in the paper a couple of images that show uh what's the difference between the behavior of the two for example going back to these two examples again the red and the yellow is one class and the blue and light blue is another class so what would happen is that their transformation will collapse the classes and make them orthogonal whereas lda makes them separable linearly and the lower level dimensionless phase but does not enjoy this property another case would be the planes that i mentioned before and what's happening here is that lda does not give a good representation

6:36 whereas the transform transformation that they propose makes the subspaces perpendicular to each other and are easier very easy to classify using some type of subspace model so going back now then to the trees we can see how could this be included as a representation on a particular splitting node so given a splitting node what they do is they randomly divide the classes into two groups that they call i plus i minus i so this randomization is uh the objective of this randomization is to avoid in the forest to have repeated trees so then they learn this transformation following the model that i explained and after the learning so they can approximate these subspaces that now are low dimensional they cannot approximate it with a dictionary to a couple of dictionaries the plus and the minus and then the at each node with learner will be composed by the transformation so that's the representation and then two projection

7:54 operators onto these subspaces so the whip learner that they use is of the following way so they are going to test the set arrival to alien split node then they compute the representation and then do the projection and select the one that produces the smallest error so one point to note here is that after they do the training the cost of the width learner is very small because the projection matrixes are already pre-completed so now i'm going to talk a little bit about some results they tested this on synthetic data and also on some real applications they used a mist that is not shown here they also use the extended dlp face detection the 15 themes and the connect data which has the depth images from human bodies in several positions and the idea to try to detect the different body parts i think there are 19 body parts so first some quantitative results here the idea is to show how the the power of the transformation learned

9:12 in in the treat so at the very last the very last row is their work and we are comparing accuracy and testing time the testing time is on i believe in about a thousand images so the the point is that this is not a force this is just a single tree and they want to show that even with a single tree by enhancing this representation at the splitting node level they can achieve very good very good accuracy with very little time the time that they have is more or less equivalent to some other random forests with other type of weak learners but they obtain a much higher accuracy of course if you add some trees to the other if you if you add some trees to the other algorithms then the accuracy gets comparable but at the cost of some higher running so well let's look a little bit what's going on so these three rows are each of them correspond to one splitting node in the tree the very first one is the

10:24 it is the root and the idea is that in the figure a the first one this that's a three-dimensional embedding of these 38 classes of different faces and as you can see there are many colors which means that under this projection they are all mixed together so then they randomly select half and half of the classes and that's what's shown in the figure e where you have all the and then the red and the two pluses and then the third picture number c is uh then the three-dimensional embedding of the representation in which one can see that the insert class variability is increased and the intraclass variability is reduced the last column is just the first component of this representation which that is seen already that is quite discriminated in all three things so the first one is the root node and the second and third are from child nodes so then increasing the number of trees also increases the accuracy

11:32 and this so that increases also the testing time linearly so the idea to show here is in these three data sets and nist and the 15 you observe the expected behavior so and finally to i wanted to they wanted me to talk to you about some extensions we are working on the idea is that this can be simply generalized to the multi-class setting in which you have more than two classes and basically the extension is straightforward is just instead of having two turns at the for the for the two classes you have as many terms as the number of classes so an example of this would be okay let's say if we have nine facial images for nine subjects so again we have a three-dimensional embedding of the original data set and all the classes are arranged in this all over the space and then when you learn these transformations you can see that the images are mapped into nice subspaces so to conclude the talk

12:57 the idea is that they presented a simple way of learning a transformation that significantly improves the performance of random forest particularly making a stronger the weak learners then this can be applied to diverse settings of plastering and classification i believe that they have some extensions that go in this direction that probably are on the archive and finally they have answered some of the theoretical questions but there are many many more to be answered and they are pursuing that at the moment so okay that's it thank you very much any questions all right so do you know how they uh this projection map has some sort of entropy based um techniques that probably use in these decisions yes what i level of understand nodes the both implied on both is very similar and i i believe that they are competitive anymore questions okay well let's thank them

14:24 right so just to say

Questions about this episode

Who speaks in "ICLR14: P Sprechmann (for Qiang Qiu): Learning Transformations for Classification Forests"?

P Sprechmann (Host), Chang (Guest), Rob (Guest).

What is "ICLR14: P Sprechmann (for Qiang Qiu): Learning Transformations for Classification Forests" about?

The main topic revolves around enhancing classification forests by learning transformation representations specific to splitting nodes within binary decision trees. This approach aims to improve the accuracy and efficiency of weak learners by discovering low-dimensional representations of high-dimensional data, thus optimizing classification performance.

What is mentioned in this ICLR episode?

5 people, works and ideas are mentioned, including LDA, Extended DLP Face Detection, Connect Data, NIST, Andrew Chang. Each one on this page links to its own page with every episode that mentions it.

How long is "ICLR14: P Sprechmann (for Qiang Qiu): Learning Transformations for Classification Forests"?

The episode runs 14 min; the transcript is about 2,020 words in 13 timestamped sections.

More from ICLR

• All analyzed episodes
• People mentioned on ICLR
• Topics mentioned on ICLR