ICLR2015-olivier-henaff

What is discussed, and when

1. 0:00 – 3:00: Heavy tails, Natural Images, Statistical observations, Oriented band pass filters
2. 3:00 – 6:00: Local low dimensionality, Nuclear norm, Unsupervised learning problem, Natural Images
3. 6:00 – 9:00: Local low dimensionality, Oriented band pass filters, Fourier space, Natural Images
4. 9:00 – 12:00: Synthesis procedure, Local low dimensionality, Statistical observations, Visual system
5. 12:00 – 14:22: Natural Images, Local low dimensionality, Oriented band pass filters, Local covariance matrices

Notable quotes from "ICLR2015-olivier-henaff"

• Recent work developing a new model for natural images so there's a long tradition in using statistical observations to motivate models for natural images for example if we simply convolve a localized filter with this image we find.

  about Natural Images · at 0:00

• Heavy tail distribution into a mixture of local distributions with varying scales was the motivation for models such as the gaussian scale mixture now in the rest of my talk…

  about Gaussian scale mixture · at 0:53

• Quantify it further and we use the nuclear norm as a continuous measure of dimensionality so the way to estimate the nuclear norm is simply to calculate the standard deviations of the local distribution along the principal.

  about Nuclear norm · at 2:35

• So to conclude thus far we have found that a novel property which is local low dimensionality which seems to be a unique feature of the responses of oriented bandpass filters to natural images.

  about Local low dimensionality · at 4:13

• For representation which is locally low dimensional now unfortunately if we simply optimize for this term we'll quickly find a globally low dimensional representation by simply…

  about Gradient Descent · at 5:11

• So what we can conclude for this is for a wide range of numbers of filters that the optimal set of filters for uncovering locally low dimensional responses are a set of oriented band pass filters but we can now ask what information is.

  about Oriented band pass filters · at 7:40

• Covariance map using in this case four filters then we generate a sample of gaussian white noise and do gradient descent on this image until its covariance map matches that of the target image and what we obtain is a synthetic image.

  about Gaussian white noise · at 8:33

• Indistinguishable our interpretation of this finding is that the computations that are happening in our visual system which enable us to discriminate these images are actually…

  about Visual system · at 9:37

Who and what is mentioned in this episode?

Concepts & topics

• Natural Images — Concept · Core topic · 6 mentions · first at 0:00
  Central theme of the episode, discussing statistical properties and modeling of natural images.
• Local low dimensionality — Concept · Key argument · 7 mentions · first at 4:11
  A novel property identified in the responses of oriented bandpass filters to natural images.
• Oriented band pass filters — Concept · Framework · 4 mentions · first at 1:25
  Filters used to analyze and model the statistical properties of natural images.
• Nuclear norm — Concept · Strategy · 3 mentions · first at 2:39
  Used as a continuous measure of dimensionality in the analysis of local distributions.
• Local covariance matrices — Concept · Framework · 2 mentions · first at 8:17
  Used to parameterize the local structure in an image for synthesis experiments.
  Image and to do so we're going to take an image run it through our filter bank and then compute local covariance matrices for each one of these neighborhoods and if we do this across the entire image what we obtain is a map of local.
• Statistical observations — Concept · Key argument · 3 mentions · first at 0:04
  Used to motivate models for natural images based on their unique statistical properties.
  Recent work developing a new model for natural images so there's a long tradition in using statistical observations to motivate models for natural images for example if we simply convolve a localized filter with this image we find.
• Heavy tails — Concept · Statistical property · 3 mentions · first at 0:14
  A statistical property observed in the distribution of responses to natural images.
  For example if we simply convolve a localized filter with this image we find a distribution of responses which exhibits heavy tails and this observation has then been used by…
• Synthesis procedure — Concept · Strategy · 3 mentions · first at 8:27
  Method used to generate synthetic images that match the statistical properties of natural images.
  Image what we obtain is a map of local covariance matrices now the synthesis procedure as fault is as follows for a given target image we can compute its covariance map using in this case four filters.
• Fourier space — Concept · Framework · 2 mentions · first at 7:03
  Used to analyze the partition and properties of filters in the frequency domain.
  On the fourier magnitudes for example so for the reconstruction term ensures that the sum of the fourier magnitudes of all of these filters tile the entirety of fourier space…
• Unsupervised learning problem — Concept · Framework · 2 mentions · first at 4:39
  Framed as a method to optimize filters for producing locally low dimensional responses.
  Producing locally low dimensional responses and to get at this question i'm going to frame it as an unsupervised learning problem so for a given image x we can compute the responses of a set of filters by simply convolving them with.
• Visual system — Concept · Key argument · 1 mention · first at 9:40
  Discussed in the context of how the human visual system processes and discriminates images.
• Gradient Descent — Concept · Strategy · 2 mentions · first at 5:14
  Used in the synthesis procedure to optimize images until their covariance maps match the target.
• Gaussian scale mixture — Concept · Model · 1 mention · first at 0:51
  A model used to decompose global heavy tail distributions into local distributions with varying scales.
• Wavelet decomposition — Concept · Framework · 1 mention · first at 7:34
  Observed in the optimal set of filters for uncovering locally low dimensional responses.
  Into three orientations and two scales and for 12 filters we again observe a sort of wavelet like decomposition of fourier space into different orientations and scales so what we can conclude for this is for a wide range of.
• Gaussian white noise — Concept · Statistical property · 1 mention · first at 8:36
  Used in the synthesis procedure to generate synthetic images.
• Eigenvalue decomposition — Concept · Strategy · 1 mention · first at 11:07
  Used to create an explicitly low dimensional representation of the image.
  Lost so to do this for every single local covariance matrix i can perform an eigenvalue decomposition in this case i had four filters so i have four eigenvalues plotted in the bottom plot that's a histogram of.
• Hierarchical representation — Concept · Future work · 1 mention · first at 13:00
  Proposed as a future direction to stack learned linear decompositions on top of covariance maps.
  Stack learned linear decompositions on top of covariance maps to turn this into a hierarchical representation and with that i'm going to leave you with these low dimensional pancakes and thank you for your attention.

Full transcript with timestamps

0:00 recent work developing a new model for natural images so there's a long tradition in using statistical observations to motivate models for natural images for example if we simply convolve a localized filter with this image we find a distribution of responses which exhibits heavy tails and this observation has then been used by several groups to optimize sensor filters for this unique statistical property but the implicit assumption with this global histogram is that the image is homogeneous and yet if we look at local distributions of responses we find a great variety in their variances so for example in this high contrast region we find a local distribution which has a relatively large variance in a lower contrast region a distribution with a smaller variance and in this blank region a lower cut lower variance still and being able to decompose this global heavy tail distribution into a mixture

0:56 of local distributions with varying scales was the motivation for models such as the gaussian scale mixture now in the rest of my talk i'm going to follow the same procedure of measuring image statistics and then optimizing sets of filters for unique properties of natural images but rather than computing these statistics from the marginal responses of a set of filters i'm going to look at the joint responses of these filters and for illustrative purposes let me just start with a simple example using two oriented band pass filters so here again if we look at the global responses of these filters to the entire image we find a distribution which has these heavy tails whereas if we look at a local region for example in this low contrast region we find a relatively compact distribution and if we move to a slightly higher contrast region we find that the variance is increased but not only has there been a change in

1:54 variance there's also been a change in covariance in going from one location to another and we can see this by the fact that the variance along one direction is much larger than in the other direction and in some regions of the image this feature is pushed to the extreme and that the variance along one direction is zero whereas there's a lot of variance in the other direction and this um and we can describe this feature as the local responses being low dimensional or lying in a one-dimensional subspace in the two-dimensional ambient space and if we move to another image we find that the responses are again low dimensional but now the subspace in which they lie has changed and we've observed this feature in many parts of the image and so we decided to quantify it further and we use the nuclear norm as a continuous measure of dimensionality so the way to estimate the nuclear norm is simply to

2:46 calculate the standard deviations of the local distribution along the principal axes and then to sum up the standard deviations so we can think of this nuclear norm as a sparse measure on the singular values and another advantage of this measure is that it will be able to use it for an arbitrary number of filters moving beyond the simple two-filter case and what we find if we calculate the nuclear norms of these local distributions across the entire image is a distribution of local dimensionality which is highly peaked around zero meaning that many parts of the image are locally low dimensional and so we can now ask what this statistical feature is due to is it due to the content of the images or of the filters and so to get at this question we constructed a new pair of filters which have the same frequency selectivity as the original pair but now random phases and what we find is a distribution of local nuclear norms

3:40 which does not have this peak around zero meaning that these responses are significantly less low dimensional and similarly if we construct a phased randomized image this time we find a distribution of nuclear norms which is heavily peaked around zero or around one this time meaning that again these responses are much less low dimensional and perhaps unsurprisingly if we construct the or if we compute the responses of random filters to a random image we find we don't find this property of local low dimensionality so to conclude thus far we have found that a novel property which is local low dimensionality which seems to be a unique feature of the responses of oriented bandpass filters to natural images but we can now ask in the more general case for arbitrary numbers of filters what is the optimal set of filters for producing locally low dimensional responses and to get at this question i'm going to

4:39 frame it as an unsupervised learning problem so for a given image x we can compute the responses of a set of filters by simply convolving them with this image and we're going to view the map of filter responses as a set of overlapping neighborhoods now for each one of these neighborhoods we can form the matrix composed of the responsive vectors in this neighborhood and for each one of these vectors we're going to measure its nuclear norm and the first term in our objective function is simply going to be the sum of nuclear norms across the entire image and in minimizing this term we'll look for representation which is locally low dimensional now unfortunately if we simply optimize for this term we'll quickly find a globally low dimensional representation by simply setting all filters to zero so we need to ask something more of our representation which is that all the information present within the original

5:28 image be also present in our representation and we're going to enforce this by taking the representation running it back through the transpose of our filter bank and then penalizing any deviation of this reconstructed image from the original one now again if we only optimize for these two terms we'll find a globally one-dimensional representation by simply setting all filters but one to zero or by having all filters be scaled copies of the identity and the problem in these two cases that the responses are globally low dimensional whereas we would like for the subspace in which these load local distributions lie to be informative in itself the change across locations and for that reason we're going to add one last term which asks that the responses be globally high dimensional in order to ensure that all filters be used so to summarize our objective term or our objective function is looks for locally low dimensional

6:23 responses while asking that they be globally high dimensional and that all the information in the image be also present in the representation and so if we optimize this um objective function for four filters what we find is a local a low pass filter two oriented band pass filters and a high pass residual and in the top row i've plotted the spatial filters and the bottom row i've i'm showing the fourier magnitudes and what we can appreciate is the effect of the different terms of the objective function on the fourier magnitudes for example so for the reconstruction term ensures that the sum of the fourier magnitudes of all of these filters tile uh the entirety of fourier space whereas the global high dimensionality term ensures that no two filters are identical and finally the local low dimensionality term determines the precise partition of fourier space by these four filters and then if we move on to seven filters

7:24 we find that the filters now divide up into three orientations and two scales and for 12 filters we again observe a sort of wavelet like decomposition of fourier space into different orientations and scales so what we can conclude for this is for a wide range of numbers of filters that the optimal set of filters for uncovering locally low dimensional responses are a set of oriented band pass filters but we can now ask what information is contained in this locally low dimensional structure in natural images and to get at this question we're going to synthesize images which are mapped matched for this locally low dimensional structure but before getting at the information contained in this low dimensional structure let us start by parameterizing the local structure in an image and to do so we're going to take an image run it through our filter bank and then compute local covariance matrices for each one of these

8:20 neighborhoods and if we do this across the entire image what we obtain is a map of local covariance matrices now the synthesis procedure as fault is as follows for a given target image we can compute its covariance map using in this case four filters then we generate a sample of gaussian white noise and do gradient descent on this image until its covariance map matches that of the target image and what we obtain is a synthetic image that is random except for the fact that it matches the covariance map of the target image so in this case for example i've um estimated local variances from large 24x24 neighborhoods and subsampled them by a factor of four in every direction and what is surprising is that this the synthesized image is almost indistinguishable from the original one despite the fact that the representation is almost twice under complete relatively to the image and yet we notice that this synthesized

9:19 image is actually very far from the original one in in terms of its euclidean distance and we can better appreciate this if we corrupt the original image with equivalent amounts of white noise so how is it that the synthesized image and the original one are very far in the image domain and yet almost perceptually indistinguishable our interpretation of this finding is that the computations that are happening in our visual system which enable us to discriminate these images are actually very similar to these covariance maps and as a result two images which have the same covariance maps are also indistinguishable from our visual system and so we can now ask what this effect is due to what information in these covariance matrices is driving this result and it could be for example that simply the variances are sufficient to be able to reconstruct the image with high perceptual fidelity so we went out to test this

10:12 um so this time we're going to create a map of local variances and to ensure that this is a fair comparison with the covariance maps we're going to estimate these variance maps from 10 filters rather than four so these two representations have the same dimensionality and yet when we when we synthesize images from these variance maps we find a synthetic image which is very different from the original one both in terms of mean squared error and perceptually so from this we conclude that encoding the joint statistics of these local distributions is essential for being able to capture all the perceptually relevant information in the image but originally i was setting out to ask what information was contained in the locally low dimensional structure in the image and to get at this i'm going to construct an explicitly low dimensional representation of the image and then synthesize an image from this

11:03 representation and see what has been lost so to do this for every single local covariance matrix i can perform an eigenvalue decomposition in this case i had four filters so i have four eigenvalues plotted in the bottom plot that's a histogram of eigenvalues and to create an explicitly low dimensional representation i can simply threshold the eigenvalues sending all of them below a certain fixed threshold to zero and then i can then compute an explicitly low dimensional covariance map from this representation and then synthesize an image from it and what we find is a synthetic image which is actually surprisingly close to the original one it seems as though the difference between the two is just some noise which has been removed and i can emphasize this point even further but by now doing the opposite operation so now i'm going to make an image locally more high dimensional by taking the square root of these local

11:55 eigenvalues making them more high dimensional locally and then again synthesizing an image from this explicitly high dimensional representation and what we find is a synthetic image in which low contrast signal in the image has been amplified thereby corrupting the image with artificial texture so from this we can conclude that locally low dimensional covariance structure captures all of the perceptually relevant information in natural images so now to conclude we found that the responses of oriented band pass filters to natural images exhibit a unique property of local low dimensionality and we're able to use this property to optimize a set of filters to uncover this property our synthesis experiments showed that all the perceptually relevant information in natural images is captured by this local low dimensional structure and moving forward with this work we're looking to stack learned

12:58 linear decompositions on top of covariance maps to turn this into a hierarchical representation and with that i'm going to leave you with these low dimensional pancakes and thank you for your attention

13:22 so to what extent is this effect dependent on the choice of um neighborhood size and sort of the flip side of that question is is there a sensible way to go about choosing the neighborhood size to do this local analysis absolutely so yeah what i didn't show you is that if you estimate these local covariances using a much denser map so evaluating them over small regions and then not subtempling them as much what you would get is actually almost perfect reconstruction so in that case both in terms of mean squared error and perceptually the synthesized images are identical um so i the the choice is basically what you're going for either perceptual reconstruction or full reconstruction of the original image so in the last experiment it looked like you're creating aliasing which was kind of an artifact i i interpret so let's in the texture it looks like a very

Questions about this episode

What is discussed in "ICLR2015-olivier-henaff"?

In order: 0:00 Heavy tails, Natural Images, Statistical observations, Oriented band pass filters; 3:00 Local low dimensionality, Nuclear norm, Unsupervised learning problem, Natural Images; 6:00 Local low dimensionality, Oriented band pass filters, Fourier space, Natural Images; 9:00 Synthesis procedure, Local low dimensionality, Statistical observations, Visual system; 12:00 Natural Images, Local low dimensionality, Oriented band pass filters, Local covariance matrices.

What is mentioned in this ICLR episode?

17 people, works and ideas are mentioned, including Natural Images, Local low dimensionality, Oriented band pass filters, Nuclear norm, Local covariance matrices, Statistical observations. Each one on this page links to its own page with every episode that mentions it.

How long is "ICLR2015-olivier-henaff"?

The episode runs 14 min; the transcript is about 2,414 words in 16 timestamped sections.

More from ICLR

• All analyzed episodes
• People mentioned on ICLR
• Topics mentioned on ICLR