ICLR2015-luke-vilnis

What this episode is about

The presentation explores advancements in natural language processing (NLP) by embedding words as Gaussian distributions, enhancing the geometric representation of semantic concepts. It highlights the limitations of traditional vector-based models in expressing the breadth of meaning and facilitating symmetric comparisons, proposing a new approach to address these issues and improve performance across various NLP tasks, including machine translation and information extraction.

Key takeaways

• Gaussian word embeddings add depth by representing meaning as distributions, not just points—because nuance matters.
• Forget simplicity; true semantic comparison lives in the details — embrace covariance, because words have layers.
• Vector space models are like maps: they're great, but they fail to capture the twisty roads of meaning.
• Embedding innovations like KL divergence show that language isn't linear—it's a dance of uncertainty and context.
• The future of NLP is like a beautiful symphony—harmonies of Gaussian distributions will redefine how we understand language.

Who speaks in this episode?

• Luke Vilnis (Host) · 17 min speaking
• Andrew McCallum (Guest) · 2 min speaking

What is discussed, and when

1. 0:00 – 3:00: Andrew McCallum, Gaussian word embedding model
2. 3:00 – 6:00: KL divergence, Wikipedia
3. 6:00 – 9:00: KL divergence, Bayesian matrix factorization, WordNet
4. 9:00 – 12:00: KL divergence, Wikipedia
5. 12:00 – 15:00: Gaussian word embedding model, stochastic gradient, Auto correlation function, linear matrix inequality satisfiability

Notable quotes from "ICLR2015-luke-vilnis"

• Things they we use cosine similarity dot products and symmetric metrics like this and so the idea is to design a natural extension current embedding models that can naturally…

  about Gaussian word embedding model · at 1:52

• Sort of expresses how much one of the distributions is inside the other one so this is saying how easy is it to encode n sub J to encode n sub I inside and sub J and if you…

  about KL divergence · at 3:25

• Lots of work in symmetric dense language models and word and vending models this is also somewhat similar to Bayesian matrix factorization and that we're associating a sort of…

  about Bayesian matrix factorization · at 7:04

• 6,000 and entailing pairs from wordnet or rather we don't train on it we test on it we train on supervised and the idea here is that we have positive examples adrenaline as a…

  about Wikipedia · at 8:42

• Things like the paper describes or experimental results this is not a useful keyword and so a predictive but infrequent phrase it's very specific but it's too infrequent to be…

  about Auto correlation function · at 12:35

• Meaning and to use an expressive and dense representation it also is very scalable learning by stochastic gradient it would train on four billion tokens in about eight hours on…

  about stochastic gradient · at 13:39

Who and what is mentioned in this episode?

People

• Andrew McCallum — Person · Referenced expert (advisor/collaborator) · 2 mentions · first at 0:03
  Mentioned as the speaker’s adviser and collaborator on the Gaussian word embedding work being presented.
  Naturally capture asymmetric comparison inclusion or entailment of concepts and broadness and meaning and so we designed this Gaussian word embedding model and we actually see…

Concepts & topics

• Gaussian word embedding model — Concept · Core topic · 2 mentions · first at 2:10
  Primary focus of the presentation, introducing a novel approach to word embeddings using Gaussian distributions.
• KL divergence — Concept · Key argument · 4 mentions · first at 3:14
  Used to measure the asymmetry between distributions in the Gaussian word embedding model.
• stochastic gradient — Concept · Strategy · 1 mention · first at 13:44
  Method used for training the Gaussian word embedding model.
• Bayesian matrix factorization — Concept · Related work · 1 mention · first at 7:11
  Compared to the Gaussian word embedding model in terms of learning distributions.
• Auto correlation function — Concept · Example · 1 mention · first at 12:46
  Example of a specific but infrequent phrase in the context of keyword extraction.
• linear matrix inequality satisfiability — Concept · Example · 1 mention · first at 13:19
  Example of a frequent and predictive phrase in the context of keyword extraction.
  Variance inverse variance and then we take the difference of those rankings and so we looked at a frequent and predictive things and we get this kind of keyword that you want a…

Places & events

• Wikipedia — Place · Setting · 2 mentions · first at 4:22
  Source of training data for the word embedding models discussed.
• WordNet — Place · Setting · 1 mention · first at 8:39
  Source of entailing pairs used for testing the model.
  Objective but we accurately learn all of the containments we want so on some non synthetic data for this kind of task we actually train on this data set of about 6,000 and…

Full transcript with timestamps

0:00 all right Luke Phil Ness I'm gonna present some work I did with my adviser Andrew McCallum on embedding words as Gaussian distributions so recently there's been an explosion of interest in these embedding models for NLP and the basic idea is that we can take difficult semantic questions and we can reduce them to sort of easy geometric questions so ideas about semantic similarity can be reduced to ideas about just neighborhoods in vector space and ideas about maybe analogies can be reduced to vector arithmetic between vectors representing concepts and this is also at great success in many NLP tasks like noun phrase chunking and part of speech tagging it's led people get away from manual feature engineering and do easy semi-supervised learning in named entity extraction people have gotten state of the art by sort of baking in semantic knowledge from lexicons into this embedding space in machine translation

0:59 it's been very successful in avoiding problems with data sparsity rare words also for language modeling and question answering and relation extraction and other information extraction tasks it's really helped with dealing with these sparse label and feature spaces for these very structured complicated problems okay so now all these models have been really successful but they all represent concepts words features labels as points in vector space and if you think about if you go kind of further with this intuition that you should be turning semantic questions into geometric questions you can see that there's a couple things missing from this approach and the first one is that there's not really an easy way to take these points and have them express any kind of breath or broadness of meaning and there's also not an easy way to do a symmetric comparisons between these things they um we use cosine similarity

1:56 dot products and symmetric metrics like this and so the idea is to design a natural extension current embedding models that can naturally capture asymmetric comparison inclusion or entailment of concepts and broadness and meaning and so we designed this Gaussian word embedding model and we actually see that the word person has a broader bubble or composer and if we have the word famous to it in the middle we see Bach is right there which is what we would want okay so more concretely in a vector embedding model you associate for each word in the vocabulary you associate a single vector and we're just proposing to instead associate a mean vector to it and a covariance matrix and in our work we use diagonal covariance matrices but you can use any sort it's still very tractable and if you look kind of at the functional form of this density function that we're parametrizing there's basically two

2:57 terms there's a logarithmic penalty on the volume which comes from making everything normalized to integrate to one and that sort of dilutes the density as you get to spread out and the other one is a Mahalanobis distance which measures closeness according to some intrinsic metric of the concept and so to deal with asymmetry we want to we want to look at KL divergence and the intuition behind this that it's this log density ratio this integral and this sort of expresses how much one of the distributions is inside the other one so this is saying how easy is it to encode n sub J to encode n sub I inside and sub J and if you look at this visualization it sort of kind of shows you the part where P is inside of the Q decreases your distance and the part where P is outside of Q increases it so this is kind of a natural candidate for modeling this inclusion okay and if we look at the functional form of this it also has

3:50 a nice interpretation the first term says that our variance Direction should be aligned the broader concept should be broad in the direction that the other concept is the other one says we should have a small distance between our means and then term is another one of these penalties coming from normalization and we actually find later that if we tweak this term here and allow some concepts to be more spread out it actually works even better for our test okay so how do we learn vector word embeddings so standard vector word embedding the idea is that we see a sequence of contacts say from wikipedia this is a wikipedia article and we are measuring the energy of two vectors two words similarity with each other by just a dot product and in this kind of demo i'm gonna have we want the energy to be high sometimes you want the energy to be low but this is our convention so we sample a word in context like musician and that's a

4:47 positive example and then we're gonna sample a random word from the dictionary say banana and that's a negative example and we just want to basically rank a positive example musician and composer above composer and banana and that's going to give us everything similar the way we want so we want to increase the energy like this okay so how would we extend this the gaussians if we look at our energy it just looks like this dot product it's the sum of our component wise products and so we can think of it as this discrete integral and if we do that there's kind of a natural way to extend it to continuous functions where we just replace it like this and this is like a very natural idea and it's actually been done before by twenty Jabar it's called the probability product kernel so we'll just abbreviate it PPK later and what's nice is because the gaussian is a stable distribution this actually just turns into another

5:35 evaluation of a gaussian PDF and it's differentiable we can back propagate to all the parameters okay and this has a nice functional form too this basically tells us that if we want two things to be close together we can either make their means closer together or we can spread out their uncertainties in the direction that's aligned towards where the the mean of the other distribution is and then we have this other sort of regularization term the teen Eliza's us for getting too big so this is this is a nice interpretation there's also you can look at it as just this area here so it's the symmetric kind of overlap measure and it's this blue area okay so to train this we just use standard like max margin ranking loss and we just want the energy of one to be higher than the other we can also actually train directly using that caled versions from before if we have some kind of actual asymmetric

6:26 supervision say that we have some idea about what concept should entail other concepts okay so there's a lot of related work in word embeddings but specifically for asymmetric comparisons and natural encoding zuv asymmetric comparisons between embeddings there's been some work in the sparse distributional community because if you have a sparse word vector it's sort of his bag of context it's like a set and there's natural ways to look at the way one could include the other but not in dense and we've actually seen also and recent work that that people have found that denser distributed predictive representations can be better than count based sparse ones and of course there's lots of work in symmetric dense language models and word and vending models this is also somewhat similar to Bayesian matrix factorization and that we're associating a sort of a Gaussian prior to everything but we're learning the

7:18 whole thing by back propagation and not by Bayes rule in this case it's similar to mixture density networks or density networks where you parameterize a distribution by a neural net and also kind of the converse GP neural net where you put a Gaussian process through a neural net and have a gaussian at every every level okay so we're gonna look at a few experiments the first one is we have this intuition about KL divergence and asymmetric comparisons and hierarchies and we want to just use some synthetic data and confirm that our intuition is right that these this is a something that that captures that so we just have some simple tree type structures as our training data and the idea is we just want to say child and tails parent so child is is included inside parent and we're going to do this we're just trying to push the ball inside the other bubble right and we're gonna do this by optimizing KL

8:19 divergence directly actually and we do all this we actually see that you can learn all these inclusions the way you want to and you can see that the the sibling nodes aren't incredibly spread out here because that's not part of the objective but we accurately learn all of the containments we want so on some non synthetic data for this kind of task we actually train on this data set of about 6,000 and entailing pairs from wordnet or rather we don't train on it we test on it we train on supervised and the idea here is that we have positive examples adrenaline as a neurotransmitter or pieces of food but then we also have two types of negative examples there's sort of nonsense negative examples like aircrew and playlist and those are very easy to learn and there's also reversed examples like whether food is a pizza of molecule is a carbohydrate so when we do this we we train two models one with diagonal

9:17 variances and one with spherical variances and we train on a billion tokens at Wikipedia three billion of newswire so the four billion tokens total and evaluate in two ways first with f1 at the optimal operating point which would just be to take the KL divergence or whatever metric we're using and to take a classifier that was predicting at the optimal point to get the best answer the other is just through average precision of whether we can rank in tailing pairs above non entailing pairs so when we do this we see that we handily beat the best previous result using these sparse word representations with our predictive representation and then also that diagonal variances give like a full 2-point boost an average precision over spherical variances so this is this is a good result we also want to try some word similarity tasks because we've shown that we can capture these asymmetric relationships but we want to

10:14 make sure that we haven't given up too much and then we can still model the kind of symmetric relationships that people use word embeddings for so we use a suite of word similarity table for example words in 353 is 1 there's all these data sets they look like this basically you have human annotators they rank similarity of words on a scale of 1 to 10 and you just look at the Spearman rank correlation of your models similarity predictions with these similarity predictions so money and Bank are similar according to the annotators and professor and cucumber are not particularly similar and then sort of everything in between and so when we do this we see that basically okay we train again two models a diagonal and a spherical model and we see that we do kind of as well or better than skip graham on these various tasks and in general we compare both using the means and using the means and the variances to

11:15 see what we're getting and when we use these spherical gaussians it doesn't really matter for the symmetric purpose of the variance too much but when we use the diagonal you really need to use the variance okay so we've kind of established that these work well for asymmetric comparisons they work well for symmetric comparisons and that kind of covers the asymmetry part of our goal but we also want to look at the breadth or broadness and how can we maybe leverage broadness of meaning in these representations in the future and so we have some just qualitative kind of results here on using this for phrase finding and the idea here is that if you had this kind of a diagram and you knew that the navier-stokes equations were some one specific and physics was a broad concept and CFD and all that what could you use this for and we're really interested in extracting salient phrases from research papers so things that you

12:15 might find is like keywords underneath the abstract and the notion here is that what makes a good key phrase you want a fairly high frequency word but you don't just want it to be high frequency you want it to be somewhat predictive of it context so to give a concrete example frequent but non predictive phrases or things like the paper describes or experimental results this is not a useful keyword and so a predictive but infrequent phrase it's very specific but it's too infrequent to be a good keyword is in our data things like Auto correlation function didn't didn't show up much and so what you really want is this kind of combination being able to predict your context and be frequent this is kind of a tf-idf type concept so things like operational semantics and regular languages and we run this on all our data we really see this what we do is we sort by um we sort by the frequency and then we sort by the

13:11 variance inverse variance and then we take the difference of those rankings and so we looked at a frequent and predictive things and we get this this kind of keyword that you want a linear matrix inequality satisfiability and then on the other end we get things that are not only uninformative but you would never see in a paper like Hot Topic okay so in conclusion we introduced a I've seen word embeddings the idea is to naturally capture asymmetry broadness of meaning and to use an expressive and dense representation it also is very scalable learning by stochastic gradient it would train on four billion tokens in about eight hours on an old machine and we have a lot of ideas for future work here we want to move to kind of unnormalized models multiple peaked models and non Gaussian distributions we want to model more interesting things like relations and semantic frames and also to try this out in other domains

14:07 because we think that this notion of moving past vectors and going to these densities could be very interesting thank you I mean oh absolutely so I think to this basically you're talking about whether the triangle inequality or something like that holds and I think that if you have I'm actually not sure if it holds with these Gaussian because all of their matrices are positive definite that they that they measure distance with so I would have to think about that but certainly if you introduced multiple prototypes or multiple Peaks and you start looking at maybe max pooling over census you can get away from this triangle inequality

16:17 you consider evaluating these these types of recommendations on classical vacations of idioms like modeling and see if there was no known language modeling specifically I think that's a good idea there are a little interesting to use as features just plugged into some other model the way people do with a lot of word representations because it's not quite clear how to put that vector in there but we do have ideas about doing that maybe in more of like a GPU neural in that kind of way yeah sure yeah if you want to do it that way for a language model and just get your categorical distribution that way that's a good idea

17:22 yeah I see yes we didn't find so firfer skip grammars or a basic vector models we didn't find like a huge improvement from using the extra dimensions on the on the word similarity kind of everything goes up a little bit and actually one thing that's interesting for the gaussians is that we found that the model capacity actually seemed a bit lower in that because you have this variance it causes your model to be a bit or like axis aligned it gets spiked year and which we we found if you look at the distribution of components it was it was more spiky so it has sort of it might actually benefit from a bigger bigger dimensionality or even I think once you move to two maybe a low-rank let's diagonal structure or something like that you could get away from some of the spike Enis

Questions about this episode

Who speaks in "ICLR2015-luke-vilnis"?

Luke Vilnis (Host), Andrew McCallum (Guest).

What is "ICLR2015-luke-vilnis" about?

The presentation explores advancements in natural language processing (NLP) by embedding words as Gaussian distributions, enhancing the geometric representation of semantic concepts. It highlights the limitations of traditional vector-based models in expressing the breadth of meaning and facilitating symmetric comparisons, proposing a new approach to address these issues and improve performance…

What is mentioned in this ICLR episode?

9 people, works and ideas are mentioned, including Gaussian word embedding model, KL divergence, stochastic gradient, Andrew McCallum, Wikipedia, Bayesian matrix factorization. Each one on this page links to its own page with every episode that mentions it.

How long is "ICLR2015-luke-vilnis"?

The episode runs 19 min; the transcript is about 2,856 words in 18 timestamped sections.

More from ICLR

• All analyzed episodes
• People mentioned on ICLR
• Topics mentioned on ICLR