ICLR14: KM Herman: A Simple Model for Learning Multilingual...

What this episode is about

The discussion focuses on developing a model for learning multilingual distributed representations by leveraging parallel corpora, arguing that extending the distributional hypothesis to multiple languages can enhance semantic transfer and grounding. By analyzing aligned translations across languages, the model aims to improve the understanding of word meanings and relationships, ultimately leading to more effective multilingual embeddings.

Key takeaways

• Multilingual corpora unlock semantic synergy, revealing hidden connections through the power of parallel texts.
• Semantic grounding in language learning mimics human acquisition; multilingual data is a clever proxy for rich context.
• Forget task-specific biases; a joint semantic space for language pairs could reshape NLP as we know it.
• Why stick to monolingual bliss? Expanding representation learning across languages unveils dimensions of meaning we often overlook.
• Combining compositional semantics with multilingual data could elevate paraphrasing and translation to an art form—like poetry with precision.

Who speaks in this episode?

• KM Herman (Host) · 17 min speaking
• Carl (Guest) · 3 min speaking

What is discussed, and when

1. 9:00 – 12:00: European Parliament
2. 12:00 – 15:00: Ted Talks
3. 15:00 – 18:00: Wikipedia

Notable quotes from "ICLR14: KM Herman: A Simple Model for Learning Multilingual..."

• Learning where we use a lot of parallel data in our case we use the bureaucratic office which is proceedings from the european parliament and basically we just trained us until.

  about European Parliament · at 9:11

• And yeah as i said so we built importance for that based on ted talks which is even more massively multilingual so we're using 12 languages here almost english plus one language.

  about Ted Talks · at 14:50

Who and what is mentioned in this episode?

Places & events

• Ted Talks — Event · Referenced work · 2 mentions · first at 14:34
  Used as a source of multilingual transcriptions for developing recursive models in the research discussed.
• European Parliament — Place · Setting · 1 mention · first at 9:18
  Source of parallel data (proceedings) used for training the model in the research discussed.

Media & podcasts

• Wikipedia — Media · Referenced work · 1 mention · first at 15:51
  Mentioned as a potential source of comparable documents for extending the research methodology.
  Much so we actually tried this i guess wikipedia would be nice books especially certainly you send it in a sense of comparable documents rather than one-on-one foreign it seemed like it seemed like very low.

Full transcript with timestamps

0:01 all right so let's try to get this out um yeah so this is about multilingual distributed representations with uh reserved word environment um so let me give a very quick summary of what a stock is going to be about um so typically when you think about sort of representational learning and language um like most student representations in general um and we kind of think that it would make a lot of sense to expand this to learning from parallel corporation quite on the side of it yet monolingual representations which have some sort of semantic grounding another language um and it further to that and you might actually be able to sort of achieve semantic transfer across languages and so yeah we're going to learn these embeddings by essentially extending the distributional hypothesis to multilingual corpora um and to the sentence and well it's just going to work the results later on um so let's quickly talk about the

1:00 distribution on the plot of this this is this idea of you shall know the word but it can't be it keeps um and it's all idea that you can get the meaning of the word by looking at the words of code first with ultra curses and large corpora and it sort of works quite sensitively on the monolingual level and for learning word representations um but and i'd like to argue that this is more powerful when you expand this to multilingual data so if you look at something like this i give two parallel english german sentences um where in one case of the catalyst michael's cuts in the other cases alignment is carpeting the male version of that journey and so by having this multilingual data um you end up learning that sort of cutscene carter should semantically be very close and um this is because they align with the same word in another language rather than because they appear in similar contexts as you know as other

1:53 animals but also in that context and similarly sort of people thereby not only lingual somebody also learns semantic grounding in some way so if you look at um like like traditional linguistics when you look at language acquisition all of this is about semantic grounding and sort of grounding your language and real world experiences and all of these kind of things which which are really hard to give to computers when you try to teach them representations and i'd like to think that's a multilingual data it's a sensible proxy of that where sort of the semantics of what you want to do exactly the same semantic city as your parallel text so um yeah because this is about fulfilling compositional semantics why do we bother with that um there's a like two easy tasks in multiplayer so one is paraphrasing where you know you want to figure out if two sentences mean roughly the same thing and so what you need for that it's not

2:49 just representations that you work out but actually representations at the sentence level um to really make this call um and if you go one step further to uh to translation um now you don't only need representations of d7 so but we also need representations that are valid across languages um so for us to figure out the floating sentence and yes so i mean there's been a bit of compositional semantics in the class in our field typically end up with two different kinds of objective functions that people use so um one of those i have so many of you have seen this before is a precursor protein products where you have objective imperfection to minimize minimize the error between some reconstructions and sort of the original inputs and it just kind of works fairly well for a couple of tasks but there will be questions of this whether there was a semantic composition uh please essentially down here which is

3:43 compression or are you saying single or is it just a small proxy that we happen to use um and then similarly something doesn't think that people would use a classification signal so this kind of the sentiment classification signal i guess of uh it was switching so it gets paid for 2016 language um this kind of website sentiment task but then there's a sentiment in the semantics of the same thing it's a good question so is all of these things you end up learning very tasks uh specific features rather than something that you sort of recall i guess semantic embeddings in the general settings um and so uh on that basis what we're gonna try to do here is to first of all learn representations in a multilingual semantic space um and try to avoid as fast possible sort of task specific biases in uh creating this space and women's representations um and finally we just want to take compositional effects very lightly into

4:44 account um uh getting this a little bit um so so this is kind of what what a simple model that would do this would look like um so imagine you have um sentences in two languages of a and b and um yeah some composition vector model that given sentence representations learned the central step representation now what we want to do if we just want to create this multilingual states the semantic transfer sentence level is to make sure that um those sentence level representations f a and g b um are closely aligned with the sentences online and uh not very new to each other in a life so so the motivation behind this kind of model would be um this has independent learning because the only objective function is that semantic equivalent sentences projected to the same point of space um we get lingual representations by having multilingual data throwing into the same space um and yeah it was joint space representation

5:45 so continual data will actually be semantically plausible it's in that space um and how do we use these composition vector models to transfer the sentence level um whereby you provide the largest amount of context but not just learning something on the word by workplaces but creating space based on the basic larger context um well so much so for the theory obviously um like in a simple way it works our objective functions to minimize the distance between f a and g b and we want to learn all of these um this can sort of quite trivially degenerate because everything to zero object every single one space so so what we ended up doing is um this is a slight so it's like variation is where the user supplies the trust of large margin solution so in this case we take our original objective to minimize the distance and put this into a hinge loss for uh noisy sentences for parallel pairs um so with this objective actually kind of

6:53 end up both forcing somatic equivalence sentences into the same point of space um but it also force as a randomly same sample sentences which are in all likelihood not semantically equivalent to sort of stay away from some larger and so much sort of the theory um the one thing i kind of skipped over so fast is composition vector model and so there's all sorts of models you could use here um we sort of decided to go with some very simple single single which is both words apologies for that um but i mean the whole idea of this is important to find out whether this bilingual objective function makes sense and subsequently i think it's quite easy to see how you can sub in any given cbm that would like to try and using a bag of words kind of forces us to evaluate the objective functions by linking the audience and write rather than taking those effects into account there so how do we evaluate the whole thing we

7:58 use this crosstalk document classification by implementing fml 2012 and um this this looks a bit like this so see if you have some english data english documents or sentences which which are classified and you want to classify some german data um based on the same based on the same classification scheme and so this is i think a really good task for evaluating social model because like first of all it's a sensible task in terms of real world interest it's something so you see how you increase this confirmation between before instance um at the same time this causes you to evaluate whether our embeddings make sense in my monolingual level and as we're trying to classify based on also whether they were a multilingual level because we trained a classifying one language and evaluated each other so what this looks like in practice is basically we take all of this data train our model and replace everything with

8:56 vector representations um given that we then train classifying the english data and classify the german data so this is fairly straightforward and sort of this is a two-stage training procedure the first part is the actual model learning where we use a lot of parallel data in our case we use the bureaucratic office which is proceedings from the european parliament and basically we just trained us until using this parallel objective to learn representations in close languages and i subsequently give those representations we train a very simple average conceptual classifier for the actual classification task um we use an average average perception of the process of germanity of these such as different systems with their results and one interesting thing we also did is we don't try the second one it's like qualified cdf plus here which we're not only trained on english german parallel data but also in this

9:55 french parallel data um and so this is this is kind of quite cool um it allows us to evaluate what i said earlier about i'm having this distributional hypothesis extended to multilingual data i wouldn't say um yeah it's multilingual grounding it doesn't prove representation building so if you write some english french data um the big questions does this also improve sort of german representations even so there's no additional german data but we have this additional grounding limited through english um sort of it asks the question whether individuals are truly multilingual embeddings where but you have a whole bunch of languages projected into one joint shared embedding space um so but so much for the modern center in terms of results so this majority baseline of the whatsapp two directions here so the left side is training on english data evaluating the german and uh vice versa training on

10:53 german data evaluating what the english test said um there's just two base sensors that are basically learned using using an interaction matrix of word enlightenment across the whole europa corpses and they usually deploy the status of learning blindness on top of that and then here's our icbm model so this is nice that's very well and so this is the one trained in english the german data only um and then there was the other model on english french data i think this is really nice and like quite satisfying especially when you you sort of look what's happening on the right hand side so the classification only on german representations and then testing the english ones actually increases quite a lot um by adding english french data um without having any additional german data um so i think this is quite quite strong evidence to support our hypothesis that we can uh extend the distribution hypothesis later

12:09 um and then well it's not representation i mean if you don't have some tcp projections so um this is um so this is the model trained on all three languages i just took a couple of uh short phrases from the english german french through the in there i think this is quite nice of what you can see you see some strong alignment between princesses the same meaning in three languages as close to each other and up there you have the female versions um and so if you have this this gender neutral english version somewhere between the uh the two gendered versions so this kind of sort of indicates that we've learned fairly sensible representations across those languages um so so this is kind of um the paper um if i had a minute um very briefly talk about a couple of things we've been doing since um so the first thing is you know i said earlier the qualities for cbm model which was back off words and so just to

13:13 prove that we can actually put something slightly more interesting in there um when you subsequently also experiment with this bygram cbm model where we use a non-linearity among light repairs in the corpus so this gives you gives you some stronger features in terms of word ordering and so these are results from the previous model and then if we stick with the bygone model that's actually that's quite nice it further improves things so using a more complex cdn does seem to help the representation learning overall the second thing that we played with was um to say well apparently we stopped at the sentence level for representation but it is a real reason why we have to stop there um so when we develop the recursive model where you don't present those representations and then

14:07 the main thing about this is quite cool is because now um so what we got rid of sofa is what we're getting rid of here sentence alignment so you can start using this on a comparable corporate you can start using this on on transcribe operator you mentioned one to one sentence and you still get a signal that you propagate back from the blockchain level or if you have sentences like dating you'll provide them signal with the sentence level signals um so for this we um build a new course based on uh transcriptions of ted talks um and well i don't have results in here but i think they're quite promising this is a nice approach that does work well um but yeah you have to wait into acl for that um and yeah as i said so we built importance for that um based on based on ted talks which is um even more massively multilingual so we're using 12 languages here almost english plus one language um and there's a multi-class

15:03 classification task so there's 15 labels and there's multiple layers we're locking brothers just one that's in the reuters thing and well basically our results and please don't say this it's a hard account possibly more interesting so yeah that's that's it thank you very much

15:29 so we actually tried this um i guess wikipedia would be nice books especially certainly you send it in a sense of comparable documents rather than one-on-one foreign it seemed like it seemed like very low hanging fruit uh i just think monolingual data into this thing on top of that i i can imagine a number of ways of doing it but it seems like it's it's impressive that it worked well as is and it seems like the yeah i mean there's definitely some way you could extend this by adding monolingual data to sort of improve your representation learning um and i guess what we try to do here is to be quite purist developers because this was really too valuable to say whether you can take a distribution hypothesis and extend it to multilingual data and i guess if we had released a huge like large amount of monolingual data on top of that some sort of model d i remember some neuroscience uh data brain imaging of early bilingual

17:11 learners which is late and you know that their representation were mixed when they were early learners and and separate later can you try this well one is kind of going ahead and and you know you start learning a representation in one language and when it kicks in later would it change the representation yeah i mean that's sort of what i i guess i wanted to show you here earlier like i mean this kind of setup you can use just a semantic transcript so if you assume that you have very very good industry presentations and you want to learn representations of some local source language then you could use this kind of approaches to just sort of like learn those secondary representations i i guess you could you could sort of open this up for example which then also improve the english translations um but yeah we haven't tried that but the question is if you uh learn it together early on or you have also this asymmetry which you

18:04 showed between english german german uh english and so is it better to go ahead with one language first and then the other ones kicking in or yeah very quick uh so in this multilingual setting do you have any comparisons between the use of reconstruction signal the one you talk about and the use of a classification signal uh just to see whether that makes sense earlier you talk about monolingual setting where you actually talk about both the reconstruction signaling versus the classification signal what oh uh well i didn't end up using that this was just to demonstrate why i think they're not very good i see okay so so this is for monolingual for multilingual do you have this kind of comparison um well i guess you could compare this i mean there's been some work recently on um i guess using using sort of auto encoders and multilingual setting um

19:14 which also works well um firstly personally i think our approach is a bit more elegant because we learn um like well the aids were faster than i think you know more sensible representations where if you use something like fortune coder and you need to predict your representations on the other society typically you end up doing this by investing into source trees

19:43 all right we should thank carl once again yes

Questions about this episode

Who speaks in "ICLR14: KM Herman: A Simple Model for Learning Multilingual..."?

KM Herman (Host), Carl (Guest).

What is "ICLR14: KM Herman: A Simple Model for Learning Multilingual..." about?

The discussion focuses on developing a model for learning multilingual distributed representations by leveraging parallel corpora, arguing that extending the distributional hypothesis to multiple languages can enhance semantic transfer and grounding. By analyzing aligned translations across languages, the model aims to improve the understanding of word meanings and relationships, ultimately…

What is mentioned in this ICLR episode?

3 people, works and ideas are mentioned, including Ted Talks, European Parliament, Wikipedia. Each one on this page links to its own page with every episode that mentions it.

How long is "ICLR14: KM Herman: A Simple Model for Learning Multilingual..."?

The episode runs 20 min; the transcript is about 3,088 words in 21 timestamped sections.

More from ICLR

• All analyzed episodes
• People mentioned on ICLR
• Topics mentioned on ICLR