Interview with Cordelia Schmid, a Principal Scientist at Google and Director at INRIA

What this episode is about

Cordelia Schmid, a distinguished Principal Scientist at Google and Director at INRIA, discusses her extensive contributions to computer vision, including her award-winning research and leadership roles in academic publishing. Her impressive career encompasses over 100 technical publications, multiple prestigious awards, and a significant impact on image matching and learning visual models.

Key takeaways

• From math enthusiast to AI pioneer: sometimes, career paths are dictated more by job prospects than passion.
• Winning multiple test of time awards proves that lasting contributions to science require both innovation and relentless curiosity.
• Local features were game-changers in visual recognition; who knew the secret to success was all about the details?
• Evaluating performance systematically revolutionized computer vision—who knew data-driven insights could be this transformative?
• Understanding visual processing: from objects to embeddings, the complex journey is a vivid reminder that progress is rarely linear.

Who speaks in this episode?

• Cordelia Schmid (Host) · 44 min speaking

What is discussed, and when

1. 0:00 – 5:00: Cordelia Schmid, INRIA, Google LLC, CPPR
2. 5:00 – 10:00: ECCV
3. 10:00 – 15:00: Google LLC
4. 15:00 – 20:00: Google LLC
5. 25:00 – 30:00: Google LLC
6. 30:00 – 35:00: INRIA, Google LLC
7. 35:00 – 40:00: Google LLC
8. 40:00 – 45:00: Google LLC
9. 45:00 – 50:00: Google LLC, Epic Kitchens, Ego4D
10. 50:00 – 55:00: BERT

Notable quotes from "Interview with Cordelia Schmid, a Principal Scientist at Google and Director at INRIA"

• University of Oxford in the robotics research group of Oxford University and since 1997 she has held a permanent research position at inria where she is a research director she is also the author of over a 100 technical Publications she has.

  about INRIA · at 0:43

• Prize for fundamental contributions in computer vision that have withstood the test of time and since 2018 we've been lucky enough that she has a joint appointment with Google…

  about Google LLC · at 2:03

• After High School I was I liked I like math a lot and so basically given that I like math a lot I was kind of thinking of either doing applied math or computer science and then…

  about Cordelia Schmid · at 3:06

• Improve the visual recognition performance significantly and then the last test of time award was for work which was done in eccv 2008 and that was to now we have all these…

  about ECCV · at 6:36

• Individual more than on yeah the environment I think I'm really interested in your recent work that leverages multiple modal modalities so you've worked on constructing high quality data sets from different modalities modifying.

  about BERT · at 15:05

• Performance then conceptual changes and I think so there it's kind of important to bring up new benchmarks I'm just seeing in the chat there's this question about ego Ford so I…

  about Ego4D · at 37:38

• Like people gardening doing all kinds of house household tasks so it's much larger variety of interaction so AB kitchen is what it is it's like interacting like cooking videos…

  about Epic Kitchens · at 45:46

Who and what is mentioned in this episode?

People

• Cordelia Schmid — Person · Main guest · 2 mentions · first at 0:05
  Primary interview subject; discusses her career arc and research contributions in computer vision, multimodal learning, datasets/benchmarks, and long-term goals around self-supervised learning and embodied interaction.

Companies & orgs

• INRIA — Company · Affiliation · 3 mentions · first at 0:48
  Cordelia has held a permanent research position at INRIA since 1997, where she is a research director.
• Google LLC — Company · Affiliation · 13 mentions · first at 2:10
  Cordelia has a joint appointment with Google Research since 2018, indicating a significant professional relationship.
• The Institute National poly technique de groby — Company · Educational affiliation · 1 mention · first at 0:18
  Cordelia's alma mater where she earned her doctorate in computer science.
  Briefly introduce Cordelia first and then we'll jump right in so Cordelia has a doctorate in computer science from The Institute National poly technique de groby and her…
• IEEE — Company · Professional affiliation · 1 mention · first at 1:39
  Cordelia is a fellow of IEEE, indicating her recognition by a leading professional organization.
  Fundamental contributions in computer vision that have withstood the test of time she is also a fellow of i e and was also awarded an arc Advanced Grant in 2013 the humbled…

Products & tech

• BERT — Product · Research tool · 1 mention · first at 52:45
  Mentioned in the context of modifying models like BERT to handle different modalities, indicating its importance in multimodal research.
• Epic Kitchens — Product · Research dataset · 2 mentions · first at 45:11
  A dataset mentioned in relation to Ego4D, indicating its foundational role in egocentric perception research.
• Ego4D — Product · Research dataset · 1 mention · first at 45:11
  A recent dataset mentioned in the discussion, indicating its relevance to current research in egocentric perception.

Places & events

• ECCV — Event · Professional involvement · 2 mentions · first at 1:16
  Cordelia served as a program chair for ECCV 2012, showcasing her active participation in the computer vision community.
• CPPR — Event · Professional involvement · 1 mention · first at 1:16
  Cordelia has been involved with CVPR as a program chair and general chair, indicating her leadership in the field.
  Between 2004 and 2012 editor and chief for icv between 2013 and present a program chair for cvpr 2005 and eccv 2012 as well as a general chair of cvpr 2015 in 2006 2014 and 2016 she was awarded the longette Higgins prize for.
• Longette Higgins Prize — Event · Award · 1 mention · first at 1:33
  A significant award Cordelia received, highlighting her lasting impact on computer vision.
  Cvpr 2005 and eccv 2012 as well as a general chair of cvpr 2015 in 2006 2014 and 2016 she was awarded the longette Higgins prize for fundamental contributions in computer…
• Kendrick Prize — Event · Award · 1 mention · first at 2:01
  One of the prestigious awards Cordelia received for her contributions to computer vision.
  Elected to the German Nation German National Academy of Sciences in 2017 and in 2018 she received the Kendrick prize for fundamental contributions in computer vision that have…

Full transcript with timestamps

0:03 welcome everyone and really nice to see you all here today we're very excited to be chatting with Cordelia and so I'll briefly introduce Cordelia first and then we'll jump right in um so Cordelia has a doctorate in computer science from The Institute National poly technique de groby um and her doctoral thesis on local gray value and variance for image matching and retrieval received the best thesis award in 1996 she received the habilitation degree in 2001 for her thesis entitled from image matching to learning visual models she then did a postto in the University of Oxford in the robotics research group of Oxford University um and since 1997 she has held a permanent research position at inria where she is a research director she is also the author of over a 100 technical Publications she has been an associate editor for itle E pamy between 2001 and 2005 and for jig jcv between 2004 and

1:11 2012 editor and chief for icv between 2013 and present a program chair for cvpr 2005 and eccv 2012 as well as a general chair of cvpr 2015 in 2006 2014 and 2016 she was awarded the longette Higgins prize for fundamental contributions in computer vision that have withstood the test of time she is also a fellow of i e and was also awarded an arc Advanced Grant in 2013 the humbled research award in 2015 and the inria and French Academy of Science Grand Prix in 2016 she was also elected to the German Nation German National Academy of Sciences in 2017 and in 2018 she received the Kendrick prize for fundamental contributions in computer vision that have withstood the test of time and since 2018 we've been lucky enough that she has a joint appointment with Google research um so welcome Cordelia very this is a very impressive bio we're very happy to be chatting with you today yeah thank you very much for this

2:23 very nice introduction and thank you for inviting me awesome um so I'd like to remind remind everyone um that they can also ask questions um and one way that you can do this is by yes exactly thank you utu that's the link for the Dory um if you're dying to ask you can also raise your hand and ask um um so let's get started um so Cordelia at which point did you know that you wanted to be a computer scientist um how early did you did you know that that was the case and what made you initially get into re research and computer vision more specifically Yeah so basically I was After High School I was I liked I like math a lot and so basically given that I like math a lot I was kind of thinking of either doing applied math or computer science and then basically by talking to people they recommended me to do computer science because there was more jobs available right so that was kind of the main thing and so basically then I

3:26 did computer science normal courses and then for my Master's Theses I worked in visual recognition and Robotics so my first project was actually detecting objects for NE robot navigation and that's kind of when I got into research so I found it really fascinating how difficult it was to get a computer understand the visual world right at the time was kind of 20 or 40 years back there nothing worked and basically even like detecting simple objects were super hard and I felt like it was a super interesting problem to see how to improve actually this way of understanding the visual world and to understand why the computer couldn't do the same things as a human can do right for us it's super simple and for a computer it was super hard and that's when I decided to do a PhD and then from there on I stayed in research Yeah so basically chasing things that are very hard but interesting that that sounds like a good

4:23 recipe um yeah and look how far we've come since then um so in fact one thing I was very impressed by when reading your bio is the the number of test of time Awards um I'm wondering if you would be willing to say a few words about um those works that won the test of time Awards yeah sure so the first test of time Awards so the test of time Awards they always awarded 10 years later right so that was like for the work which was published in 96 so that was actually the work which I did during my PhD and it was on visual recognition based on local features so at the time most of the works they used actually geometric inveran so they first extracted edges and Contours and then used those for recognition and so the approach we introduced was to use local features to recognize objects and it was the first time or one of the first times where it was possible to recognize real objects in larger collections of images so this

5:27 was the first test of time and then this second test of time award there was actually work which I did jointly with my first PhD student and so the work was like a followup basically where we kind of investigated what was like the the blocking point of this approach right so it wasn't a valuation study on how to design better local features and how to detect them where to detect them and how they would impact the performance and it was actually one of the first Works which performed a large scale evaluation and introduced syn systematic metric right before that many people just published results and a few images and there we had like this large scale evaluation that was the second test of time award and then the next one that was awarded that was work which was done in 2006 so that was with another PhD student and collaborators in the US UI and there was actually on how to design a spatial like now we have these local

6:28 features and how to put a spal pyramid on top of those to improve the performance iFly and that showed that by adding spatial context we could really improve the visual recognition performance significantly and then the last test of time award was for work which was done in eccv 2008 and that was to now we have all these local features how to use them for large scale search there we designed an an embedding and a spatial consistency constraint to search in very large scale data sets and you had this very nice demo where we can search the time in millions of millions and billions of images very effectively that sounds great and it sounds like there's the way you explain it also a very natural progression where each each uh new work built on what was before and looked for the next challenge um what's still missing and what can we still achieve um I was wondering if you can give a little bit of advice on how

7:32 to search for the next challenge what's the next big research project it sounds from your experience that um you have good intuitions about finding things to work on that have lasting impact and do you have any advice on how advice that you would give to a more Junior researcher about how to decide what to work on next yeah I think the advice would be to search for hard problems right right just to think like what is hard today and not just search for one thing which is easy to do like maybe you think I want to publish a paper and then you do something incremental so that's actually not giving you probably not giving you the test of time in par but to think like what are the hard open problems and then kind of try to solve those maybe also don't rush too much right today a lot of people they just rush from one paper to the other and I mean even I do it right but basically to not rush from

8:25 one paper to the other but to take some time and really think what are the things you want to achieve and how do you want to achieve them and and then really work on those do you think that the hard problems are typically problems that a lot of people are trying to solve or are there benefits in trying to find problems that are maybe a bit less areas that are a bit less overcrowded or is that not really a factor in this type of decision making yeah that's actually a very good question and it's like a discussion which comes up over and over over again right I mean if you do something where everybody's working on you kind of safer you feel safer that you'll have some progress but on the other hand if you pick some problems where which are not so overcrowded and where you feel you can have some impact maybe might have more impact right so it's it's kind of tradeoff and it's too if you work on something completely

9:20 uninteresting it might be harder to get some impact but I guess it's it really depends I think you have to really see where you phly can have the biggest contribution I think so there's no clear answer on this question I think have you thought about um at all how to deal with managing risk when it comes to research or is that not really do you mostly go with what you find interesting and don't really take that into account too much well I think it's typically a question which comes up if you have PhD students right so basically there I mean you cannot say oh let's do a project which you know that in 10 years there will be some output so you have to have some kind of risk management because you have to see that basically maybe the first year the first two years you can spend on something very risky but then if it doesn't turn out you have to see say now I have to kind of scale down the expectations and do something where I

10:13 know it's easy to do and publishable and then kind of like have this I mean there there clearly you have this trade-off right you don't want somebody not to publish at all and not do their PHD so you really have to say now I have to kind of make the decision to go for something it's simpler and and they're also like in some cases you might be publishing something which otherwise you wouldn't publish because just the person needs it to finish their PHD right that is interesting um and it kind of ties into a question that I wanted to ask later but maybe I'll ask it now since we're kind of on the topic which relates to um what the experience is like um in Academia versus industry in terms of the types of research that one does um of course you you have affiliation with both sides so I think you are in in a position to to make the comparison and um I I get the sense that you have a responsibility towards the people you're

11:12 managing in both scenarios but um if if you're managing PhD students how does the research that you focus on differ compared to managing people in in Industry yeah I would say I mean basically managing phds students I think it's very uni in a way it's very uniform right because basically in Academia you get your grants and you look for interesting problems and you can kind of decide what you're working on and I think in Industry it depends much more where precisely you are what you want to achieve what the goals are right so I guess it's much more dependent on the project you're working and like what are the current goals right so basically for example today there's in Google research there's a lot of freedom for doing research search but other companies they kind of say you have to work on exactly this problem and then you have to kind of execute on this problem so it really depends and and I think currently the

12:08 differences are that in Google there's much more data right I mean personally for me there's much more data and there's much more computer right so that's kind of makes it possible to do larger to attack larger problems and in particular like I've been working a lot on video that was something which got really hard in Academia right because there data was missing and also the resources were missing and then B basically at some point it gets really hard to make significant progress I mean has improved a bit people in Academia have been ramping up to get additional resources so there have been efforts to have these centralized clusters and things like that so it has changed a bit but I think currently for me that's the main differences yeah that that makes a lot of sense do you think that if um what advice would you give to someone who um is deciding between whether they want to go to industry or Academia as a as a more

13:02 Junior researcher um I guess there's not one size that fits all but what are some factors that you think would affect that decision yeah I think it's depends how you see your long-term goals and as I said basically today there's like many industry Labs where you can do very similar types of works right that you can have your own career and I think in in Academia you can potentially grow much larger team even as a junior assistant professor while in Industry I think it takes more time to grow your own team so I think maybe if you're more after managing people immediately I think I would say Academia is better and then if you want to do more your own handson work I think industry is better and then obviously there's also the the factor of salary right I mean a lot of people there also they want to go to Industry because it's better paid right I think that's kind of also an argument which I hear many

13:58 times maybe in Academia you also have to work harder I think that's kind of also if you want to really achieve something you have to work I mean I'm not saying people industry don't work hard but I think in Academia you have to kind of work harder because you have to manage the students you have to teach you have to get the grant so I think it's it's more more work at least at the at the starting level right and then you also have to build up your career and you get evaluated you have to get your tenure and stuff like that so it's actually harder right that all makes sense um do you think there's a bit of a difference in culture regarding the uh working or work life balance in the two place I guess there's a lot of variants because industry depends where exactly um yeah I think it also depends in on the individuals right I think it's more I would say the variance is higher between different people than between

14:53 the two the two I don't know I mean it's yeah I don't know I don't have any good measures but I know people in both places who work a lot and in both places who don't work that much I mean so it's like really it really depends on the individual more than on yeah the environment I think I'm really interested in in your recent work that leverages multiple modal modalities um so you've worked on constructing high quality data sets from uh different modalities um uh modifying models like Bert to handle different modalities and and so on and so I'd like to pick your brain do you think what are your thoughts on why it's important to have these types of cross model um approaches and are there new challenges that are associated with figuring out how to leverage multiple modalities efficiently yeah so first of all I think it's an interesting problem right to handle not just one modality but multiple modalities and then from from

15:54 an understanding side I mean you have much more information right if you just look at one modality by itself it's much harder to make sense out of the content right while if you have multiple modalities you can really show that it improves the performance and then if you think of how humans interact with the environment they also they also use the multiple modalities right and so one thing I mean this is like the multile modalities but I also think it's interesting to see how people interact with the world right so it's not only multiple modalities but also the interaction with the world which I think is is very interesting then you really have the full picture of what's happening right and then maybe on a lower level with the bird models I think that's something which has given a large Improvement right before having these multimodal models it was always hard how to combine modalities and with this bird style

16:45 models it's much more unified to combine the modalities and I think the next steps is to see how to further improve this multimodal models and to learn even better ways of combining them by having like some idea how the different modalities match and how to join them and as you mentioned one thing is obviously very important is the data right so basically there's a lot of image Text data sets around but for video text video multimodal data sets especially for long for long interactions and things like that there's not that much data around and so that's one of the things also I'm very interested in that's also something where Google has a lot of data and I guess one of the things we're always trying to do is to make the data also publicly available so that that basically other people can also profit from the data and so do you think that if we had the right data um and we have um we can

17:40 somehow Leverage The interactions with the world and also the different modalities um that we won't need any explicit supervision or do you think that this type of data can replace supervision alog together or do you think that um we still need explicit labels in order to achieve some goals well that's that's a good question so I think the longterm goal would be to to learn without any labels but then I guess you need some kind of interaction to correct for biases right so I think basically only having the data probably you're not going to solve all the problems but probably you can get far away and then basically having maybe some interactions with with the models You' have learned and to improve them I mean basically that's kind of where if you could interact with the environment or have some kind of teacher like automatic way of teaching the system then you could improve it online and

18:33 actively right so you don't have to go around and just label all the data but you could have an interactive way of improving your system so I guess that would be the final goal and then obviously if you look at like day-to-day tasks you're getting very far by using the labels right so this is like if you want to solve today concrete problem you're probably better off to say I'll label that much data I do something where I have like a pre-trained model I find you in my model and then I can Sol solve the problem I guess from a research point it's interesting to see how far we can get without any label data right or like what weekly label data like which is this multimodal data or by interacting with the B totally and then I guess if we did have some label data like you said to kind of correct the model in some situations that would have to be task specific in some way so we instead of getting a general model that can then be

19:26 applied to solve any task whatsoever yeah right or or you could have like a generalist model which you then adapt to the task right depending on what want and I guess that's something which is also very important is the explainability right that you don't just have a model which predicts stuff but you can also explain why and how what's going on right so that you you kind of understand why you're predicting this label for an image and if you want to kind of remove maybe noisy counted then you can actually say this is not correct I can use this explainable explainable fact to remove the data or use some additional knowledge to say this is not what I want that makes sense um how far do you think we are currently from having models that can explain their predictions in a way that allows for this um to iterate in this way meaningfully do we think we have uh a long way to go still for that yeah I think for now the models are

20:28 pretty weak right I mean they're kind of very close the prediction is very close to the explainability so it's not like completely complimentary and we don't have a good idea how to do this in a completely General fashion so I think for now the approaches they're quite straightforward and not and maybe for some context they're pretty easy but then otherwise in general I think we don't really know yet how to do this and I guess it goes also together with understanding having a more frent understanding of the content right because if you don't really understand you just predict some label or some caption or something then it's really hard to make it explainable well if you have a much finer grasp of the content the interaction of the objects the 3D then it makes it much easier to have some kind of explain explainability as it in addition to what you're seeing right so it kind together with

21:18 understanding better the content interesting and um taking a step back and thinking about the data sets themselves that we use to train these models um you've done some work in creating data sets from different modalities um can you talk a little bit about that and in particular one um I i' be curious to hear your thoughts about the following um do we just want larger data sets or to what extent does high quality matter and does the answer to this question change depending on how we wish to use the models after the fact yeah right I guess it depends on how you want to use the models obviously but I think if you want to have like for your in particular for prediction I mean there it's really important what kind of data use to train the model so like in our recent paper I don't know if you're referring to this like the video CC paper we really show that by having much better high quality data we can get much

22:22 better zero shot prediction and we did some additional experiments can really show that depending on what type of data you use I mean which kind of makes sense right if you feed the model noisy data it will learn much worse and then at some point kind of doesn't learn at all anymore right if you have too much noise doesn't learn if it's kind of in between still learns some some predictions and then if it's really good then it's much easier to get a good prediction and I guess one thing when basically we looked at this multimodal data and one thing which is much harder with multimodal data or Vision data is that basically there's much more noise intrinsic noise right because if you think of text corpuses there's like you can look at books and all the books the the text is correct right so there wouldn't be that high of an amount of noise well and you can kind of get good data much easier

23:14 than for videos right if you go to YouTube and you just crop random videos you don't really know what the content is do you have like some labels but you're not sure if the labels are matching you can use for example we did one work we used the YouTube search and to get like relatively clean data this is another data set which we have and there we can show that the performance is quite good but there you kind of rely on all the filtering which is done by YouTube already like all the Legos meas and all that to clean the data and so then it's better I guess if you just SC scrape random data from the web it's really very noisy right kind of so I think this is something which is really interesting is how to make the data better right I think like and it will help a lot in particular for this multimod models right so are there any existing automatic ways of um cleaning up a data set like that

24:09 without having to uh manually inspect the labels and see if they match the video correctly yeah I guess the the video CC what we did there is we kind of took image as in captions to M videos then you can see if like for example you have some meas me if the captions measure the video content you can use some kind of other additional metrics to to clean up the data right so there's a lot of things and I think one thing I'm interested in is to seeing if like for example you can detect objects based on the captions and then see if this is another way of filtering out the data and then obviously you can also go to other other sites where the data is a bit cleaner so whatever some video content sites where people actually annotate I think the the goal is really to get this idea how to get more dat data automatically right because you cannot manually annotate all the data maybe what you can do is you can use

25:03 some annotated data as PL labels to KI start the process right this is something we're also investigating to see how we can use these pseudo Loyals to get better data and I'm also curious when it comes to um scaling up um maybe for moderate sizes of data it is really important to have as high quality data as possible but if we were to scale up to train on the entire internet or something larger than that does the quality still matter or in in that phase is it less important no I think the quality is still matters a lot right because if it's too noisy you just you cannot learn right I mean think of if you have like label noise which is larger than 30% then you just don't learn anything and so if you just scrape all the data and it's completely noisy then system there's no hope that the system will learn completely automatically right and if you think of a human you also don't just learn random you don't do random

26:06 stuff right you watch maybe created content or you have a book you read from a book so you also have material where you actually read or in inest clean content right and so if you just take all the data which is on the web which I mean basically some cases you can make sure like some of the images and caption data set they're quite clean if you go to vik so basically you can actually make some assumptions that the data is clean right but then you really you're really careful about which data you're you're getting right and it's not it's not about just having any random data so it's not possible I mean basically for example if you look at this how to one million data set which has like random instruction videos you can actually show that even by scaling it up you get a better pre-training performance but at some some point it saturates and if you look at zero short learning the performance

27:02 is not that good alog together right so there really you can see that you kind of blto off at some point interesting um that that makes a lot of sense to me um do you think it contradicts to some extent the success obtained by um this is not in the world of video but models like clip that are just pre-trained to match um images to their cap to their correct captions um which I mean those don't require labels so it's a different story maybe but it seems that they are able to do by just pretty much uncurated data to acquire abilities like f shot learning well I guess they're quite curated right all this clip methods they're quite curated because basically what people do is they kind of make sure that the content of the captions match the images and I think like if you go to edia Pages you have I mean some of these Pages they have really very clean even manually created content right because

28:05 it's people who upload the images and the in the and the videos and so I guess there I think a I think there's quite data is quite clean I think what we have there is quite clean and B it's not yet reaching supervised performance either right so basically the two things and I think for videos there's much more noise and so that's kind of Why by maybe leveraging this these clip models we can just Kickstart the recognition for videos and then improve the performance right so that's kind of I think I don't think it's contradictory right it's kind of the data is clean and then basically you have to make sure to get the best possible data and then see how you can scale up there yeah that's a good way of putting it I wonder how much clip would suffer if if um I don't know if somebody run this experiment I mean it's it's only interesting from a research perspective it's not really practically useful but

28:55 if if some noise was introduced to the training data so some of the um image and caption pairs were actually not really matching but we told clip that they are matching and then trained with this slightly noisier data how much the performance would degrade I wonder how robust clip would be to that type of noise yeah I guess you need to do a full study right to know how much the noise is hurting you and then I guess that's if you then that's when you can know how much it hurts but yeah I I don't think we can foresee that oh awesome um a thought experiment like that was done in the aligned paper apparently do you want to tell us a bit about that uh yeah sure so from what I've remember from reading the Align paper it's similar concept to clip but it's a it's a a paper that was written by people here at Google and uh the difference the main difference I think versus clip is that they they use larger data that's more

30:05 noisy uh like a orders of magnitude larger than the data set that was used for clip um and I think the the noisy part is is what the N represents maybe I I'll have to read the paper again or at least look at it to remember but I thought that's that's the difference so I I yeah that's all I have yeah thanks good good reference um okay so taking a step further back than um um or was that the sound of someone oh okay there we go um yeah I sorry I put the link to the paper it says noisy text supervision So yeah thank you awesome so um taking a step back and um looking at your research on a high level Cordelia it looks like there's a pretty nice thread of evolution throughout and I'm wondering is there um or rather what is the North Star that you are hoping this to lead to is there a particular goal or task that motivates you yeah I think the North Star would be what I said before that basically the

31:29 system can learn by itself right and that you kind of achieve the same capacity as a human maybe even more but basic the same capacity without any or very little supervision right that you can just say I want to learn from the data and then I obtain ideally right I mean it's like the Northstar right that I have the same idea I can learn about the objects the relation between objects I can really understand the visual content either in 2D 3D or what's necessary to interact with the world and then the next thing would be that also not only can I understand the content but I can also interact with it right that I can just I can navigate around house and I can grasp things and I can manipulate objects so I think that would be kind of the North Star and I'm currently also so at Google I'm working at video but at in I'm also working on like robotics applications and visual navigation so that's kind of it's still

32:24 hard right to put the two together because it's like two separate works and it's it's hard to combine them but basically the the goal would be in the long term to be able to combine them and then just navigate around an environment and then by interacting with the environment you can actually learn what's there and by touching and the hope would be by touching and manipulating the objects you can actually learn additional knowledge about the objects and I guess some of the problems if you just have static images are hard to solve I mean if you have videos at least you can segment out objects but basically by interacting with the objects you could learn a better representation better features a better way of just understanding what represents objects that's quite the north star that sounds like an an amazing goal but it also sounds like a quite challenging and complex goal because there's all these

33:18 aspects to it learn to represent the world learn to interact learn to update your beliefs it sounds like there's an online component to it as well how do you think how can we measure progress towards that goal how close are we what are the best ways of um evaluating success there yeah I think each of the two each of the problems come with sub goals right I mean you can pretty clearly measure that I mean for example for the interaction you can measure how how far you get like I mean it's still really hard but can really can really measure what types of interactions you can accomplish for the understanding you can see like whatever what types of understanding you have for the video content so I think you can very clearly structure sub goals what kind of learning methods so as you said one of the things is online learning so how do you formulate things as online learning with reinforcement learning for example

34:13 or combining reinforcement learning with learning from demonstrations optimal control so all these things I think you can pretty clearly make some progress and I think what you're saying it's harder like the end goal is quite hard right so it's not clear how to get there today but you have to kind of solve substeps immediately right so you cannot hope that you'll solve the problem immediately but you can very well make progress on each of the areas so that's kind of and then I guess the question is then the next question would be how long will it take to solve the complete problem and that's kind of unclear right I mean basically sometimes things get make progress quicker and like there's like step forward and then sometimes things plate off and I guess that's something we have seen I've seen several times in my career you kind of get stuck and nothing works or there's like very little progress and then either you

35:04 change the method or you change the problem or you work work on something else but basically I think that's kind of anyway that's kind of the North Star right and then we'll see how far I mean you can ask me in five years again and then you'll see how far we have gotone I hope that I get a chance to chat with you again in five years so that I can ask you but for now um a follow-up question is um you know that makes a lot of sense breaking it down into um different Milestones um which one of those do you think is the hardest which thread of of these different um um directions do you think is the hardest right now or or the one that you're most excited about I think the hardest right now is to get some improved visual representation by interactions right so that's the thing which I kind of feel very hard that somehow the problem of interacting if the objects is sufficiently different difficult still

36:02 before making an improvement to the representation so that's kind of the thing where I feel like it's very hard but I mean that's kind of the thing where I don't really know how to solve it or how to what's the solution right now so and another thing that I wanted to ask you while we're talking about um evaluation and assessing how successful we have B towards this goal is um I wanted to know what you think about the role of benchmarks so you mentioned earlier when you were talking about the test of time awards that um your paper was the first to do an evaluation a larger scale evaluation and and before that in that area people were looking at um just a few images um what do you think is the role of having more well-defined benchmarks to track progress and are they helpful or do we overfit on them too much and yeah I think it was very helpful to have this benchmarks so at the time like when I mentioned it there were basically

37:08 no benchmarks and so then several people brought out benchmarks there was the Pascal I don't know if you remember that are you too young but there was the Pascal VC Challenge and things like that I think it was very important to measure progress and then I think at some point people started to overfit on the benchmarks right so so it's kind of like for example image net I feel like today it's very over fitted there are other benchmarks kinetics also for video understanding where people completely overfit and it's more about like minor things which kind of improve the performance then conceptual changes and I think so there it's kind of important to bring up new benchmarks I'm just seeing in the chat there's this question about ego Ford so I think it's important to think of new problems and bring up new benchmarks right to measure measure progress right because you cannot say I made progress if you cannot measure it

37:55 so you have to have some kind of measure but you probably have to find the the right Benchmark and that's something which is getting harder and harder right because basically the task getting more and more complex and you have to think more and more about how to set up these benchmarks how to define them well right to have not just like classification labels or simple object detection but like more more complex problems so yeah that makes sense um in the um research area that I've been working on one of the challenges is you want to show that a model can be good at transfer learning or few shot learning or something but so and so you tested on some Downstream tasks but it never feels like we have enough of these Downstream tasks to really be able to show that you know this model can do transfer learning and it seems that maybe all that we can say is that it can transfer learn to these very specific few data sets but

38:51 how do I know that it can really transfer learn it feels like we need a very different scale to be able to say this um more broadly do you think there's a similar um Trend in in terms of like how do we Benchmark understanding the world from visual representations yeah it's clearly similar maybe just one comment about I think in Google there are many applications where it's really interesting to Benchmark on this real applications right so this is maybe one thing you asked about the differences between Academia and Industry and I think so there you really have this large scale real applications where you can measure if your approach really improves right so they're basically the benchmarks they're kind of set by the users and you can really evaluate how much improvement you get with your model right so this is like maybe one thing where you can really say that it's not academic benchmarks it's not a

39:46 comparison of all the methods but you can really see that you can evaluate your methods on a much larger scale and I guess this is something which is going to be more and more true for example for searching with in videos and things like this that it's like really interesting to see how much information we can Leverage The users and how good the quality is for this yeah that's such a great point I guess at the end of the day everything that we do is aim aimed at at the end of the day solving real problems so drawing inspiration from um actual applications to Define benchmarks makes a lot of sense but on the other hand benchmarks and emia sometimes are designed to more specifically isolate different aspects or be or designed specifically to answer some research questions so it seems that the setup is cleaner but sometimes it's done in such a toy way that the conclusions there don't necessarily transfer to something more

40:48 realistic so I guess it's hard to to combine both you want realism but also some benchmarks that can isolate some interesting search questions hopefully yeah but I think it doesn't exclude the two don't exclude each other right I mean it doesn't mean that you have to be stuck on toy data sets just because you're working in Academia so I think this is something which is also something where Google can play a role to have better benchmarks right to think of what would be problems which are interesting to solve and put up benchmarks and then maybe inspire people from Academia to move away from toyish problems and work on more more real realistic problems right so I think this is something clearly where you can also put effort into it right so it's not it's not set in stone that you have to work on uninteresting problems or toyish problems just because you're in Academia so I think it's just that sometimes

41:42 people get stuck in this like simple benchmarks and then it's a hard it's very hard to develop data sets also right it's kind of something which is very hard and takes a lot of energy and often also not that much appreciated right because it's just people then say it's just it's just a data set paper so basically you have to also then contribute with the method so I think it's something maybe where people also should spend more time and like I guess there are some efforts now where people really have put put in efforts to do this right so I guess yeah that's such a good point we' definitely have have had some trouble publishing Benchmark paper before I think now narps has a new data sets track which is really good for encouraging this type of research yeah we had this paper this Ava paper where we had like this super great data set and it was so hard to publish it right and then basically in the end

42:38 we only got it published because we added the method to it right so it's like really and then in the end of the day this makes probably this stops progress also a bit because all these papers they're really hard to publish and people don't see the contribution of I mean obviously you don't want just publish a data set with no no interest but if it's like a new interesting problem I think it should be actually possible to publish it right so it's NPS now has this new Benchmark for data sets but guess maybe that's kind of a way forward or just put them online and people use it right that's also possibility it's quite um disappointing throughout this conversation it seems that benchmarks are quite important for making progress and yet I'm curious why it's why we are incentivized again writing data set papers and of course like you said not just a paper with a data set but a paper that actually poses

43:33 some interesting questions um to Drive Research in some directions based on a data set it's not just the data itself asking questions sounds to me equally important as having models that try to solve them why are we so biased against that yeah I guess people feel like it's somehow not scientific right I think that's kind of that it's kind of just collecting the data is like something which anybody could do and that's kind of why I think R viwers are kind of biased against I don't know I mean it's just stress observation right so yeah um okay I think um given that we have roughly 10 minutes left I'm going to switch to asking questions from the Dory so if you haven't yet added a question there now is a great time to do that and I'm going to start from the one that's ranked from the top and um maybe you've actually answered this a little bit but the question says I really wonder your opinion on recent data set like ego 4D

44:35 how do you feel about such egocentric perception problems yeah I think it's a great data set we just started using it so I don't I don't really have any insights yet on how how difficult how good it is but I mean we have kind of started to downloading it and they have a lot of annotations a lot of tasks so I think it's definitely very interesting it's very SK also and created so it's definitely an interesting data set could you say a little bit about it for those who don't know um about this data ah yeah that's a good idea thanks for presenting yes so it's it's an extension I think of Epic kitchens I mean I haven't I really haven't looked into it a lot but I think it's an extension of Epic kitens where people are manip pating objects and so there you have tasks of finding the objects seeing how the people manipulate the objects how to predict do future predictions of the manipulation so you see a clip where

45:33 people manipulate the objects you want to predict what are the next steps so you can use it for example for video prediction video generation so I think and it's I think it's not just kitchens EP kitchens just kitchens and there it's like people gardening doing all kinds of house household tasks so it's much larger variety of interaction so AB kitchen is what it is it's like interacting like cooking videos where you see from from an entric Viewpoint right how people are interacting with the objects and then what the what the tasks are are localizing the actions detecting the actions forecasting the actions I think ego 4D is similar but again it's just it just came out so I haven't like looked at it into detail awesome that sounds interesting um the next question um is what do you think are the next challenges that we should address in the context of computer vision well I think there's more than one

46:30 challenge right but I think what we should do is to go more towards object Centric representations and maybe towards more 3D object Centric representations and move away from this full image or video representations where you don't really know where the objects are at least that's something which I think is important and then the other thing obviously is towards multimodal models right so here computer vision it's just one modality but I think there are many cases where having multiple modalities is interesting and then again you have this multiple modalities the hope would be also to be able to move towards more object Centric representations so basically localize the object and the corresponding sound or the words and the text and the corresponding objects cool the next one says this is more of a management question since you started your joint role at Google how has your day-to-day life changed if you

47:34 manage other people at Google how different is it compared to managing PhD students and are those cross collaborations and sorry are there cross collaborations between both of your Labs uh yeah so basically it's quite different to manage people like Google because basically they're like full-time employees so you have to like take care of their career career and make sure that they evolve in their career and do performance evaluation while PhD students I mean sure they also have to have their their goals of finishing PhD but it's kind of more education goal and in France the phds are three years so basically it's a quite short period and you have to kind of make sure that they're doing their PHD it's kind of different from Google where you people stay for much longer time right so there it's like more long-term career Evolution and then are there cross collabration between your two Labs there

48:27 there's some collaborations but basically the the understanding is that basically the two the two types of Works should be completely separate by definition because Google doesn't want like information from Academia go to in the industry and the other way around as well so it's like the the my goal is to keep the two as separately as possible and then obviously sometimes their collaborations of people doing internships but I think the line is to keep the two as separate as possible which is maybe a Pity right but anyway that's how it is a quick followup on that is um how do you um as a manager kind of decide what type of work to do in Academia versus industry if the two need to be separate are there some guidelines about that or is it mostly just determined by the interests of the people working in either one of the labs yeah I think there are some guidelines and how like some areas of like work which are specified by a

49:30 contract between in and Google so basically which speci deze which type of work is done work and so then obviously like the sub problems are are dealt with on a day-to-day basis so basically if it's images videos and multimodal information then obviously like the sub problems are defined on a day-to-day basis but the large works for example I don't work at all in robotics at Google right while I work on robotics at in so that's kind of much larger separation of the tool right um the next question says it seems like multimodal data sets are related to supervised learning for instance images with captions that supervised learning if the researcher adds the labels but it's multimodal unsupervised learning if the caption image pairs are found on the internet yeah it's completely true I mean I would completely agree right I mean if you annotate the captions then it's supervised right well if you find

50:29 them on the internet then it's weekly supervised and I guess the the the boundaries are kind of like as I said before there many web pages where for example have people have annotated or their videos where people have really put the annotations for example on Vicky how you have annotations for images videos and steps so this is really annotated you could you could say that it's weekly supervised but it's really has been annotated by people maybe not by researchers but it has been annotated by people the same thing for example there's this recent web withd data set where actually people have annotated and it's kind of people say it's scraped from the rep but in the end of the day it's more it's closer to supervised labeling because people have annotated but anyway there's kind of some boundaries right some fuzzy boundaries between the two yeah interesting and this ties in to the discussion we were having earlier

51:21 about the noise and the annotations and maybe there's a spectrum and that's somewhere in the middle where you have some yeah and even for annotations right there's also some noise right so that's what we found for example if you have to annotate humans and their actions it's it's very radar dependent so you have to get very good guidelines to get like clean annotation so I think there's like a a large spectrum of of differences cool the next question says and the last question in fact says can you comment on healthy successful collaborations with product teams what are the signs of a successful future collaboration people benchmarks users and so on yeah I think a healthy successful collaboration with the product team is to have a clear defined goal or then basically you can experiment the methods which you have developed and see how much gain you get right and that's also like the successful collaboration would

52:24 be if you can show on this benchmark that you get an improvement and then they're relevant for the product teams and they actually adopt they adopt the solution right that it's not just you're trying out something on a benchmark and then nobody's interested but it's like a benchmark which people are interested in and then they're kind of happy and adopt the solution do you think so okay so you mentioned before that Bert models allow to easily fuse different modalities maybe better than approaches that we had before and I guess transform in general are pretty widespread these days do you think that we still need to make progress on the architecture side or do you think that we can make basically reuse Transformers from now on and just work on getting better data cleaner data or more data um and call it a day no I think Transformers is just like one step on moving towards more powerful models right I mean it kind of makes me

53:22 think like couple of years ago convolutions were the the state-ofthe-art everybody was using Co evolutions for this and this and this and applying them all over the place and then showing that there was some improvement think the same thing is happening now for Transformers I mean obviously it's a very nice tool for this multimodal Fusion I think it's not the end of the story right there's like they're pretty comput intensive they don't they can't model very long interactions so I mean they think there's a lot of limitations and I think I don't think I mean hopefully we're not back with this Transformers for the rest of our lives right so I think there will be something else and the first thing is obviously making them more sparse more more manageable extending them maybe to hierarchies having longer longer range interactions if you have if you deal with images you want to have better interactions with the different regions

54:13 of the image and so I think no it's not the end of the story it's just like one step forward so uh yes uh thank you so much H Cordelia for your time and for sharing your insights today we all learn from one another research retrospectives are a series of conversations the dive into a researcher's technical body of work to understand how papers build on top of each other and create a legacy we learn about paths not taken serendipitous events collaborations and setbacks that shaped their research our interviews are informal friendly and fun welcome to perspectives

Questions about this episode

Who speaks in "Interview with Cordelia Schmid, a Principal Scientist at Google and Director at INRIA"?

Cordelia Schmid (Host).

What is "Interview with Cordelia Schmid, a Principal Scientist at Google and Director at INRIA" about?

Cordelia Schmid, a distinguished Principal Scientist at Google and Director at INRIA, discusses her extensive contributions to computer vision, including her award-winning research and leadership roles in academic publishing. Her impressive career encompasses over 100 technical publications, multiple prestigious awards, and a significant impact on image matching and learning visual models.

What is mentioned in this Google Research episode?

12 people, works and ideas are mentioned, including Cordelia Schmid, INRIA, Google LLC, ECCV, CPPR, BERT. Each one on this page links to its own page with every episode that mentions it.

How long is "Interview with Cordelia Schmid, a Principal Scientist at Google and Director at INRIA"?

The episode runs 55 min; the transcript is about 9,755 words in 57 timestamped sections.

More from Google Research

• All analyzed episodes
• People mentioned on Google Research
• Topics mentioned on Google Research