Molmo 2 | Counting objects and actions

Who speaks in this episode?

• Zama (Main speaker)

What is discussed, and when

1. 0:00 – 3:00: Space Needle, Momo 2, Gemini 3, Zama
2. 3:00 – 6:00: Momo 2, Gemini 3, Space Needle
3. 6:00 – 9:00: Gemini 3, Momo 2, Mount Rainier
4. 9:00 – 12:00: Momo 2, Gemini 3

Notable quotes from "Molmo 2 | Counting objects and actions"

• Hi, I'm Zama, one of the developers of Momo 2. In this video, I would like to show you what Momo 2 can do for video counting and pointing, which encourage both object counting as well as action counting. And in the second part of the.

  about Momo 2 · at 0:02

• Video. Okay. Yeah. So, it's spot on. It gets this pretty easily. and let's try something else point to space needle which again I think it should be able to get this correctly…

  about Space Needle · at 1:03

• Interesting that it's taking such a Okay, so let's see. Oh, okay. So, this one, Gemini actually does a better job at temporal understanding because it identifies their later step towards the end of the video. So, yeah, it looks like Gemini.

  about Gemini 3 · at 10:24

Who and what is mentioned in this episode?

People

• Zama — Person · Main speaker · 1 mention · first at 0:02
  Introduces themself as a developer of Momo 2 and leads the demo and comparison against Gemini 3 for video counting/pointing tasks.
  Both object counting as well as action counting. And in the second part of the video, I would like to show you a comparison of Momo 2 versus Gemini 3 on video counting and…

Products & tech

• Momo 2 — Product · Business focus · 6 mentions · first at 0:02
  Primary product discussed in the episode, showcasing its capabilities in video counting and pointing.
• Gemini 3 — Product · Business focus · 8 mentions · first at 0:16
  Secondary product compared to Momo 2, highlighting its performance in video counting and pointing.

Places & events

• Space Needle — Place · Setting · 5 mentions · first at 0:41
  Landmark used as a reference point in the video demonstration.
• Mount Rainier — Place · Setting · 2 mentions · first at 0:46
  Natural landmark used as a reference point in the video demonstration.
  Built on top of language model which has knowledge about Mount Reineer and even if not, it should be able to identify the only mountain in this slow motion video. Okay. Yeah.…

Full transcript with timestamps

0:02 Hi, I'm Zama, one of the developers of Momo 2. In this video, I would like to show you what Momo 2 can do for video counting and pointing, which encourage both object counting as well as action counting. And in the second part of the video, I would like to show you a comparison of Momo 2 versus Gemini 3 on video counting and pointing, which I would say both models do a pretty decent job in counting and pointing, but they're still not perfect, and this are still very hard task for today's visual language models. Okay, let's get started. First, let's try with a pretty simple video. For this one, I chose a classic beautiful skylight view of Seattle that has Mount Rineer and Space Needle. Let's start with something simple like point to Mount Reineer.

0:49 For this one, I think our model should be able to get it correctly because it's built on top of language model which has knowledge about Mount Reineer and even if not, it should be able to identify the only mountain in this slow motion video. Okay. Yeah. So, it's spot on. It gets this pretty easily. Uh, and let's try something else point to space needle which again I think it should be able to get this correctly as well. Um, because it's a well-known landmark that our language model should know about and even if not it does look like a needle.

1:24 So there is a good chance our model knows about it and yeah um our model got this one correct as well. It correctly point to space needle right on top of it. Okay. So now let's try something a little trickier. What about point to the tallest building? This is a more ambiguous query because there are so many buildings here. So it could point to Space Needle, but it could also point to another building. So let's see what our model points in this case. Okay. Yeah. Again, it it points to Space Needle as the tallest architecture in this video, which is correct. Okay. What if we try point to the rightmost building? So this one should have a more clear answer. So it's the this building right here. This small building on the very right. And because our model was trained with this this queries involving spatial references, it should be able to get this this one as well. Okay. Right.

2:25 Yeah. So it does point to the correct building here. Um okay. So, let's try something even harder. I see that there is a half of a ferris wheel here. It's half hidden by a building in front of it. So, I'm actually not sure if our model will be able to identify this, but let's just try going to the ferris wheel. Because it's so small, our model only sees a few pixels of it. Especially in this veto case, well, every video frame, it only sees a low resolution image of it. Okay, but here it gets it correct. So, it points to this ferris wheel here, even though it's half hidden by another building. So, that's pretty cool. Okay, now let's um how about let's try something a little harder. Let's do even harder. How many buildings are there?

3:27 This is really hard for the the model because there are so many buildings and they are also blocked by each other. So in this case of occlusion, the model might not be able to see every building clearly. Okay. So this is what our model points to. So you can see that it does point to most of the buildings uh like Space Needle. this one uh even though some of the points are slightly off um mostly because the building are so cluttered but it does try to point to all the buildings. So um yeah this one it seems like it got this one wrong because this is a tree but you might think that there is another building behind it. Um but yeah, it tries to point to most of the buildings and in this case it says there are uh 46 of them which is roughly correct.

4:20 Okay, now let's try something a little different. What if we try a different video with um slightly more motion? Okay, so we see that this is a video with a dancer doing uh flips. So, for example, I'm curious how many flips does the denser do? And since our model is trained to point and count actions as well, and let's see what it points to. So, this case it points to the this is the first flip, second flip, third flip. and then fourth flip. And then this is the fifth flip. And that's correct. The girl does a total of five clips and a flips. So that's pretty neat. Okay, now I want to shift to the demo comparison.

5:22 So let's try um for example on the same video, we can just use the same query. Actually, let's also ask this question to Gemini 3 to see what it answers. And let's just um call Momo 2 again to just have the model outputs side by side for comparison. Okay, so our model already returns to an answer and let's just wait for the frames to be visualized as well. Okay, it does it's taking a long time for Gemini 3 is partly maybe because it's a really large model or maybe because it's using some sort of test time scaling but we we don't know under the hood what it does versus I guess in comparison um since Momo 2 is fully open um yeah you will know exactly how Momo 2 does Okay. So, yeah, let's see again. I think yeah, this is basically the same output.

6:29 Uh, as we saw before, Momo 2 identifies five flips and that is correct. It points to the center of the girl every time she does the flip. Now, let's just wait for the output. Okay, so Gemini's output are also visualized here and we can see that it also does in this case it actually identifies four flips which is one less than the total number of flips she does. Okay, so it looks like Gemini also does pretty well at identifying the timestamps where this action happens. Even though in this case the point is slightly off the girl. Uh I guess this is because Gemini outputs bounding boxes better. And in this case we also prompted Gemini 3 to output the bounding boxes and we visualize both the center point of the bounding boxes for a more direct comparison as well as the raw bounding box output.

7:37 Yeah. So in this case as well the bounding box is roughly correct uh but again it's slightly uh off and not directly on the girls and this is I guess slightly worse. This is actually not um on the girls center. So yeah so this is what Gemini 3 answers to um this question. Okay. Now let's try something else. What if we try um yeah on this video let's try again um let's try a harder query. Why don't we try for example um because this one has the time difference from day to night. Let's try something like point to mouse rainier at night and let's see what Gemini 3 and Momo 2 point to that. So I would expect the model to point to like a later time stamp instead of the earlier one because that's date and then um yeah we asked it to point to the M ring at night. So this one not only test the model spatial understanding but also its temporal understanding. It needs to uh do well to be able it needs to do

8:55 well in both to be able to answer this question correctly. Okay. So yeah, looks like our model just output this time stamp like right around um second six which is roughly at night. Um it's not perfect but I guess it's acceptable. I think a better answer would be even later time stamp but I guess our model can improve on the temporal understanding part of the this task. Okay. So Gemini is still thinking. It's taken I guess more than one minute to think about this query. Uh but that's okay. we will be patient and wait for its answer.

9:42 And of course, we didn't do um any sort of like crazy optimization on the spit. Uh but we could like catch the video and spit it up. But we are not also comparing the spit here. So we just want to see kind of what the output looks like.

10:24 Interesting that it's taking such a Okay, so let's see. Oh, okay. So, this one, um, Gemini actually does a better job at temporal understanding because it identifies their later step towards the end of the video. So, yeah, it looks like Gemini understands time better than Momo 2. um through this one example uh we can also see how it does on other kind of queries. So uh what if we try the same queries we tried before? How many buildings are there? Um because this is a pretty hard one. And let's just run Gemini 3 and Momo 2 again to v visualize the output side by side.

11:20 Okay. So yeah, again we see that uh this is the output by Momo 2. Uh it's basically pretty much like before uh what we've seen before. It just yeah tries to point to as many of those buildings as possible. Uh even though it misses a few ones like this ones uh kind of in the middle um between like small and and this really tall buildings.

11:53 Yeah. Again, Gemini is taking a long time, but we'll be patient and wait for its output. Okay, it the output comes back. Let's see. Okay, nice. So, I think it looks like Gemini identifies the bounding boxes of the buildings pretty well. Uh even though it fails to do this for the smaller buildings uh in the front because they're it's kind of hard to tell them apart. Yeah. So this is how they do and that's it. Thank you.

Questions about this episode

Who speaks in "Molmo 2 | Counting objects and actions"?

Zama (Main speaker).

What is discussed in "Molmo 2 | Counting objects and actions"?

In order: 0:00 Space Needle, Momo 2, Gemini 3, Zama; 3:00 Momo 2, Gemini 3, Space Needle; 6:00 Gemini 3, Momo 2, Mount Rainier; 9:00 Momo 2, Gemini 3.

What is mentioned in this Ai2 episode?

5 people, works and ideas are mentioned, including Momo 2, Gemini 3, Zama, Space Needle, Mount Rainier. Each one on this page links to its own page with every episode that mentions it.

How long is "Molmo 2 | Counting objects and actions"?

The episode runs 13 min; the transcript is about 1,719 words in 14 timestamped sections.

More from Ai2

• All analyzed episodes