Advancing efficient ML

What this episode is about

The discussion focuses on advancing the efficient use of large language models (LLMs) by exploring their deployment across various settings, such as on small devices and neural accelerators. Key points include the importance of optimizing these models while minimizing performance loss during compression and pruning, and the need for new inference optimization techniques tailored to specific hardware rather than relying solely on traditional methods.

Key takeaways

• Democratizing LLMs could be the key to bridging tech gaps, but tuning latency vs. accuracy is a complex art.
• Surprisingly, 90% of LLM latency comes from autoregressive generation, not prefix processing—time to rethink our assumptions!
• Innovative techniques like Tandem and TreeForer could optimize LLMs immensely, proving smaller models can pack a powerful punch.
• Memory bottlenecks hold back feed-forward layers—Heap tackles this idleness, ensuring your GPU doesn't twiddle its thumbs.
• Rethinking LLM architecture is crucial; unlocking their potential requires a fresh perspective on existing optimization techniques.

What is discussed, and when

1. 3:00 – 6:00: Tandem, Google LLC, Lama, Tre forer
2. 6:00 – 9:00: Tandem, Google LLC, Speculative decoding, Palm
3. 9:00 – 12:00: Mat forer

Notable quotes from "Advancing efficient ML"

• Where a lot of context needs to be put in the llm so that it can learn and attend to those context so tree forer is able to solve that problem significantly and then we have…

  about Tre forer · at 3:33

• Capacity model say a Lama 70 billion parameter model and the secondary model to be a much smaller model maybe a Lama 7 billion parameter model or 1 billion parameter model…

  about Lama · at 4:51

• Exactly same as that of the primary model so how do we do that well for that we can combine tendem with this technique called speculative decoding or speed which was also…

  about Tandem · at 6:00

• Model from which we can read of whatever model you want for the application that you have right and that is where Matt forer comes into picture right so mat forer is a new…

  about Mat forer · at 9:02

Who and what is mentioned in this episode?

Companies & orgs

• Google LLC — Company · Business focus · 2 mentions · first at 3:03
  Google is mentioned as the company behind several techniques and models discussed in the podcast.
  For that we can combine tendem with this technique called speculative decoding or speed which was also designed by my colleagues at Google so in speculative decoding what we…

Products & tech

• Tandem — Product · Case study · 4 mentions · first at 3:09
  Tandem is a technique developed by Google to reduce inference latency in large language models.
• Mat forer — Product · Case study · 3 mentions · first at 9:07
  Mat forer is introduced as a new Transformer block that allows for flexible model sizes and configurations.
• Tre forer — Product · Case study · 1 mention · first at 3:21
  Tre forer is a technique that makes attention computation cheaper for large context lengths in language models.
• Lama — Product · Case study · 1 mention · first at 4:51
  Lama is referenced as a model used in the Tandem technique, highlighting its role in optimizing large language models.
• Palm — Product · Case study · 2 mentions · first at 3:13
  Palm is mentioned in the context of models that can be optimized using the discussed techniques.
  Variety of ideas right so for example we have a technique called tandem which is able to reduce the inference latency of palm twist style models by as much as 2.4x without any…
• Heap — Product · Case study · 1 mention · first at 3:44
  Heap addresses the memory boundedness of feed forward and softmax layers in language models.
  From the Ram or the main memory of the device onto a cache sitting on the device right so it is sitting idle Heap is able to address this issue up to some extent by using a…

Concepts & topics

• Speculative decoding — Concept · Key argument · 1 mention · first at 6:07
  Speculative decoding is a technique used to improve the latency of language models without sacrificing quality.
  Exactly same as that of the primary model so how do we do that well for that we can combine tendem with this technique called speculative decoding or speed which was also…

Full transcript with timestamps

0:09 all right so welcome to this research at Bangalore event in the next 10 minutes or so I will talk about our efforts in the domain of democratizing these large language models to a variety of settings like for example like you know how to run them on neural accelerators or on like small devices to work them in a variety of domains like maybe Vision specific domains or language model specif domains as well as to variety of deployment scenarios like do you want realtime solution or do you want a batch mode kind of solution now these models seem to unlock very special and exciting skills as we scale them so one of the most critical questions in democratizing these models and exploiting these skills is how can we optimize the influence of these models so that they can be used effectively in multitude of these scenarios now let's be clear there has been decades worth of excellent research in the domain of systems and machine

1:01 learning to come up with more compressed ml models and reduce their floting point operations or computation operations and successful techniques here include distillation pruning compression whatnot right however as llms are essentially trying to memorize this entire world's information that means that if you want to compress them or prune them it might lead to significant loss and performance furthermore many times just reducing the floting point or computational operations of these large language models is not enough because our modern accelerators can paralyze operations very well and can pack in a bunch of floating Point operations very well so we need to rethink about these problems from scratch so what I am advocating is drilling deeper into these large language model architectures designing inference optimization techniques from scratch for these models and for specific Hardware so let's take a closer

1:55 look at how these generative llms work suppose we are given a uh query or a prefix right then most industry grade llms generate these tokens one by one in an auto regressive manner right and within each token generation itself when we look at what is being computed well it turns out that a bunch of attention computation is being done followed by feed forward and then in the end we apply this large soft Max layer to predict a distribution over next tokens now this prefix or query processing requires running a lot of computation operations right but most of these computation operations are paralyzable which means that on our modern accelerators we can compute this prefix work pretty easily like with very low latency on the other hand Auto regressive generation which requires actually an order of magnitude or even two order of magnitude smaller amount of computation operation ends up taking

2:49 most of the latency requirement that is in many of the standard settings this Auto regressive generation takes almost 90% of the latency and prefix generation is accounting for almost 10 % of latency only which is quite counterintuitive right now A team at Google has been trying to fix this issue uh through a variety of ideas right so for example we have a technique called tandem which is able to reduce the inference latency of palm twist style models by as much as 2.4x without any quality degradation then we have a technique called Tre forer that can make this attention computation to be significantly more cheaper for very large context length this is quite handy when you have scenarios like Ral augmented models where a lot of context needs to be put in the uh llm so that it can learn and attend to those context so tree forer is able to solve that problem significantly and then we have this technique called

3:46 Heap which is able to address this issue of memory boundedness of these feed forward and soft Max layers what I really mean by that is that when we look at these feed forward and softmax operations they are mainly large Matrix Vector operations and here most of the time our accelerator like a GPU or TPU is sitting idle it is just waiting for the large model Matrix to be transported from the Ram or the main memory of the device onto a cache sitting on the device right so it is sitting idle Heap is able to address this issue up to some extent by using a variety of new ideas now in the next few minutes I will provide few more details about tandem so in tandem we basically disaggregate these decoder only llm into two components we have a primary model which is essentially doing natural language understanding that is it is taking the prefill or the query and it generates representations for that query right and

4:41 then we have a secondary model which is doing the actual natural language generation that is it is generating the actual tokens right so imagine this primary model to be a very large capacity model say a Lama 70 billion parameter model and the secondary model to be a much smaller model maybe a Lama 7 billion parameter model or 1 billion parameter model right and to ensure that this secondary model can attend to and exploit these representations of the primary model we have a projection layer sitting between them right so now one obvious question is can the secondary model adapt and in fact uh use this primary models representations to provide more accurate outputs well no prizes for guessing that the answer is yes so we have this tendem technique which is on multiple benchmarks is able to perform much much better than the secondary model in terms of accuracy and is actually much closer to primary model right in terms of its

5:33 quality and furthermore it can provide a significant speed up of primary model that is it can be 2.5x faster let's say right so that's great but still here we are fighting two battles right that is we reduced our accuracy up to some extent while gaining latency so it's not clear like you know how to do an apple to Apple comparison here so this problem we can reduce by saying that let's fight only one battle we will only fight the latency battle we want our output to be exactly same as that of that of the primary model so how do we do that well for that we can combine tendem with this technique called speculative decoding or speed which was also designed by my uh colleagues at Google so uh in speculative decoding what we are doing is through this secondary model we are speculating a bunch of tokens that the primary model will generate right and once these tokens are generated we will run a primary model to verify what all

6:27 tokens are correct and wherever we find a mismatch we will then go to that point and start generation from there on right so here just to take an example we have uh let's say this query which is described the Himalayas for that the secondary model says that the output should be Himalayas is a range right now after generating these four tokens we will pass them through the primary model and primary model says the Himalayas is a good set of tokens but mountain is what you should have instead of range right so we have a mismatch at this fifth token so we will say that okay at this fifth token we will backtrack and we will start generating new tokens from that point on right so this ensures that the quality of this model is exactly same as that of primary model and now the main game is latency that is if the secondary model is able to propose or guess what primary model is going to generate for the next few steps then we

7:21 are good to go we will have very good latency deduction and that is precisely what we have that is by using secondary model of Tandem we able able to get primary models output almost 4.3 times out of five right and that can then ensure that we have a speed up of almost 2.4x over the basic primary model while ensuring that the output is exactly identical to that of large primary model so that's great we have like you know this tandem style technique that can reduce latency significantly without affecting quality right but I would sort of wager that inference optimization is not the only key requirement in democratizing LMS there are many other aspects one of them is that of including flexibility of llms right so for example if you take a step back here how does our llm work like how are they deployed in practice so what happens is we have some sort of deployment constraints that are predetermined we figure out that

8:17 based on our like you know data center or uh clusters and all we can support this much RAM we can support this much of latency Etc right and based on that we will say that from the set of models that we have available so let's say our Palm two set of models let's say this particular bison model or otter model fits best for me right so while I might have capacity for a larger model just because I'm restricted to these four models I have to pick one of them right so the question that we are asking is can we design llms in a way so that we don't have to force fit these models or our application into the available set of models that is can we have a a scenario where we can have a universal model from which we can read of whatever model you want for the application that you have right and that is where Matt forer comes into picture right so mat forer is a new Transformer block where a bunch of Transformer blocks are embedded

9:17 within each other so for example our Transformer Excel block has a Transformer L block embedded or nested within it and within Transformer L block there is a Transformer M block and Transformer s block right so we can have this nested structure embedded within Transformers now what we can do is we can stack these different size blocks together to form different sized models so we can have say stack all the s- sized blocks to get a Model S all the Exel sized blocks to have model XEL all these are sort of embedded within each other so that's great but I am giving you still only four options so where is this sort of you know big flexibility goal being satisfied here so that is where another key idea comes into picture which is that of mix and match that is suppose I want a six layer Transformer architecture what I can do is in the first layer I can have a Transformer M Block in the second layer I can have a

10:08 Transformer s block and so on right so I have exponentially many models to choose from so I can cover pretty wide range of sizes right and I can deploy them but one worry is that I did not really train for those weird mix and match models that I'm generating I just trained for four models so why should my uh llm perform properly on those weirdly generated models well that is where a little bit of magic happens that is what we observe is that our Baseline models our anchor models like um XL model L model M model and S model not only they are very accurate compared to a baseline architecture which is trained from scratch we are able to observe that these mix and match models that are represented by stars here they are also working almost as good as what we could have hoped for so that means that we can basically train just one large model and we can train hundreds of thousands of the smaller models which are working

11:03 pretty well right and the exciting thing about mat forer is that is not specifically restricted to say language domain uh it can be applied to Vision it can be applied to multiple sort of deployment scenarios that is it cannot it can also be used for retrieval it can also be used for embedding generation it can also be used for your let's say image classification so in summary we are advocating for llm specific inference optimization techniques as a key aspect of democratizing these LMS and another key aspect that we are advocating is that of flexibility right and that we believe like you know would be important to ensure that these large language models can be democratized to multiple scenarios and applications thank you so much for your time happy to chat more about these topics during

11:58 breaks

Questions about this episode

What is "Advancing efficient ML" about?

The discussion focuses on advancing the efficient use of large language models (LLMs) by exploring their deployment across various settings, such as on small devices and neural accelerators. Key points include the importance of optimizing these models while minimizing performance loss during compression and pruning, and the need for new inference optimization techniques tailored to specific…

What is mentioned in this Google Research episode?

8 people, works and ideas are mentioned, including Tandem, Mat forer, Google LLC, Speculative decoding, Tre forer, Lama. Each one on this page links to its own page with every episode that mentions it.

How long is "Advancing efficient ML"?

The episode runs 12 min; the transcript is about 2,175 words in 14 timestamped sections.

More from Google Research

• All analyzed episodes
• People mentioned on Google Research
• Topics mentioned on Google Research