23578 Efficient Large Vision Models

What is discussed, and when

1. 0:00 – 10:00: MIT, Tiny Chat, CPPR
2. 20:00 – 30:00: Tiny Chat, Llama 3
3. 30:00 – 40:00: Runway, Sora, Stable Diffusion
4. 40:00 – 50:00: Stable Diffusion
5. 50:00 – 1:00:00: Stable Diffusion

Notable quotes from "23578 Efficient Large Vision Models"

• Speaker is very good sonan from associate professor at MIT his research group the H group specializes in efficiency in neural models and his group made important contributions in the field of model pruning quantization compression.

  about MIT · at 0:00

• Score there's a new Benchmark called video QA Benchmark video mme we also rank pretty top on the open source version but is behind jam and gbt 4 o and gb4 V but not too far so here are some examples of visual language mod of the in context.

  about GPT-4 · at 7:28

• Available for both MacBook Intel and the media book and different laptops and to close a loop tiny chat also support visual language models okay so we have this llama 3 V 1.5…

  about Tiny Chat · at 25:36

• Mature technique to quantize those visual models yes we have tried diffusion model to quantize diffusion model also using the smooth Quant so that technique that res that intuition to migr quantization difficulty is irrespective of vision.

  about Stable Diffusion · at 28:25

• Measure the latency on device so what's the most standard way to calculate efficienc yeah there is a good Benchmark called ml perf which is the industry like Olympia game from…

  about MLPerf · at 30:02

• There's couple here walking hand in hand people walking talking on the streets and today Runway announced their gen 3 model with this super impressive sample which is a…

  about Runway · at 32:07

Who and what is mentioned in this episode?

Companies & orgs

• MIT — Company · Affiliation · 1 mention · first at 0:02
  MIT is the affiliation of the speaker and his research group, which specializes in efficient AI models.
• Runway — Company · Referenced company · 1 mention · first at 32:13
  Runway is mentioned as the creator of the Gen 3 model.

Products & tech

• Tiny Chat — Product · Case study · 3 mentions · first at 5:54
  Tiny Chat is an efficient inference library mentioned for deploying AI models on edge devices.
• GPT-4 — Product · Referenced product · 3 mentions
  GPT-4 is mentioned as a benchmark for visual language models.
• Llama 3 — Product · Referenced product · 1 mention · first at 25:48
  Llama 3 is mentioned as a model that can be deployed using Tiny Chat.
  Laptops and to close a loop tiny chat also support visual language models okay so we have this llama 3 V 1.5 and V 1.5 13 billion parameter model okay so FP 16 in the fourth…
• Stable Diffusion — Product · Referenced product · 4 mentions · first at 36:51
  Stable Diffusion is mentioned as an example of a model that can be quantized.
• Sora — Product · Referenced product · 1 mention · first at 34:58
  Sora is mentioned as an example of a video generation model.
  Going on there's so much interactions happening and the same is obviously also true for stuff like this pretty famous sample from Sora we have like cars driving in the…

Places & events

• MLPerf — Event · Referenced event · 3 mentions
  MLPerf is mentioned as a benchmark for measuring the efficiency of ML models.
• CPPR — Event · Referenced event · 1 mention · first at 6:02
  CVPR is mentioned as the venue where the V visual language model was presented.
  Those text only capabilities so those are some of the learnings from vaa on The mmu Benchmark is on the Tex set is currently the best one on the open source category so the…

Full transcript with timestamps

0:00 speaker is very good sonan from uh associate professor at MIT his research group the H group specializes in efficiency in neural models and his group made important contributions in the field of model pruning quantization compression uh among many others this work has been cited over 50,000 times and today's talk is about a slightly different part and namely The Edge deployment of a vision language model called okay hello everyone I'm glad to talk about our research V and efficient visual language models so today's AI are too big so I've been working on efficient air Computing for the past decade particularly since 2016 the work on deep compression started using pruning and quantization for mod neural networks to reduce the number of weights and reduce the Precision per weight followed by the efficient inference engine which can directly accelerate inspires and compress the neuron network

0:58 with the algorithm and Hardware code design and recently pruni and sparcity are getting quite popular with respect to the number of Publications so that's some background uh later I worked on NE architecture search basically the on for all Network where uh you can TR in one network and surprisingly you can get 10 to the 19th different sub networks that can independently operate uh so then we can search the best sub network based on the hardware resource to enable such Hardware aware new architecture search so different different from pruni and quantization which compress an existing model this technique actually can design a efficient and compact model to start with and reduce and amortize the search cost by training this M for all model and then have these different specialized models depending on the model constraint the hardware constraint for example you can have a lot of resource like in the cloud AI we can run

1:56 a larger sub Network on the right hand side if you only have a microl controller we can run a tiny model and this is achieved by Weston model which is the winner for uh three low power Compu Vision challenges and also is the core technique of omn startup acquired by nedia now we also imedia also enable that for w for networks for large language models so later I worked on the project called mcet to go one step further bring AI to iot devices they are much smaller and much tinier we codesign the tiny Nas with the tiny inference engine for algorithm and system codesign which is very crucial especially when the design space is very large and also the uh conine is pretty heavy so we can reduce the memory footprint by almost 2x every year until just a few uh kilo tens of kilobytes to do this person recognition detection different tasks very lightweighted just a few hundred kilobyte so this is about M unet for

3:00 inference and we believe on the edge is very crucial to also perform training so we performed UND devised training under only 256 kilobyte of memory on this microcontroller uh this is enabled by uh three techniques so one is sparse update so not all the layers are crucial we can sparsely update the layers and we can use uh quantization aware scating to enable eight 8 bit forward and backward locally on the micro controller so uh quantization we getting really just two lines of code what can make the training very stabilized and finally we work on Tiny training engine to implement the algorithm saving into measure the speed up so initially the model cannot detect a person versus no person now we have two buttons we can press the green button to show there's no person and press the red button to show there is person to teach the microcontroller and everything running locally on the iceam and and Flash Flash is read only only

4:03 the iceam is vable and after a while here the microcontroller can recognize person is in the red on the top right corner and no person on the on the top right corner to show in green and also it can transfer to the uh real life setting to detect person versus no person learn purely on the microcontroller so those are some of the background which I categorize them as h ai1 1.0 okay so these models are highly specialized for different tasks like on the bottom um you have to train different models and different data to for different tasks and the lack of negative sample for training really makes device AI more challenging with limited generalization and failure of corner cases so with the advance of uh visual language model HF 2.0 enables this general knowledge with General models with the World Knowledge okay General model means we train just one model to handle multiple modalities multiple

5:06 tasks uh those are enounced by large language models World Knowledge okay we uh I believe language is a very condensed version of knowledge and learning from language can enhance those visual perception um with Advanced reasoning capabilities instruction following proficiency and they are even Deployable on the edge so I will talk about the whole pipeline by first training the multi model uh the first multi model first talk about the multi model pre-training method for V the visual language model and Then followed by the quantization technique the awq quantization approach for four B quantization that can make a large V small V so that we can deploy them on the edge device ad Oro even with the tiny chat engine so those are the three sections of the main talk let's start with v I'm pre tring for visual language models at this year cvpr so we find that the data and tring recipe matters more than the

6:11 architecture so VA is using the same architecture as lava but uh enabled many more interesting capabilities for example we find Visual instruction tuning is not enough we want to uh to do visual language preing okay to UNF phrase both viit project and also the um the and image text pairs are not enough but inter image Text data is very cral so here uh we have the image token the language token with the projector concatenate them together but using the inter image text image text is very crucial and the visual in context learning requires un phrasing the so here we have three training steps initializing the projector inter leave the pre- trining and visual text joint as F during the joint fsft phase we find re blinding text only data can recover those text only capabilities so those are some of the learnings from vaa on The mmu Benchmark is on the Tex set is currently the best one on the uh

7:17 open source uh category so the yellow parts are closed source and the blue green is ours the white ones are the open source ones and this is the overall score uh there's a new Benchmark called video QA Benchmark video mme uh we also rank pretty top on the open source version but is behind jam and gbt 4 o and gb4 V but not too far so here are some examples of visual uh language mod of the in context learning and multi-image reasoning capability like giving the company logo we can automatically deduct the tasks is to give the product name and also the home to outstanding Raman and here is can say home to the greatest Pizza to talk about the food associated with the city versus Multi Image understanding which is a cool feature enabled by IM inter image text pair for example asking what is the total price on the of the beer I can say the beer on the table is Magna by OCR $6 each on the manual two

8:23 of them by detection and then you can tell is $112 in total so this is visual uh Chain of Thought like giving the prompt think step by step it can answer it correctly and reasoning across multiple image as a new feature in VA is critical like this case for environmental study we can ask what is the implication of temperature can tell the temperature is getting warm because the Arctic Ice is Getting Thinner from 84 to 12 VA is also capable of video understanding thanks to the interlift image text pair inter leave image Text data for example here what might be the next step according to this video is going to say is uh going to bake it in the oven similar in this case what is likely to happen next you can tell it's going to blend the strawberries in the blender which can help building the world model for robotics to understand the world and Next Step likely here what is likely happen next and it's going to say it's grind

9:27 and mix the ingredients together together which is actually having quite good reasoning capability and the modeling the world like in this football game I can tell what is happening in the video the ball goes into the goal celebrates with his teammates the crowd cheers and hugs each other another example of V to help pus driving where s is going to present some of the demos as well so uh we use this demo actually I should this video when I was driving the Tesla and just feed it to V to test our V's performance you can say you should reduce the speed using the low beam highlights and keep the safe distance from other vehicles for handling such Corner cases another example for V for autonomous driving given this image should you honk at the pedestrians crossing the road it says is not appropriate to honk it's implied and disrespectful we didn't learn those from the driving school but we learned from textbooks when uh so I

10:32 think it's more efficient to combine these large language models with driving tasks not only exterior but also in the car the driver monitor system is the driver distracted yes the driver is distracted this is the same model actually U but can handle a lot of different scenarios without having to train a particular data sets showing pretty good um generalization capabilities there's an interesting case here here with zero short learning V for healthcare will the practice cause pain to the patient in the image and V says it's a stimulation not a real patient does not cause pain it's a manquin not a leing person and one of my friend went to Italy and sent me this image uh and as where is this it's going to say uh I can correctly tell where it is although it feels not quite straightforward but showing the world World Knowledge is pretty good and there's another example for V for video according to the video

11:43 finger for those who know how to play guitar watch the entire video and several tutorials and finally the little finger is now used how many kinds of for are played with the video is going to say seven all together is sh seven different fors the amazing stuff is we also enable V to be Deployable on J on Jon oring which is the edge computer by Nvidia for driving applications so we can deploy it locally on this Edge device without using the call GPU for robotics applications or driving applications which is a usual body scene is going to say several safety concerns for example one turbine and here in the in the factory what is how many cars are jacked up there are two cars jacked up in the image there's a person whose head is under the car what color is gloves is orange so we can enable this a applications using v um on a wide range of tasks so how do we deploy v such a big model on Jon oring so there are two

12:55 uh considerations the first one is to reduce the model size the smallest of vaa is only three billion parameters um so and the next one is seven 8 billion 7even billion 13 bilon all the way to 40 billion uh for and then we use a quantization technique which I'm going to talk next to further compress the model so the three bilon parameter vaa after quantization is only 1.5 Gaby so I'll first introduce smooth Quant a popular technique to quantize large language models is now available in Media P rtlm there is plenty of work on quantization in the cnh but we find what is new for the large language model is that there are lots of activations that has huge outliers they're having very big spikes in certain channels like here we have three channels that has big numbers that is going to clamp all the other uh but going to waste the dynamic range because all the blue activations will be clamped to zero but the weight

13:56 is relatively quite easy to quanti the range is notice here is five by here is 70 so here we have a CL solution to actually migrate the quantization difficulty from activation to weight since matrix multiplication is linear we can multiply the uh activation Channel by a number like 0.1 and multiply The Weight by a large number like 10 so my 0.1 * 10 is still 1 by one thus we can migrate condensation difficult from the activation to the weight so the activation you can see is much smoother the weight is a little bit bumpier but still very easy to quantize here is a real world uh here is a simple example we have the activation two channels of out layer we have 16 times the weight one uh we find the square root of that to be the quantization scaling Factor so that we can quantize them so that 16 * 1 = to 4 * * 4 okay so they are still mathematically equal similar here 8 * 1 = to 2 * 4 okay so they're

15:07 mathematically equal but the tensor activation tensor here is much more smoother than the left uh in practice rpha 0.5 is a very good hyper parameter already but if you want to do a little more carefully you can actually search uh this uh scaling factor from 0 to one and then you multiply x with the s minus one and S on the right hand side associate with W so they cancel each other okay for having the inverse of w s cancels each other this can be folded into the weight so no inference overhead this can be folded into the iog of the previous activation layer so no inference overhead if you implement them efficiently in the system and the the result is to mathematically equal so that effectively get rid of the uh fine tuning and even without fine tuning we can very well recover the accuracy and having the same even shorter latency dis spite we can have four gpus rather than eight gpus and having the GPU

16:12 memory so without fine tuning reducing the number of gpus by half and maintain the same latency okay so uh that's the streaming uh that's the smooth Quant what about UND device L we find a key challenge is actually the uh weight memory so the weight memory is bonding the inference latency okay so um especially at the low batch size when we are deploying deploying V on the edge uh memory is bonded and memory is bonded by the weight so here similar to the E which had designed like eight years ago we used the 4 bit weight and 16 bit activation and 16 bit arithmetic so that we can focus on reducing the weight and here we use the activation aware or awareness to preserve the sailing weight Channel by scating according to the activation magnitude not the weight magnitude as a result we can deploy them on a laptop and this is the key technique key quantization technique behind M as chat with RTX where you can have a laptop and

17:19 run large language model 7B 13B locally on the laptop uh so theoretically from 16 bit to 4 bit you can have 4X saving and we have about 3x measured speed up compared with fp60 it also get the best paper award at mlc so I'm going to spend some time to talk about the details originally using round to near is you have a big degradation of the perplexity so how do we deal with that surprisingly we find keeping only 1% of the channels unquantized can actually great greatly improve the perplexity but um how do we select those setting channels and can we also avoid those uh floating Point numbers and mix the Precision and you avoid that actually conventional Wim says if you're quantizing the activation you should look at the activation but quing the weight you should look at the weight but actually we find it's helpful to look at the activations okay if you just look at the weight um scaling according

18:24 to the larger weight you don't scale that if the weight is large if the weight is large then you don't quantize that this recipe doesn't work well random recipe doesn't work well either but there is a big Improvement if you look at activation okay if the activation channel is large then you don't quantize that weight channel so that only solve the first issue look at the activation activation awareness there's to the second issue okay is it really necessary to introduce mix Precision you need a specialize kernels you need a different treatment for building a product this is especially challenging for users to try introducing too many hyper parameters actually we find mixed Precision is not necessary there is a more clever way uh to reduce the perplex perplexity which is by scating up those sting channels okay so the second channel is Sting we only need to scale it up like by 1.5 by two by four the prop plexity

19:25 immediately drops we're quite excited about that that and intuitively this is due to if other channels Remains the Same you're scating up one channel by 2x you are effectively adding one more bit imagine you have 0.5 and one previously they are both quantize to one if you use run nearest but if you multiply them by two 0.5 becomes one one becomes two and now you can distinguish one and two okay so you are effectively adding one more bit so this is a mathematical Reon why skating up that channel certain channels not all of them can reduce the quation error so we have a quantization scalar and determined bybs Max so the uh the quanti verion times x equal to the scaling Factor thank you times the rounding result and times the activation um so if we are scating up the uh weight Channel okay the rounding still error still stays the same because it's minimum value is zero Max is 0.5 since 0.5 is the maximum rounding error

20:40 the mean is actually 0.25 okay so it stays the same but if you're scating up the S the total result will be smaller assuming you are not scating up everything so the total dynamic range stays the same so using this technique we can evacuate reduce the perplexity and they this can be also similarly implemented as smooth Quant so for building a product is super easy if you have smooth Quant you can easily have awq using this uh mathematic Eco equation to have S and S inverse multiplied with w and and X respectively so it's performing quite well when we are do doing the uh perplexity analysis this is the number of calibration sequence with smaller calibration sequence the perplexity is much lower and the generalization capability is quite well since I like gptq which require calibration set and require back propagation to do optimization which tend to over fit uh ours doesn't over fit since it doesn't

21:50 have any back propagation and using two different data sets one for evaluation one foral caliberation the differ is much smaller very close even if you have different calibration and evaluation data sets 0.5 versus like 4.8 this is the accuracy of awq for into three into four uh 5.6 versus the fp6 5.47 uh perplexity which is quite well maintained so smooth Quant and awq are now available that tens RT tens RTM is the state of the art inference framework from Nvidia and it can enable a Falon 180b parameter model in a single h200 dpu using in 4 awq thanks to the um large model capacity large memory 141 gigabyte and can Feit 180 Bon parameter model and now it's quite widely widely used with over 1 million downloads from hacking phas we are actively maintaining that uh with have also buil a tutorial how to add support for new models so feel free to check it out together with this companies and finally the last step how

23:10 do we deploy them how do we implement the forbit Quan High model since conventionally computer architectures computers are bite aligned minimum units is one bite is 8 Bits so handling the dequantization from 4bit to 8 bit is very crucial and how do we use the minimum number of instructions to to De quantize the max number of Weights that's very crucial so here we use only three instructions only three logical instructions to decode 32 numbers okay by this Hardware orware packing so rather than arranging them arranging them in the linear manner w31 and 30 all the way to w0 we align them or we pack them in the inter Le manner okay so W3 1 followed by W5 and W6 w0 okay so we can shoot we can mask it by Z F0 F to get the lowest uh number of bits and then we can also shift the right by four bits and then mask it to decode the lower bit okay so Al together we can use only three instructions um a shift a mask or two

24:23 masks to decode 32 numbers and is much faster than if you don't use the packed interlift packed weights and also we we implemented the fuse kernel for tension and if fuse the kernel for Jam to reduce the D axis so compared with fp16 uh we are about 3x faster compared with fp16 on 490 GPU on laptop GPU and also on Jon or mobile GPU we build a tiny build a tiny chck computer actually uh so that you can check locally on computer with 3D printed this box is having a j Oro in the back very cute it runs the S billion parameter model uh with relatively uh real time latency so such data privacy and security is well maintained by such on device this is on device this is the device on device deployment um even without internet for customization and personalization so this is a running on the laptop feel free to download it uh the from Tiny chat GitHub it's available uh for both MacBook Intel and the media book and different uh

25:42 laptops and to close a loop tiny chat also support visual language models okay so we have this llama 3 V 1.5 and V 1.5 13 billion parameter model okay so FP 16 in the fourth the a100 tokens per is improve is almost pretty much more than doubled 490 tripled or also tripled and the performance is actually very well accuracy very well maintained and here you can interact with the image like what is unusual in this image there are lots of deers have to be careful Crossing this crossing the roads and should the driver uh keep going and it's going to say no the driver should not keep going because of the deal okay and also we can run visual language model on your MacBook all right to summarize we uh talk about AI 2.0 with the visual language model okay uh we need full stack optimization uh with three parts from training and application side and also on the system and hardware and inference side on training we talk about

26:51 this training recipe called V the v a visual language model V is actually driving up the demand of computer Computing model compression bridge the gap between supply and demand of computing by using 4bit awq we turn a large model into a smaller one so that I can alleviate the memory band memory bandwidth imitation and followed by the tiny chat efficient inference Library we can quickly deploy them and using the a b aligned computer to perform such system and the code design to enable the theoretical saving from forbit position into measure the speed up all right that's all my presentation thank you thank you very impressive work and and glad to see that so much of it is open source as well audience questions and if audience ask question could you please repeat it as well yes

28:04 so the question let me repeat the question is do I specifically quantize LM part or the VI Part I did quantize the vision part because we profiled the latency and it occupies not more than 95% of the latency so we only quantize LM part for visual part I believe it is relatively mature technique to quantize those visual

28:34 models yes we have tried diffusion model to quantize diffusion model also using the smooth Quant so that technique that res that intuition to migr quantization difficulty is irrespective of vision task or language task as long as Transformer matrix multiplication the same principle will hold

29:02 so are you also working oning of right soia we're also working on that part so there are two challenges one actually the actually one challenge is the fine tuning we find for quantization it's a low high fruit because you don't need to find T and for reducing the weight you need fine tuning to reduce the accuracy job and either using pruning or architecture search actually media has this technique using this once4 approach to generate many sub networks that can independently operate generalize it from see large language models as well but iire me right one last question thank you so much for the talk uh I was wondering what is the most standard way of measuring the efficien lat of ml models is it like floating Point operations or should we directly measure the latency on device so what's the most standard way to calculate efficienc yeah there is a good Benchmark called ml perf which is the industry

30:11 like Olympia Olympia game from from the industry uh so there are several Dimensions okay one is the throughput one is the latency okay a single batch and large batch you have to consider both of them uh for single batch is memory bonded for larger batch is sometimes even compute bonded so you have consider both and also there is the open track and the Clos track closed track you cannot change the model architecture you can do quation but open track you can also do PR you can also do neural architecture search so feel free to check out the ml perf Benchmark with some standard workflows me know yeah I think you're yes beautiful okay please yeah um yeah as I said um we tried our best to contribute to this space in the past um and yeah yesterday uh I actually realized that there's a lot of diffusion talk distillation talks in this in this Workshop um so I last minute added some other stuff uh to make it a little bit

31:15 less focused on desolation um I think we had like yeah um se's talk before and I think Tim's uh on different distillation methods I'll add our perspective to that which is based on at serial training um which really enables like super fast synthesis in one or two steps um but also talk a bit about other stuff so um I think it's pretty impressive actually where we got so far so this is this is a sample from a text image model um and today these models are able to generate very complex scenes you can really dive in you can explore what's going on there's so much interactions happening um and the same is obviously also true for stuff like this this pretty famous um sample from Sora uh we have like cars driving in the background there's couple here walking hand in hand um people walking talking on the streets um and today uh Runway announced their gen 3 model with this super impressive sample um which is a

32:23 middle-aged sad bald man becomes happy as a wake of curly hair and sunglasses Falls L on his head and this is I think is super impressive it really follows this prompt prompt quite well um and what what all of these have in common is that we got there through um one major ingredient which is uh scaling obviously so um the architectures and the algorithms that we use to drain these image and video generation models they are pretty scalable um we have a lot of data away aailable in both domains um we have the scalable algorithm which is the fuse that has a stable pre-training objective and we have scalable architectures and and this combination is very much uh the ingredient that these are the ingredients that you need to achieve like this high quality on the left here is this standard classic plot from the gbd3 paper uh which really visualizes the scaling loss and on the right we have something from the

33:25 diffusion Transformer paper uh which also shows that model quality sample quality increases when you increase either the model size or um the sequence length so both both contribute to the amount of flops that go into the model during training or you can of course also just increase both um and yeah this this works pretty well uh so question could be um is that all we need are we done and um sorry I think uh to some yeah that that that is an answer if you're just interested in raw performance and have like a lot of money a lot of compute um then that is a very reasonable way um but obviously there's not all institutions have infinite money or infinite compute um if you train a large model uh that is typically on the order of millions of dollars to train um so one major motivation ation is to find still very scalable um training algorithms but make them more efficient um so you want to make sure

34:36 that you're not wasting your precious compute and stuff that your model can't really capture or that you as a end user of the model don't really care about um the other is uh inference so at these state-of-the-art models uh image generation is typically on the order of seconds uh video generation is on the order of minutes we don't we know for Sora and gen 3 for Luma which is another video model that was released last week it's 2 minutes so it's pretty slow um it's far away from the Real Time experience that um s just talked about and I think that motivates um another field of research which is just like finding these more efficient sampling methods for these pre-trained models so this is what I want to talk about first part is some work that on finding more efficient and scalable uh training algorithms models and the second part will be about these more efficient sampling methods and the

35:36 first one is um from this paper here that we published in March scaling rectified flow Transformers for high resolution image sythesis um and and basically our motivation for this was that back in November we were really sure that we wanted to scale our models to larger sizes before that we had published s L uh which was it's a medium I think in today's standard probably a small model and um yeah we really wanted to scale it up but on the other hand we didn't have infinite resources so before we went into the full uh scaling phase we were really really asking ourselves do we yeah what what kind of diffusion formalism do we actually want um to scale what kind of architecture do we want to use and um there's especially in in the diffusion field there's so many different formulations right you have the classic ddpm you have an Epsilon prediction you prediction you have EDM you have stochastic

36:38 differential equations there so many different formulations of the sometimes of the same thing um sometimes they're a bit different um so there's really the zoo of different formalisms on the other hand you have different architectures um stable diffusion for example was based on on a unit convolutional unit um more recently shifting towards Transformer based models so um the diffusion Transformer that I mentioned before it's one example it's just like pure Transformer um other examples are uh something that's called uvit which is a mix of a a few convolutional layers and a Transformer model so also there there's like this this zoo of different architectures and we really wanted to make sure what to find out what is actually for all use case text image synthesis um the best controls such that we don't race to any compute um and yeah before I go into into that I want to do like a quick um

37:36 quick intro actually or recap or to to flow matching because that's what we ended up using uh in the end so I'm pretty sure that most of you have seen this um this form here formula uh this is uh just a way to construct the training data for diffusion model typically so uh you you have some training data the training data distribution P data you sample data point from that and then you add noise to it and you do that by so-call noise schedule which is defined through these prefactors a and BT and yeah um in depending on how you choose like at and BT you can recover these classic formulations like EDM ddpm prediction and stuff like this but one of the methods that we also ended up using in the end um that's gaining like more popularity recently is this so-call rectified flow belongs to the class of flow matching models and um that one basically just interpolates between your data and noise

38:45 so a is one minus t BT is t t runs from zero to one um and um if you plug that into the the loss formulation for your for your um conditional flow mat in uh you actually end up with this loss which just means that uh your prediction Target for your neural network is the vector the points from data to noise and after after you've trained that um you can sample from the model by just solving this differential equation um and on this slide you basically have all the ingredients uh for implementing uh this rectified flow framework it's super simple that's why I a huge fan of it it removes a lot of the unnecessary um theoretical overhead that you have in in diff from um formalisms or formulations and it's actually also more General than diffusion models um just like very quick as an overview um it is um it's not necessarily restricted to that forward path that I had on this slide here you can basically interpolate

39:54 between different distributions um as a wish um the one that I have chosen here is the one where you can recover classical G diffusion um but but yeah you can basically use everything um and that makes for interesting applications like you could interpolate between a low resolution and high resolution image if you want to do something like upscaling or stuff like this I think this just has a lot of flexibility but the main feature of it is in my opinion is much simpler um so yeah we do not need these complicated Frameworks anymore um and we can just WR this down in a in a few lines of code um super straightforward to implement so yeah huge fan of that can highly recommend to look into these papers they basically all discovered the same thing at the same time um and yeah that's um that's that's what we uh also ended up investigating in addition to the classic um to the classic formalisms and one uh one thing that

40:57 made the initial diffusion model super successful um in the beginning was ddpm in 2020 was that it implicitly discarded scales noise scales that were perceptually irrelevant so the loss was designed in a way such that it was trained on the relevant perceptually relevant scales per default um yeah which just makes the training CLE more efficient and um we were asking ourselves the same might be true for this for this rectified flow formalism here especially if you consider um that for the end points of the trajectory so for tal Zer and tal one you're either predicting the best solution to predict is either um the noise mean or the data mean and there's then not much more to be learned so one intuition that you can derive from this is that you don't don't want to sample all of these levels uniformly because then you're wasting compute and stuff that you don't really count for the reduce or which is basically trivial to

42:06 predict um but you rather want to focus on the scales in the middle of the schedule and this is what we did here we chose distribution to parameterize uh the sampling the time step sampling distribution with this loged normal distribution um this has two parameters uh me and variants and uh those you can tune to kind of bias the time Steps either for towards higher noise scales or towards lower noise scales um it gives you some flexibility to to investigate which noise scales really matter in training and then we U yeah trained a few models um and compared that to Classic Epsilon the Epson prediction with a linear schedule V prediction with a coine schedule EDM all these diffusion formalisms and actually inter setting I think this was class conditional synthesis image net lat defusion uh we found that the rectified flow with this new um focus sampler gave the best results you also see that the rectified

43:18 flow formalism um this is just a feature by which it which it has per default um is better in the low step regime so between 10 and 20 steps it achieves the best results um and yeah especially around 10 steps it is it is still quite good all right so that is on finding more efficient noise scales to sample form during training the other thing is that I mentioned in the beginning is we had like these different architectures right so unet u d different especially with units there's like so many different type of parameters you can tune um if you want to scale up that kind of makes it impossible to search the full space um so we actually ended up discarding the unit and just uh looked at different Transformer based implementations which is the uvat DAT a variant which um has cross detention layers from the text embeddings and then something that we um developed for this paper which we called

44:24 M mdat multimodal diffusion transform for and this model um since we were doing text IM synthesis um treats the two different modalities text embeddings and image embeddings in two separate streams within the within the Transformer model so we have um weights that handle text embeddings weights that handle the image embeddings and then in each block they are joint through ad joint attention layer so that model has the same basically the same amount of flops as a classic dit but doubles the amount of parameters and if you look at the um the metrics here that we calculated over the course of training you see that this architecture actually performs best so that's what be used and then uh scaled up and we got some pretty nice uh scaling plots I would say uh as an side we also found that the validation loss is actually a strong predictor of model performance so you don't really need to sample the model which is us usually

45:32 kind of slow and a bit um TDS to do during the training uh but this is actually nice because the validation of something that you can uh compute in a forward pass so yeah and if you plot that validation loss um for different model sizes we get a clear scaling Trend which is pretty nice to see and uh of course this is much more interesting uh samples from the model uh was the largest model which went then was an 8B model uh which was pretty competitive with stateoftheart text image models like ma journey and ideogram and it also solved this problem that a lot of models have solved in the meantime which is the taex rendering um where one aspect as we think is that it really has like this dedicated um text Transformer stream all right but um this brings me to the second part so we have this nice model it's actually quite huge if you want to sample from it it's pretty slow uh usually even on

46:37 like very Modern Hardware uh like A1 100s h100s you have to wait a few seconds to to get a sample out you still have to use multiple steps like 50 steps to get the best results um and this is yeah it can be can be pretty frustrating especially when you're designing a talk and then I realized oh man this has a classic AI hand here if you look at it so I did it again uh the laptop is still floating in the air um but yeah um I think you get a point so we want faster faster synthesis and um we've seen a bunch of things now here in the workshop um but the one that I want to talk about is this address serial diffusion distillation which is a line of work uh that we contributed to this distillation research and um it actually started with a failure so we we really wanted to get to like one step text to image synthesis and our first approach this was something that one of my uh co-workers Axel has a lot

47:38 of experience in which is pure Gan training so he had done like this work sty g t at Nvidia uh which is a large scale text to image pure Gan model and we said okay let's try this let's let's make it even better um but yeah as you can see here after training and we actually spent a lot of time and resources on it it didn't really look super good it is okay um but it has quality issues and it has mode collapse issues the face of this teddy bear here looks always the same same for this prompt which is one of our favorite testing prompts cinematic shot of Professor sloth wearing a tuxedo at a barbecue party um yeah it's not really something that you I don't know I want to publish as some high quality results um so we looked into other stuff that was out there one is uh consistency models and consistency distillation there was this model LC MXL which is um latent consistency model applied to stable

48:37 diffusion XL and actually for this model the one step samples looked already pretty decent so you could see like diversity and structure except for the fact that obviously the the quality isn't there um it's it's blurry and um yeah kind of one one like one intuition that you can have when you look at the stuff is that very far away it kind of looks as you just would need to add a bit of image detail to it to it and then you're done so that was our motivation which we thought okay we thought yeah let's let's use this and combine it with an ADV serial objective um and this ended up being our method here ADV serial diffusion distillation addd where we started from stable diffusion XL as a teacher model um initialize the student model and I'm just talking about the unit here um from the same weights and then combine two different losses a distillation loss which just is uh yeah which just tells the student to match

49:39 the teacher output for certain time steps and adviser loss which we implemented through a discriminator that discriminator was based on the same discriminator as the St again um T discriminator which is based on features of dyo V2 um a convolutional network so for that we had to go into pixel space apply it and then calculate the loss um and yeah that really improved the results over this uh LCM method so these are again onestep samples and we see that they're much crisper was pretty um pretty good actually uh I like when my coworker first shared a notebook with us where he demonstrated the inference I actually couldn't really believe it how good it was because it was I thought it was there must be a mistake but it was just a single step so was really nice um the other uh nice feature about this is that it or or we can maybe actually see how it works here is that it it reduces the output diversity of the teacher quite a

50:43 bit but it's able to achieve a bit higher um image quality and crispness so in the bottom we see a sample for from the teacher or four samples from the teacher in the top we see um four from the trained student and yeah the images are crystal but the diversity the Poes are much less um it's much less divers same for this um the problem with this is is that this approach is not really scalable because as I said we had to decode into pixel space to apply this DT based discriminator backbone and um that discriminator backbone was also just is only applicable on a fixed resolution so you can't really which is I think 580 58 pixels so you can't really go beyond that um nowadays the standard in text image is at least one megapixel Generation Um so yeah that makes it pretty pretty pretty hard to scale and especially the fact that we go out into pixel space back propagate through all of these steps makes the training really

51:52 really expensive and you can't really go to higher uh to larger models it's just just very um a lot of work so this motivated our follow-up work to this this first um advas serial distillation work which um a latent advis serial diffusion distillation and uh kind of Natural Evolution I would say um you like the main motivation was we should really avoid the decoding step into the pixel space um and you do really get the best result results in these adversarial training scenarios when you have like some pre-trained feature extractor where you can start from right that was in the previous method add was just this stov V2 discriminator in the lat space we usually don't have that so we thought okay let's use the pre-rain diffusion model which is our teacher anyways as a feature extractor um and that's what we yeah what we tried here um so we have the teacher on the one side um and the discriminator but

52:55 it's actually the same model except for the fact that the discriminator has some learnable heads on each of the um layers where we extract features from and then we compute the adviser loss based on these learn features we also initialize the student from from the teacher model again like this all um all the same uh Transformer architecture so yeah we we also switched uh to the Transformer based architecture that we did for stable diffusion 3 this was not um not the uh unit from stxl anymore um and another nice feature about this is that is like modality agnostic because since it's a Transformer it just operates on sequences so you can also use this for video um for example and it also natively supports multi-aspect uh resolution um high resolution because we don't have to decode into pixel space and that means we can distill much larger models and yeah it also um enabled us to get rid of the distillation loss which was

54:02 kind of a debatable thing to do because adversarial training is some people think it's a bit of a heck um some not I think it's it's actually great for for achieving this under constraint resources um and uh yeah what what we saw is that actually the setting with just the ADV loss without the distillation loss works best and since we were using this uh pre-rain diffusion model as our feature extractor for the discriminator another um property that this model has is that we can actually during training control which scale the discriminator should focus on because since since it's uh using these diffusion features um you already have to pass in a Time step and um yeah noise noise latent and if you uh in very high time steps or noisy latens then usually these correspond to low frequency features and if it's low noise scale it typically corresponds to high frequency features so we can bias the discriminator into

55:10 certain directions U we did that again with this loged normal distribution I described earlier and uh yeah we found that we actually have to shift the schedule a bit to the right to higher um noise scales to achieve the best results of this technique and then the last ingredient that also helped here is to apply um direct preference optimization so both through the teacher model um and the student initialization but also after training the advis serial student we can reapply the lower weights uh that we trained to achieve the DPO version of the model to the trained um student and that actually works out of the box it's a bit of a merge magic hack but um yeah in this case here it it further improved the quality um and yeah if we if you run some user studies on this this outperforms other methods like LC MXL stxl lightning our own uh model SXL turble both in prompt alignment and just in quity and yeah some some samples um from

56:21 this model so we see um it is actually these these are not one step to be fair are two twostep and four-step samples um but it retains the ability to spell uh it's pretty diversal has a lot of styles in it it's yeah U it's really nice uh but it comes with a few limitations one is that the prompt following is not as good anymore uh so sometimes it it fails for like these yeah more more more difficult prompts um so obviously uh one one thing to fix is ex exactly that just improve the distillation methods uh find better algorithms maybe combine it with stuff like um new distillation techniques like moment matching um yeah then there's always the debate of getting rid of the advis serial training um I think right now and in the regime of small models or models with like limited capacity we probably always need it because these are Reg ression based objectives that we have in other distillation techniques uh they

57:31 kind of if your model doesn't have enough capacity you will always end up with something blurry um and the serial training is one way to circumvent that then you get like other trade-offs but it's like one way to get like at least crisp image quality out of such a distilled model and then obviously the other thing is applying that to video generation I think having something like real time uh text image video generation is an extremely attractive uh thought so uh I think I'm out of time I actually wanted to talk a little bit about video Generation Um but I'll skip that actually have another talk at the AI for Content creation Workshop later today so if you want to go there then um and yeah maybe one one thing um which is this general question of uh which was raised the the announcement of well maybe maybe even before but at least this announcement here of TBD 4 uh which can do multimodal

58:34 generation it's a question of what's actually uh the status of diffusion versus aut regressive training um and all regression actually has although like all of this that I was talking about is based on diffusion right or flow matching um aut regression has a few advantages it is like one thing is there's massive research going on uh in optimizing the inference optimizing the training there's even dedicated Hardware beinging built for this kind of stuff seems to be like an advantage for it the training efficiency is much larger because you uh train on all the token transition probabilities at the same time in Auto regression whereas in a diffusion model you only sample a single time step um and then denoy St so the training signal and diffusion models is much sparer and uh that means you just need a train for much longer to achieve the same quality and then they also have different inference Behavior right in

59:31 Auto aggressive model you can um you can cach intermediate activations and only have to recompute them for the new token the fusion models you usually have to uh put the whole sequence through the model again and again for each D noising step but then on the other hand to really have like high quality auto regressive generation you would need some really good tokenization and that's where the current bottle L is is for these methods and they also don't have like this native support for image editing image impending that diffusion models have it's a bit of an open question but I think it's very interesting to um to consider we might see a come back of Auto aggressive models I don't know um maybe a combination of the two um and yeah that's that's basically it um if you have any questions feel free to ask them think we time for say two questions hi um I have couple questions about that you

1:00:37 present y um the first one is compar to PR you have this additional Tex brand keep updating so I was wondering how expensive that is um that INF or ter of computations that using so oh that it's um that's question the other question is about the data that you use I was wondering whether you are using any synthetic data by like larg models and if so have a like replicate that yeah good questions so on the first can you repeat oh yeah sorry uh first question was um on the multimodal D architecture with the two um Transformer streams and on their efficiency um it's a really good question uh so the efficiency is let's say you compare for example training interation speed it is actually quite similar as in the base model um but I would say if you scale to really large architectures you might want to reconsider this Choice um and um because because these dual streams provide kind of an inductive bius and at very large

1:02:01 scales you don't really care about it uh or you don't really want inductive biases in your model so maybe something to reconsider um maybe do a combination of this plus very classic diffusion Transformer um yeah uh the second question synthetic data yeah I didn't mention it but uh for the for the second approach that we had um this latent advis serial diffusion distillation that was actually trained exclusively on synthetic data that was generated before training on um from the SCP teacher model so um that is something that works really well like an advantage of that is that you stay strictly in distribution of your teacher um which makes the training task a bit simpler on the other hand it kind of prevents uh improving upon the teacher because you can't incorporate real data um but there's a bit of this trof but yeah it's it's definitely possible to include these kind of data yeah I follow question is I was

1:03:06 wondering whether you have a recipe to generate such data like how you see okay question is what's the recipe to generate the synthetic data um one answer is to Simply use you have some data set that you train your base model on you can just use these prompts um you can also use if you have data set of prompts that are really popular then maybe it's something that you want and you kind of implicitly get a preference optimization into it um yeah different different methods but I would like the simplest is to just go with your training data problems all right let's thank our speaker again up next we are moving on to Spotlight talks don't leave yet launch is until 1:45 so we have plenty of time thanks

Questions about this episode

What is discussed in "23578 Efficient Large Vision Models"?

In order: 0:00 MIT, Tiny Chat, CPPR; 20:00 Tiny Chat, Llama 3; 30:00 Runway, Sora, Stable Diffusion; 40:00 Stable Diffusion; 50:00 Stable Diffusion.

What is mentioned in this ComputerVisionFoundation Videos episode?

9 people, works and ideas are mentioned, including Tiny Chat, MIT, GPT-4, Llama 3, Stable Diffusion, MLPerf. Each one on this page links to its own page with every episode that mentions it.

How long is "23578 Efficient Large Vision Models"?

The episode runs 1 h 4 min; the transcript is about 9,750 words in 59 timestamped sections.

More from ComputerVisionFoundation Videos

• All analyzed episodes
• People mentioned on ComputerVisionFoundation Videos
• Topics mentioned on ComputerVisionFoundation Videos