23714 Edge AI in Action Practical Approaches to Developing and Deploying Optimized Models

Who speaks in this episode?

• Shan Ahmed Shafi (Speaker / Team member (audio & multimodality))

What is discussed, and when

1. 0:00 – 15:00: GN, Edge AI, Jabra, Panagas 50
2. 15:00 – 30:00: Edge AI
3. 30:00 – 45:00: P20
4. 45:00 – 1:00:00: Panagas 50
5. 1:00:00 – 1:15:00: Qualcomm
6. 1:15:00 – 1:30:00: Apple Inc.
7. 1:30:00 – 1:45:00: Qualcomm
8. 1:45:00 – 2:00:00: Qualcomm, Panagas 50
9. 2:00:00 – 2:15:00: Onyx, DVC, Gaze Correction
10. 2:15:00 – 2:30:00: Gaze Correction, Edge AI
11. 2:30:00 – 2:45:00: Gaze Correction, Artificial Intelligence

Notable quotes from "23714 Edge AI in Action Practical Approaches to Developing and Deploying Optimized Models"

• I'm going I'm working with h gors also focus on multimodality but for cameras then I'm part of the video tack team for DN hi everyone my name is Anu and I used to be a part of the javra team before I joined Adobe recently and this was our.

  about Jabra · at 0:16

• On video collaboration products for meeting rooms and some personal cameras where my job was mostly taking the models from research to production and everything in between…

  about Artificial Intelligence · at 1:02

• Newer recruits of Jabra as a team and I'm lucky enough to work with these guys over here and my expertise is more in the audio domain and we are exploring more into the…

  about Shan Ahmed Shafi · at 1:48

• Why okay then some idea what you are going to see today in terms of learn outcomes initially I'm not sure if everyone has the background regarding to Edge AI but we are going…

  about Edge AI · at 3:27

• The way that you are going to deploy the model to optimize and deploy the model basically it is the same then based on this we have two case studies is going to explain to you…

  about Qualcomm · at 4:54

• As I'm a teach it works no okay sorry for that what I was explain okay we bought a company called pacast and now we have this cameras that is called the Jabra pacast the P20 we also have this one that has a very wide.

  about P20 · at 6:41

• That for different kind of things Delta Airlines uses that for productive maintenance in airline industry we of course use it in Jabra panagas 50 panagas 20 panagas 50 VBS and…

  about Panagas 50 · at 13:08

• Midex Brin Chaka Google htpu Raspberry Pi and stuff like that similarly there are couple of Frameworks that you can use for deploying and optimizing your models so you have tensorflow light micro specifically for.

  about Onyx · at 15:05

Who and what is mentioned in this episode?

People

• Shan Ahmed Shafi — Person · Speaker / Team member (audio & multimodality) · 1 mention · first at 1:39
  Introduces himself as a newer Jabra recruit with audio expertise, aiming to combine audio and video for improved far-end experience—positioning multimodality as a key direction.

Companies & orgs

• Jabra — Company · Business focus · 1 mention · first at 6:45
  Jabra is a key company discussed in the podcast, focusing on video collaboration products and AI models for meeting rooms.
• Qualcomm — Company · Business focus · 4 mentions · first at 1:08:12
  Qualcomm is discussed in the context of its chipset being used in Jabra's cameras for AI model deployment.
• GN — Company · Business focus · 2 mentions · first at 5:31
  GN is mentioned as the parent company of Jabra, providing context to Jabra's operations and products.
  The time you can see in the slides that I have the word GN and sometimes we call the Jabra then do we work at G or do we work at Jabra in fact G is a big company okay it's one…
• Apple Inc. — Company · Referenced expert · 1 mention · first at 1:23:30
  Apple is mentioned as an example of a company using Edge AI in its products.
  Cones and things like that John de uses that for their C and spray technology for weed detection Apple uses that for different kind of things Delta Airlines uses that for…
• Bose — Company · Referenced expert · 2 mentions
  Bose is mentioned as the former employer of a Jabra researcher, providing context to their expertise.
  Research that we did across three and a half beautiful years at Jabra for optimizing our models my experience has been I have a masters in Computer Engineering and I started my…
• Adobe — Company · Referenced expert · 1 mention · first at 0:35
  Adobe is mentioned as the current employer of a former Jabra researcher.
  Then I'm part of the video tack team for DN hi everyone my name is Anu and I used to be a part of the javra team before I joined Adobe recently and this was our research that we did across three and a half beautiful years at Jabra for.
• University of Missouri — Company · Referenced expert · 1 mention · first at 2:30
  University of Missouri is mentioned as the alma mater of a Jabra researcher.
  Colleague here nari he is online on Zoom NY please could you introduce yourself hi everyone my name is nari I an AIML researcher at Jabra previously I did my M's and PhD from…
• University of Missouri Kansas City — Company · Referenced expert · 1 mention · first at 2:30
  University of Missouri Kansas City is mentioned as the alma mater of a Jabra researcher.
  Colleague here nari he is online on Zoom NY please could you introduce yourself hi everyone my name is nari I an AIML researcher at Jabra previously I did my M's and PhD from…

Products & tech

• MedEx — Product · Case study · 4 mentions
  MedEx is discussed as a product similar to Jabra's cameras in terms of AI model deployment.
  Studies is going to explain to you how to use the qualum because some of our cameras use the qualcon chip set and the other one I'm going to explain to you how to work with the…
• Panagas 50 — Product · Case study · 3 mentions · first at 13:13
  Panagas 50 is mentioned as a product utilizing Edge AI for smart meeting room experiences.
• Onyx — Product · Case study · 2 mentions · first at 2:09:09
  Onyx is mentioned as a product related to AI model deployment and optimization.
• P20 — Product · Case study · 1 mention · first at 37:29
  P20 is mentioned as a product line of Jabra, focusing on video collaboration.
• DVC — Product · Case study · 1 mention · first at 2:07:06
  DVC is mentioned in the context of AI model deployment frameworks.
  Midex Brin Chaka Google htpu Raspberry Pi and stuff like that similarly there are couple of Frameworks that you can use for deploying and optimizing your models so you have tensorflow light micro specifically for.

Concepts & topics

• Edge AI — Concept · Core topic · 4 mentions · first at 3:40
  Edge AI is a central theme of the episode, discussing the deployment of AI models on edge devices for privacy and efficiency.
• Artificial Intelligence — Concept · Key argument · 1 mention · first at 2:42:27
  AI is discussed extensively in the context of optimizing models for edge devices and the challenges involved.
• Gaze Correction — Concept · Key argument · 7 mentions · first at 2:04:02
  Gaze Correction is mentioned as a feature in Jabra's products, enhancing video collaboration experiences.
  Or the P what do we would like to prove it provided to the use a different feed Weare then if you want to against correction you can download the feare if you want to rout…

Full transcript with timestamps

0:00 I've been work I've been working with multimodality in general for interfaces working for some years with voice interaction even before Google apps Google application devices and nowadays I'm going I'm working with h gors also focus on multimodality but for cameras then I'm part of the video tack team for for DN hi everyone uh my name is Anu and I used to be a part of the javra team before I joined Adobe recently and this was our uh research that we did across three and a half uh beautiful years at Jabra for optimizing our models my experience has been I have a masters in Computer Engineering and um I started my career with Bose in terms of optimizing neural networks for running on your headphone devices and other kind of audio devices so mostly audio uh research and deep learning for that optimizing the models and then I joined Jabra where I worked on video collaboration products for meeting rooms and some personal uh

1:06 cameras where my job was mostly taking the models from research to production uh and everything in between starting from uh data search generation optimizing the models coming with architectures with hardware and looking at the AI models not just as a some kind of a software but looking at the hardware and making it both hardware and software code design

1:34 sorry for that technical problems hi everyone my name is Shan Ahmed Shafi I did my masters at uh Labis University in Germany and I'm one of the newer recruits of Jabra as a team and I'm lucky enough to work with these guys over here and my expertise is more in the audio domain and we are exploring more into the multimodality as an exp aspect and I'm being I'm going to be working more into how we can add audio and video together into giving a much better experience in the far end and we also have a person in the far end who yeah because he had problems with Visa but we also have one online colleague here uh nari he is online on Zoom NY please could you introduce yourself hi everyone uh my name is nari I an AIML researcher at Jabra previously I did my M's and PhD from University of Missouri Kansas City and during the same time I worked as a research intern at zos working on uh mobile biometric

2:41 applications and in Jabra my work mostly uh started as an Aji model uh developer and right now I'm working mostly on synthetic data sets and uh uh developing synthetic data set pipelines and uh uh testing out different models for different Hardwares okay thank a lot um it's not able to see Zoom okay let me change robot now can you see the slides on Zoom RC I can see the slides yes yes yeah okay don't I don't know why okay then uh some idea what you you are going to see today in terms of learn outcomes uh initially I'm not sure if everyone has uh um the background regarding to Edge AI but we are going to give you one overview that at least you can understand what is the concept of edge AI um then it's basically practical if you have experience in developing models and neuron networks in general let's now have another point of view how to deploy this for small devices then this is the concept we're going to

4:03 introduce for you in the beginning once you have the knowledge about this module uh development we're going to show to you how to deploy the modules of course like at Java products we have uh you are you do not have access to modify your the firware from our products but I brought for you here uh this camera and this camera for example it's uh looks on camera and it has basically the same architecture as our P2 camera uh the only difference here this is open source you can using python you can use a c to get your custom modu and deploy directly to the camera then basically you can also Implement some interaction directly into the camera the pipeline is the same it's because we have a closer fiware but the way that you are going to deploy the model to optimize and deploy the model basically it is it is the same then based on this we have two uh case studies is going to explain to you how

5:06 to use the qualum because some of our cameras use the qualcon chip set and the other one I'm going to explain to you how to work with the MedEx that is quite similar to this camera in fact it could be quite confusing for you because all the time you can see in the slides that I have the word GN and sometimes we call the Jabra then uh do we work at G or do we work at Jabra in fact G is a big company okay it's one of the biggest company from Denmark and an old company I guess it's around 155 years old and on that time they produce the the communication between Denmark and China and as you can see it's a very old company in terms of Technology when we talk about Jabra in fact Jabra it's a small one of the companies that belongs to GN we have different brands at GN and what could be more confusing for you it is that you know Jabra because the headsets were quite famous about audio and what are

6:16 here to talk about video it could be confused because in fact one of your our products it is the past we uh bought the camp the the we bought the company does it work yeah yeah it works no okay I don't know what happened but as I'm a teach it works no okay sorry for that uh what I was explain okay uh we bought a company called pacast and now we have this cameras that is called the Jabra pacast the P20 we also have this one that has a very wide uh field of view and more products regarding to the video then we here are part from the video department and our objectives is that we could develop machine learning to execute directly into the camera then I brought it to you this small video that basically show one overview in terms of the video technology produced by Jabra nowadays okay it's very short then we can start the workshop after this video what up

8:09 now okay then thank you for your attention in this opening let's start by the introduction of AI thank you for uh just a uh quick raise of hands just to see how many people here have ever worked with the edge device it could be like a ardino Raspberry Pi anything or I've heard the term okay nice quite a few people nice how many of you have deployed a model ever on any of these devices it could be like a fun hobby project proof of concept anything well yes okay okay awesome so a lot of you so um in terms of introduction all of you know uh that you have worked with some kind of a small device so what we are going to see is what is AI a lot of us have heard a lot about the Cloud but let's also see that how SGI differs from cloud AI uh what are the components and what are the advantages disadvantages what are the challenges so let's go ahead with that so in very simple terms uh H is bringing your neural networks

9:45 and trying to run them efficiently on your Edge devices it could be anything from your smartw watches to your phones to your earbuds to your laptops to your mobile phones or any other devices and at the core of this is models are running locally having private compute so that none of your data goes out to the cloud or any anywhere else you have enhance privacy and security you have operational reliability so you are not reli uh looking at okay my internet went down my model suddenly doesn't work most of my features are gone so everything is running on device you have Energy Efficiency because you don't need to connect over Wi-Fi all the time it's all local so it can be much more efficient if they understand the hardware and you have of course lower latency so you are not uh worrying about how long the latency is from uh taking the data from your mobile device or from your laptop to the cloud and coming back with the

10:37 data because it's all running locally and Cloud a as everyone of us know it's much more scalable uh so I can deploy the same models across the world in different different zones uh it's much more accessible uh you have cost Effectiveness there because initially you don't need to buy the hardware you are basically subscribing that if you're using a software you're subscribing to some services or some services are being given to uh to you as a free subscription or something like that integration and collaboration is kind of easy because you might be using some kind of an application or some kind of a device so things might come preconfigured or the OS developer or the application developer might integrate those features later on for you so you don't have to worry about where the devices what different kind of devices are it's continuous Improvement because you can always push a new model and you

11:23 always get the latest improvements whereas as compared to Cloud if we go towards the edge you have if talking about just the iOS devices you have uh different kind of iPhones which have uh the Bic processors but in Android side uh the market is much more cluttered you have devices that have only the CPU only a CPU and a GPU some have a specialized dsps for accelerating the models and some have npus but not all devices are the same so basically what do you optimize your model for do you optimize it just for the CPU just for the GPU just for the DSP or all of them depending upon like which devices would you support so in terms of just model training itself if we see uh the biggest models with the Transformers and all they consume a lot of energy and they have a huge carbon footprint but this is only floating Point 32 model training what about inference of course we say that the models are kind of smaller you can

12:15 reduce the bit bit to say in8 or in four you can have them run faster but you you understand this right the training is happening only say I did like 10 20 100 experiments my training is done but then I am deploying the model for inference it's getting scaled for billions of users who are making thousands or millions of API calls almost every hour or every minute so over time this basically scales up as compared to running things on the edge where you can run as uh frequently as possible as energy efficient as possible without with low latency looking at some of the prominent examples in the market where sdii is being used so Tesla uses uh AI for running uh models locally uh for detecting uh where your Lan lines are where your different kind of objects are whether it's traffic lights traffic cones and things like that uh John de uses that for their C and spray technology for weed detection Apple uses

13:08 that uh for different kind of things Delta Airlines uses that for productive maintenance in airline industry uh we of course use it in Jabra panagas 50 panagas 20 panagas uh uh 50 VBS and other products for uh smart meeting room experiences and then uh NASA used that for the perseverance Mars over which is on Mars so uh inherently The Edge devices give you security and privacy so you have data encryption data is not leaving your device all the inferences happening on your uh mobile devices or on your laptops you only get like basic firmware upgrades where there might be uh model upgrades or basic functionality upgrades but you are not relying on anything that you cannot see basically if it's happening just in the cloud and you don't even know what happened there data is anonymized because you are uh just doing all the processing on the device most of the devices now even come with trust zones where you can encrypt your

14:02 data and you can have uh encryption based uh inference using a machine learning models and then you are also safe from regulatory compliances because if you are deploying models in different kind of regions for example in Europe you have to comply with gdpr compliances where you have to you cannot store the data for more than a certain amount of time or you have to give reasons for why are you storing the data so it's always better to bring those models down to the device where you can avoid these compliances so uh just looking briefly at what are the P pillars of EDI what are the main components the main components look like this so you have a EDI device one part is the hardware one part is the AI model and one part is the software framework which is bringing them all of them together in terms of just looking at the landscape of the SGI Hardware you have a wide variety of Hardware which could go

14:52 from all the way from an small aruno uh Nano or aruno microcontroller all the way to like Nvidia Jetson or some some other uh gpus that you can run these models on so here are like some of the examples like R microcontroller Intel midex uh Brin Chaka Google htpu Raspberry Pi and stuff like that similarly there are couple of Frameworks that you can use for deploying and optimizing your models so you have tensorflow light tensorflow light micro specifically for microcontrollers pyo has pyos mobile execute torch uh Onyx runtime uh qualcom has Snappy uh framework Intel has open a framework and Ed impulse is another one where you don't even have to worry about uh the Frameworks they basically take care of things so bringing it all together what does it look like in terms of what are the main steps uh if you have a model that you have developed what are the main steps of deploying it on a device

15:46 that you want to productionize so you did the model uh development it could be model training in py tensorflow Jacks whatever framework you love the best you just take the model weights and the computational graph along with that and then you perform model optimization on top of that so you might want to do model pruning model quantization knowledge distillation or other kind of techniques depending upon what your hard End Hardware supports where you want to deploy your models once you have your optimized model weights and the model graph then you can export it to a intermediate representation if your model requires or if a compiler can directly take these models as it is then you can just pass them once you have uh your intermediate representation model ready you just perform model compilation uh it could be either from the uh vendor from whom you have taken the hardware they have usually the compilers that

16:34 they provide for doing graph compilation and they have some kind of performance improvements for example they might do kernel Fusion where they might combine the bat normalization layer and fold them into the convolution layers or they might do your post trining optimization in terms of uh quantization and things like that at the end what you get is kind of a binary that contains both your model weights and the model graph combined and you might end up with a file that looks something like a PR file or XML or DLC or a blob file depending upon whichever vend you are using once we have this compiled binary the next step is basically to take that binary and integrate that into your end product so say if you have a microcontroller and if you're using say C C++ or any other language there you basically will take that file and basically it becomes like a small model which you can just do the um inference using uh any kind of cc++

17:23 code so you can basically write some Primitives where you can just Define okay this is my input answer this is my output tensor you can pass the input tensor through your bin file or through any of the other compiled binaries and you will get the corresponding output the important thing here is to note that you want to just make sure that whatever performance you were getting uh at the first step you want to retain the performance as close as possible on the fifth step as well because otherwise there is a lot of loss in performance so we'll go through some of the techniques which will help you in keeping that performance while deploying all the models on this device so just quickly looking at what the what are the challenges because we talked a lot about Edge devices and compared them with the cloud devices but it should not be that easy right so let's talk a little bit about the challenges of course uh latency is one

18:12 of the advantages of deploying models on the S device and you also want your model to be efficient they will be always be available and they will always be private but that also comes at a cost for example you uh don't want a self-driving car having too much latency for a model processing because if it's running at say a certain uh kind of speed if the model is running at too much latency then it could hit another car similarly you don't want uh certain kind of devices looking for Wake words to do you want them to do the processing as much as possible on the device you don't want your words and your voice to go on the cloud every time you say a keyword but in this the landscape looks something like this where we have a lot of different kind of Hardwares we have all the way from CPUs gpus dsps and very specific deep learning accelerators so in this there are two kind of things that happen as you go from the CPU to

19:04 the Deep learning accelerator your flexibility keeps on decreasing while your efficiency keeps on increasing how if you want to deploy a simple say Transformer uh model if you want to deploy it on a CPU or a GPU you can do it right now like the day Transformers came out in 2017 you had the code you could easily compile that you can train it you can do the inference on a CPU and a GPU but to deploy the same model with optimization on on a deep learning accelerator and to get the same inference time and the latency and the memory bandwidth optimization that you need it took some amount of time why because they do not have that kind of flexibility and it's all in the hardware it's a hardware software code design problem it's not only software it's not only Hardware so you need to support certain kind of layers like in terms of if it's a int for quantization the hardware doesn't support it the hardware

19:52 needs to support it before they can even run a Transformer layer there similarly in terms of efficiency the Deep learning accelerators are very efficient because they have uh CNN accelerators or Matrix multipliers that are really efficient to do these processing really fast at a very low power constraints but in the terms of a CPU if you run the same model for inference or training on your uh say a laptop CPU it will definitely consume more amount of power so uh how does the model inference uh look like we talked about uh the five steps about okay we have a model we start from there we compile the binary and we can deploy it so how does it look like in practice say I want to perform say image classification I have an input image ncsw format uh number of uh channels the my bat size height and width so basically I have a compiled model binary which contains my model compiled graph and the model weights

20:46 which have been optimized now in terms of the memory bandwidth what does this mean uh when I'm doing an inference on the hardware the memory bandwidth comprises of a couple of things it's my input image or the input tensor that I'm passing it's my constant model weights that have been compiled into the binary it's my output and the intermediate activations and depending upon how much bigger or smaller your input is your activation size will keep on changing so these things comprise of your total memory bandwidth but this is only a small part right because if you are making a system if you're making a camera system it's not only a model it has audio it has video it has other things and audio and video as you know have much uh tighter latency constraints because you don't want the audio to be coming after the video has already arrived or you don't want the video to come after the audio has arrived right

21:38 so you want all the things to uh work and you want to give the higher priority to audio and video but at the same time you want to run the model and if it's a unified memory architecture where all video audio and the ml are using the same memory so that means you are constrained by the total amount of memory that you have available so if video and audio are already already using a memory which is higher priority then in models you only have a certain amount of memory bandwidth that you can use so based upon this you want to constraint and understand that what your Hardware offers and what are the memory constraints and what kind of memory does your model consume in total runtime so that will be your total runtime memory your model activations model weights inputs and outputs plus the rest of the things that's your runtime memory bandwidth so looking at all of these things just to go a little bit more into

22:24 the depth and understanding that what takes uh it to understand how to deploy models on the devices there is a model so that's called a roofline model let's look at that so what the roofline model does it's basically a plot between the operational intensity of your Hardware versus the performance that you can get out of it so basically number of operations that you have to perform per bite versus the number of operations your Hardware can perform per second and the slope here defines the memory bandwidth that you have so when you're deploying a model on a hardware device you basically want to be as right as possible of the infl Point why say I I can assume like I have a processing element which is capable of processing a say a single Mac operation multiply accumulate operation now let say I have uh 10 processing elements but my memory bandwidth is low so that means if I cannot get the data to my processing

23:17 elements on time my processing elements are sitting idle so that means I am bandwidth limited I cannot provide the data as fast as possible to my processing elements they are sitting idle so that's of no use to me whereas if I go on the other extreme all my processing elements are uh fully utilized but my data pipeline now has to stall a little bit because uh my processing elements are taking a little bit more amount of time so what this tells us is that it we need to focus on both the memory bandwidth as well as fully utilizing our processing elements so ideally we would like our processing elements to be used 100% at the utilization but that's theoretical in Practical cases even if you get 60 to 70% of utilization that's like really good scenario so in terms of operational intensity and uh the performance of the model on a certain Hardware you want to be as right as possible or at least at

24:08 the inflection point and that's where you can say you are fully utilizing your Hardware in terms of processing elements and the memory bandwidth and you're optimizing a models for the certain Hardware so we talked about the roofline model which gives us the graphical representation uh to illustrate architecture performance across the different levels of the operational intensity that we can get for certain model on the hardware any questions so far oh thank

24:45 you yeah my father usually say that when we have a lecture and no one asks question to possibilities you understood everything or you didn't understand anything I expect it's the first one okay then this is the first part as now we have an overview about what is the core of the idea of edge AI Nars is going to introduce you how to develop models uh think about the the the concepts of edge AI okay I'm so sorry n supposed to be here today with us but regarding to his Visa he couldn't attend uh he is in indiia now if he has some kind of technical problems I will recorded a video One offline video then I can can present the video but I expected everything's going to uh work today okay narcy can can you hear me yes I can hear and let's change now I'm going to stop sharing here if you can share your screen with us um yes now everyone can see your video yes thank you a lot then modu development

26:01 for AI hello everyone so for my part of the presentation I will be talking about uh mostly like our what was our thought process when we are developing a segmentation model during 2021 to 22 for our web cameras especially for P20 uh personal web camera so in this uh presentation as an overview first we'll talk about what was the hardware we used and then what were the operating specifications we consider for our segmentation model and uh what what are the limitations and problems we face during the model designing Then followed by data set and training procedure and then we show some video results uh so first first of all the hardware we used is mirex uh Vision processing unit and what we see here is a Blog diagram of the uh the the AI side of the system of the chip which uses the vector Crossing uh part so here uh I don't know if you guys can see my uh cursor but the bottom uh left part

27:12 you guys can see something called shave processors those are the programmable Vector processing units and it has 16 of them so basically they're designed in such a way that you can run multiple deep learning models on uh uh on a combination of these shaves and another part of this um uh s so is the cmx memory which is called uh uh connected Matrix memory which uh which is total of 2.5 megabytes but it is divided into 15 slices of 128 kilobytes and basically the way we can use the hardware is we can combine a set of shaves processors and uh slices of memory to uh uh perform different uh operations for example we can use four of these shaves and four of the slices to create a head and body detector de planning model and we can use another set of shapes and slices to do any other de planning model or a vector Processing Unit uh together in parallel another part of the uh madex chip is the neural compute engine it has

28:16 a set of uh deep learning layers which are optimized to run on the mirror deex uh secondly the operating specifications for our model is that uh when we start started this project we already know that there are some pipelines already existing on the uh camera few of them are the video and a audio processing pipelines they're already using a set of shaves and slices of memory on the camera and also along with that we have other deep learning model such as head and body detector which is running uh parall in the hardware because of that uh when we decided to add a body segmentation model we were left with only two shaves and two slices of memory so going back uh mirex uh advertis their Hardware can run at maximum or peak of 400 million operations per second that means when we ended up with two shapes and two slices we have very less amount of compute and very less less amount of memory to work with because of that we

29:17 decided not to go into like a really complex segmentation model for person uh instead we just decided to go with a simple uh and uh specialized segmentation model in our case we decided to only work uh above the body uh uh the Torso and the head and also we decided to use the region of Interest we can get from the head and body detector which is already running inside the camera and at the end the whole segmentation pipeline should run in real time which means uh when we get a frame uh get a re region of Interest crop from it and perform some pre-processing then run the segmentation model and then perform any post processing like background removal or background blurring everything together should run uh 30 frames per second or within 33 milliseconds that means at the end our segmentation model should be really really fast uh so we started working on model designing and we fa some of the limitations with

30:17 the hardware we have and one of them is that uh mirex during 2021 we still have working with 16bit floating point and then that is the only one which is available in the hardware uh another one is the neural compute engine uh the thing with neural comput engine it has a really optimized uh layers however there are only limited of them for example when we look at the normalization layers uh uh only batch normalization was uh optimized if we want to use any other normalization techniques like instant normalization or layer Norm they were not optimized so they will run directly on the uh shave process processors and they execute really slow so we need to make sure that we are using only optimized uh layers so that we we get the best performance out of it and uh next thing is that the madex doesn't support sparse Matrix uh uh operations because of that we cannot use any sparse pruning techniques and only we need to

31:20 depend on structure pruning techniqu such as uh removal of layers or like uh channels in the uh convolution layers which was really difficult to train so we uh with the along these limitations there is one more thing we fa during uh testing out different models which is regarding the memory because we left with two slices of memory which is like two 128 kilobytes of memory which will be used by the model weights and also the features uh feature vectors there is lot of uh uh copy and fetch operations happening between the cmx memory and the ram because of this we Face two problems one of them is the issue with the spatial resolution of the input let's say we have an input of 512 by 512 with three input channels and we want to perform a convolution operation on it we can see that as we increase the number of convolution of Channel outputs the execution time of high resolution input is going up and it's going up really uh

32:24 fast what we found out is that uh even though there are less number of channels the way the mirex operates with the memory is it takes a lot of time to uh fetch uh data from the ram into the cmx memory because of this when we have larger spatial resolution the execution time is higher uh so when we have smaller resolution for example here we are showing a 128 by 128 uh spatial resolution with 48 channels we can see that it's almost performing 10 times uh faster than uh 512 input and the solution we found out is that we can use at the starting of the input of the convolution uh our de planning model itself to use like large convolution uh with large uh stride to reduce a spatial resolution as fast as possible and then perform uh any complex operations on it and when it comes to Output we can use uh dep to space uh operations such as pixell shuffling to get back the spatial resolution another issue we found is

33:27 with longer skip Connections in uh we had a problem here with the microphone give me me a second yes to see if it returns yes yes now we have the we have the microphone back if you can yes restart uh is it the previous slide no only the only the next only yes I guess okay okay yeah we lost like your last explanation about this this slide okay so um especially for this issue regarding the skip connections what we found out is that uh in usually in segmentation model we use an encoder decoder architectures for semantic segmentations which is like a standard and unit is the initial original segmentation architecture here what we found out is that during our experiments especially when there are longer skip connections from input of the encoder to the output of the encoder uh the copy and fetch operations for Skip and concatenation are taking a lot of time here we are showing with like an exaggerated example because we never use

34:38 this big of a unit in uh a applications but we wanted to show the ex like operation times for skip skip operation and concatenation operation and as we can see that as the feature size increase the execution of those operations also take time so the solution we came up with this is that whenever we have a long skip connection from input to Output we try to use a simple uh one by one convolutions to reduce the feature size so the the storage uh the the execu like the memory size would be smaller so copy and uh fetch operations will be also faster so overall in our model designing uh whenever we want to make a our model we want to make sure that the layers are uh available in neural Computing gen so that they are optimized and run faster and then to reduce the issue with the memory for spatial resolution we use large kernels and large strip convolution layers and uh pixel shuffling at the output uh for that and

35:42 for reducing the memor footprint we used smaller feature sizes and also Whenever there are longer skip connections we used one by one convolutions to reduce the feature size and through our research we tried different convolution blogs like we tried resonate uh block and mobile net blocks and dense net blocks and during this time during 2021 a paper published uh on byet for semantic segmentation where the proposed a shortterm dense concatenate module and we used a modified version of that in our model at the end this is the model we uh stick with on the left side we see our segmentation model which has encoder and decoder block uh unlike unit architecture in decoder we we did not extract features at each layer of the decoder instead we found out that simply upscaling the encoded features and combining them and performing a small amount of uh uh convolutions on them we were able to extract uh good enough

36:45 results and secondly we can see that what we talked about using a large uh convol strided convolution layer at the input of the encoder and pixel shuffling at the output to reduce the spatial foot print and then we use the one by one convolutions to squeeze the features whenever we perform the skip connections at the end we were able to end up with the model size of only 300,000 parameters and were able to run the whole model on our two slices and two shaves at 18 milliseconds so we have the model we need the data set uh first we started with collecting as much as data we can from the real and we did an in-house data collection for a single person with a personal P20 cameras with different poses lighting conditions uh any occasions like wearing a face mask or like a long beard headset and everything and this work was mainly done with my ex- colleague his name is Harry zilani they he worked with the t uh with the

37:46 company to collect the data and pro did some proc initial processing on it uh but no matter what it's really difficult to collect a large amount of real data and also so uh annotate it so we decided to also add our own synthetic data set to the pipeline so that we can increase the overall training set this work was done by me and we used a bunch of uh uh 3D models we bought and comb and simulated everything in the blender to generate uh different poses for the person uh during our training one thing we found an issue is that if you just simply combine all the real and synthetic data sets together and just train we have really lot of issues with flickering or like sometimes it doesn't even segment the person so we started playing with distribution of real and synthetic data for each batch at the end we found out that using like 30% real data and 70% synthetic data in a batch and then performing um a bunch of

38:48 augmentations on it and for training it actually gave us uh pretty good results and for training initially we used uh s self-supervised learning technique dyo just to have some pre-train weights and then we used uh then we trained the model on our training set for almost 10,000 uh uh batches and what we see here is an ideal condition of how the model uh operates like it gets a region of Interest crop and it's above the waste and only the person is uh in that and right now the model runs on Hardware at 18 milliseconds and one more example we're going to show is like for when we use the full uh frame from the camera instead of using just the region of Interest here you can see that as we trained only on the single person training data we have some issues like for example if a person comes into the frame you can see that it's it has difficulty uh blurring out in immediately um and sometimes there are

39:54 like trees and objects like that they come into frame so yes uh this is our model uh which we vote uh initially around 2021 and 22 to develop it for our Hardware uh and this is uh I hope I provided some examples of how we thought thought through our experiments like when we uh prepared this model and if you guys have any questions please let me

40:48 know sure now see we have some questions yes okay question was it okay so uh the question is uh for the segmentation was it the dino or the unit model and if Dino was used or unit was used how was the training like what was being trained and what happened where so so for Dino we just replaced U I think Dino was initially using vit like Dina uses a single model for as a teacher and a student so in our case we did the same thing we just used our own model as a teacher and student for training and when we uh when we did an experiment with image classification uh just on the self-supervised model at that time the performance went from like 13% accuracy at top five to 60% uh accuracy at top five uh uh like detection

42:01 yes any things that we should check any exhaustive list for the processor to check out before uh like optimizing the model for running how you did that uh I can go into what we did uh we started using mass as an initial uh initial experiment where we tried to see if existing NASA architectures can work and we had an difficulty coming up with the model using that so we just said uh screw it just brute force and create an Excel sheet of all the layers and all the uh activation functions and anything we can and just try to Loop it through our Hardware to see if we can find any uh information and through that what we found is that the memory issues especially and that's where uh that's how we found out oh this is we know that there is some memory issues on this architect uh Hardware so what kind of models we can look into that's how we try to limit our search in terms of what kind of uh experiments we can

43:20 do yeah so the question is based upon the hardware how do you decide like based on the problem how do you decide which Hardware to use so for our case I think Hardware already decided yeah we do not have an option for that we we we came into the we already have the hardware and they asked us like what we can do it with that yeah if we had option we would have uh oh yeah a different approach maybe so this camera was already in production and and then uh we came in and we started doing the optimizations sometimes we have a fight with the engineer team these guys optimize the models and we sent to the engineer team we have different shades Shades available and Shad is us it and it's always a fight okay you give me one more shav for device no because we can Reser the shap for another application the the is predefined this is the we much toward around the idea is if we start implementing many models that it cannot

44:37 be supported because the limit number of Shades available both in the P2 or in the ble or the P what do we would like to prove it provided to the use a different feed Weare then if you want to against correction you can download the feare if you want to rout segmentation of the firmware but can you see that in terms of usability and use experience this could be quite difficult because it's not all the users could be able to download the new firmware and deploy into the cam and we can also start have sever problems with the customers then the hard is this one and we should work around the limitations provided by the and everything that is available is still devop resources yeah so so a fun story around this is basically if you see what n presented in terms of shaves and slices so if if the processor has six shaves and slices so what nari got was only I think one shave and slice to work with

45:39 so you can think of it as like maybe you are getting just one or two CPU threads and a GPU thread something like that to work with for running this but you still want to run it as efficiently as possible so again it's a trade up between accuracy and latency and accuracy and memory bandwidth just as n mentioned thanks n thank you one oh okay we have one more question yes try out different architectures do you train the whole model and then to figure out the accuracy Andy and another Bally training for each VAR then okay so n the question is like uh for figuring out like the latency memory band and like which is the best model do we train uh the whole model for all the different variants that we are trying and then figure out the latency memory bandwidth for that or do we have a different approach uh we uh we know that we already have a limitation with the hardware and the execution time so

46:46 whenever what we do is that we do a literature review of all the models we find we try to create like a abstract model of what we can run on the hardware and if we can see that we can run it around like around 20 milliseconds or faster then we start training the model on that and see if we can get a good uh results out of it uh that that's usually how how we uh work on it because we already know what was our hardware and what the limit we have on our Hardware so we just take the models run it and if it doesn't perform uh like doesn't run faster we just move on with the architecture okay we have couple more questions yes like mentioned as well and you knowem transation that the edge device is limited compute Li yes cannot put everything on it yeah did you guys experiment with the hybrid kind of an architecture where you were doing some processing preprocessing on the S device and then you off the more expensive

47:45 compute task CL okay so we did not because uh I can answer this I guess uh or unless n want to take this no you know better on okay so the reason being at the core of all our products is user privacy we don't want to send any data we don't collect any data even if you see those devices this device any other device you can turn off the Wi-Fi and the internet you can run them yourself all the computer runs on the device nothing leaves the or goes to the cloud it's being powered through that USB or for the bigger devices they are being powered through uh like the plug-in socket that's it so that's one of the reasons we don't do any kind of data collection or anything and we did not want it to go hybrid of course that's one of the approach uh that we could have taken and we could have run into like multiple bigger models that's one of the reason second reason reason is we are latency sensitive because uh for our use case if

48:38 we are running in a personal use case or a meeting use case the people are moving all the time or people are talking all the time we want to look at those things and response respond as fast as possible so given the latency addition latency that might be there for hitting the cloud and coming back that was not simply possible so these are like couple of reason for that may I add just one information privacy is so important for us in Jabra that if you imagine we have probably the best source of collecting data that is the camera we somehow could collect the data from you guys and they have the best data set in terms of the world because most of our customers are companies we also have those big metals and we could collect the data and work with not only image interaction in general I could see human perception I could have body movements hand gestures for Elizabet but it's so important that

49:37 we never touch on the data from our customers and we don't have access to the video stream that comes from because of this we always think about how to process this in the camera and even the things of background because most of you probably using Zoom the things for for and hybrid and probably you enable the background segmentation but the video that came from your your web and going to the computer see third part application could have access to the backround when we mat that when going to execute the background segmentation directly to the camera the video streen provided to the computer it's without the background information then privacy is the key our team and we never touch the video screen from our customers and we always try to protect you in terms of video ands of privacy we had one more couple more questions yes please any don't worry we have time yes question

51:01 oh like you mean like open source code for the models so these are all these are so the question is like do we have any open source uh code or anything for these models uh so these are all proprietary models with the data set and like uh the other gentleman also mentioned one of the workshops it's not about the models nowadays it's becoming more about the data set and that's where uh the core is like we have our own pipeline of synthetic data set and all those things in house so yeah the models are not available it's everything is available just on the device for

51:38 inference uh okay NY the question is size of the data set for dyo pre trining uh I think it's just a scratch of 20,000 images we randomly collected like green uh Greenery there is no human in those data uh it's just like objects and uh like trees mountains and all this stuff just to create some Randomness in the instead of having some random weights just creating some weights in the model yeah so yes so Doo is basically what we're doing is instead of training the model from scratch we are basically using Doo to pre-train the model WID so that it knows something about how to extract features and these kind of things after that our data set uh that we have collected is a combination of real and synthetic data set that's where the 10,000 bad size number comes from that's what it after that yeah uh yes please so the camera itself has some kind of models running inherently um did you guys try to experiment with like an

52:39 N2 model which you can optimize you could you optimize those moning on the device for head detection and body detection yes so we have optimized but could you expand on what end to end means rather than providing provided by who we are providing the rois I thought the camera itself yeah so it's our camera in the camera how the stream works is we get the entire full field of view whether it's 180° in these kind of cameras or whether it's 90° or configurable in those kind of cameras based on that we run our person detector and other kind of models from there we get a region of interest that where the person is and then based upon that you can either run segmentation on region of Interest or the full field of so it's end to end pipeline basically that different hardw yes so for panagas 20 and this camera we had a mered x Intel mer x uh for the panagas 50 VBS we have a mixture of Intel mer X and A qualcom S so that

53:43 we'll talk about in the next yes please sure uh nari can you expand a little bit more on the synthetic data set like how big it is and uh what were the challenges and what challenges you are facing even now with that uh synthetic data set I think we limited it to 50,000 images or something at that time challenges is basically I have no clue about three modeling uh that's the biggest challenge I don't know why I took up that initially when I did uh cuz I just saw python in blender I'm like oh cool there is python I can try something and that's how it started and I think at this point I was able to develop a pipeline for blender and I was able to generate a lot of data but the issue we ran into with that pipeline is that uh whenever someone else try to install it in their Hardware it doesn't work properly uh example is ano we tried to set up on this uh system and it did not work but it's only working on one system

54:56 uh initially we started like that but now like this year we mve away from blender to Omniverse uh we don't have anything yet to share on that but we have good start uh in in terms of what we can do in Omniverse yeah I also have a couple of stories around that so some of the use cases that we was we were seeing was some of the models would detect detection uh like Mis detections or uh should not would not perform well on segmentations for example if I have my hands up or hands behind the head so in those cases it was kind of a challenge I remember Nasi was working with that and we were trying to get the hands in the right position but sometime it'll go like this go like this or go just like in a different direction altoe uh not usable so there and another thing is like most of the models are like frame based like generally models are trained on just individual images but since we are working on video based there is a

55:50 temporal aspect as well which actually helps in stabilizing the whether it's like object detection in terms of bounding boxes or even segmentation maps and things like that so for that you need to now generate video frames and not just the image frames because image frames alone won't work so there's a Time dependence and time continuity that's required there so there have been like a lot of challenges that we have been going through in terms of synthetic data set generation but now as nari mentioned we are moving a little bit away from that so hopefully that will help let's see yes please so uh video frames basically mean like uh if if I am training a model on Independent images for example let's take a simple example of say image classification if I want to train a model for image classification on image net I'll take the image net data set those images are separate individual images right they don't have for example

56:41 for a person detection or person classification they don't have the same person across all the frames but if I record a video of myself using one of the cameras I am consistently in the frame so for from each frame to frame if I'm moving around a little bit there is that tempor oral aspect that's coming in terms of detecting me as compared to just taking my individual frames when I'm standing here there or somewhere else so you are missing that time temporal aspect like how did I go from here to here it will miss that if I'm just capturing images but if I'm capturing videos so then it knows that these are the steps that he took to go from here to here so in between I can then use some layers like com Gru or lstm or those kind of layers that can capture that temporal aspect along with the normal convolutions or normal Transformer layers and that actually helps to smooth the bounding boxes or

57:27 the segmentation maps in the background because over time it knows that if the person was here in one second the person did not disappear the very next second the person is still there so it helps to stabilize those

57:45 things so nari are we using any temporal layers in our model or in postprocessing uh like in the final model we developed we used the convolutional gru uh at the end of the encar block uh that's the only temporal like layer we were able to add because temporal layer uh adds another complexity like adds more uh parameters and more um operations per unit so we just decided not to use a lot of them and like a one simple convolutional Gru that's what we used yes implenting the temporal component into the network it addition on the hardware when you running or you have to do something speci to make that so nari the question is did we had to do any additional thing or challenges to add the temporal thing and then run it on the hardware efficiently uh mostly the way we needed to run was um the hardware doesn't support memory uh like it doesn't save any memory so we actually needed to just create like a

59:01 separate variable where we copy the output of the gr unit every time when we run a new frame other than that I don't think we add did any U additional stuff yeah but just using the lstm itself does add a little bit more to the memory bandwidth because now until you reach the output you still have to maintain all the intermediate step for all the time steps so that's additional memory that you have to keep keep in mind while adding that that's why only like one layer in there and that to like instead of normal LM you can use like a convolutional gr or something so that you can use best of conation little bit of Gru a mix of that and still get the temporal stability yes

59:58 so what was the challenge of using Nas basically for like was it Hardware in the loop or taking too much time or not having good architectures what happened I think at the time the hardware uh architectures we were trying out initially like before we did The Brute Force experiment initially we tried all the known architectures which are which are like like mobile Nas or like mobile net architectures we tried using those but still at the end we were able to tr like run the model on real time uh in the camera uh on two shaves and two slices that's the main issue uh but I think when thinking back now once we did The Brute Force we know what kind of layers work and what kind of uh architecture Works maybe if we go back again to Nas and try it out we might find better model probably any other yes please

1:01:02 make it okay so nari uh one second let me just share the slides NY slides n can you share again your can you share the slides and go back to the training pipeline the training pip pleas uh these are the only two slides yeah

1:01:38 y anything else yes so mentioned that having yeah good question so the question is how did you train the model on both static images and the video data to add the temporal aspect in there oh so when we uh add a temporal component to our model we have two uh two sets of training uh in the same uh iteration first we get a set of batch of uh images for uh which are like single Frame data and we just train the model on that uh for few box and then we add slowly add more video data into our training uh that's how we uh did the training but at the end we have both single frame and video data but uh the weightage is added more onto the video data so that the the temporal unit the convolutional gru can uh learn on that yes so thatan for the U the static images did you freeze the your so for static images did you freeze the gru while training with static images no I don't think I freeze we just FL it uh

1:03:04 as it is Oh you mean like yes like during the first EPO when we don't have the temporal data added to the training yes we did not train the gru unit but once we started adding the temporal data uh we started training the gr unit even even even like at the end when we add the video data we still have the Steel images right at that time also the gr unit uh weights are trained and there's like another second approach which is like you can train your entire model with the images first and then if you have video frames then you can specifically train the gru itself the layers for that and you can keep it frozen for that time so that also okay so it's time for um just a second yeah then then I uh thank a lot yes thank a lot for your presentation let me see here what do we have it's break okay then it's time for a break um in fact thank you a lot for the questions this is the way that I I like to have lectures and interact with

1:04:32 my students then with you it was an amazing section uh let's see I'm going to give you a bit more than half an hour we will be start again quarter to 4 okay then we can continue with the modu deployment thank you a lot for the moment see you you half an hour see yeah I'm going to stop recording I was thinking we could have done it till 3:30 and then start from there and get more time there yeah because of the you before you said half an hour okay let's let's restart can you hear me yeah then now uh it's going to be in two parts about the modu deployment for AI then we will start with narsi explain how to deploy uh models for qual then the second part I will explain to you my experience in terms of deploying modos for uh media decks okay then let's start sorry I said Nar that's I wouldn't mind that's okay thanks for okay uh I hope everyone is still good and with us okay uh so model deployment so far we have looked at what

1:06:10 nari talked about in terms of segmentation model what all challenges he had to face how he developed the model but there are couple more steps that you can look at from a general sense so I would like to um take you through a little bit of the model journey of how things work in terms of a normal General uh pipeline all the way from developing your models doing uh neural architecture search and things like that from a very high level and what you can expect in terms of the performance improvements and then going deeper into step by step what other things you can apply on top of that and then finally looking into two case studies how we deployed models efficiently on our devices so here's the agenda we'll look at a model compression techniques uh we'll try to Define some key metrics so that we can look at each of the compression techniques and compare them in terms of performance and other kind

1:07:01 of things that how we are doing with them like what kind of things do these things offer otherwise what's the use of just talking about just a compression if we don't have any metrics to compare them on then uh we'll talk about the case studies and a summary so uh in terms of model deployment for SGI uh model deployment is a critical phase because you have now uh trained your model using pyto tensor flow any kind of framework work now you want to optimize your model you want to deploy a model but it's not that easy on the edge as compared to uh maybe on a CPU or on a laptop uh on the GPU that's where you need to compress your model because these devices as Nasi mentioned don't have too much amount of memory and compute and other kind of resources to run them efficiently if I want to run a big model or train a big model uh or do the inference of say a Transformer model I can easily push it

1:07:55 onto a CPU or a GPU it will still sweat a little bit on the CPU on the GPU it should be pretty much okay given that I'm running with like say Ino quantization or maybe with half Precision at floating Point 16 or something like that should be totally fine but when it comes to an S device like the intermex that we showed or Qualcomm device or even your phones that you have in your pockets right now it becomes really challenging because these devices are running your entire operating system let's talk from a phone perspective say for example it's running an entire operating system there are certain uh processes that are happening in the background uh there is audio video co-processing systems and other kind of things that are working and then model and the other neural Hardware becomes a very small part of it and that to has to run multiple models for giving you multiple different kind of experiences it could be running in

1:08:43 parallel it could be running sequentially depending upon how the pipeline has been implemented so model compression becomes really really necessary where you can deploy on a CPU or a GPO model which could be like say a couple of megabytes to to maybe like gigabytes on his devices you might be limited to even like say kilobytes or to one megabyte or a couple of megabytes not more than that so that's where model compression comes in model compression is basically in brief terms the Art and Science of making an AI model that you have trained to run efficiently on a small device Edge device but without losing the performance so it's not like you have made a big model and you want to compress it but then you say that it's not accurate enough for example if I had a good object detection model it works pretty well when I trained it at fp32 but now I want to deploy it at fp16 or inate it doesn't Works equally well but

1:09:37 I am getting really great latency that model is of not much use to me because if it has ton of false positive false negatives that model I cannot deploy in production so this is where we'll try to uh figure out that what are some of the key metrics on which we can compare all of the optimization techniques and we'll talk about some of them so here are some of the key metrics that we looked from a both hardware and a software point of view because it's a hardware software code design problem we'll talk about size reduction of course size reduction is important because the model as I told you before in terms of a single inference model weights model activations inputs and outputs that com comprises of the runtime memory bandwidth that you need so size reduction is important because your model size goes down so you use less amount of memory model accuracy is equally important you want a good

1:10:27 accurate model you don't want a model that's run faster but doesn't is of no use at all uh model throughput uh the amount of uh processes that you can do using your processing elements basically the amount of processing that you can do or number of inferences you can do in an inference point of view uh that you can do per second so that becomes kind of your throughput power and energy consumption because if your model or your Hardware is kind of like say a bigger GPU kind of a model there you know you have to have bigger heat sinks because the more amount of power it consumes the more number of tops it has your uh thermal design for power actually goes up so it will dissipate more amount of feet in these small kind of cameras if they become too hot they'll just stop working memory bandwidth as we talked before and the the latency because all the things need to run in a certain time frame for

1:11:19 video and audio all the things need to run within uh the 33 millisecond latency limit they cannot be go beyond that and if a model is a part of the video processing or the audio processing pipeline it cannot delay the entire end to end latency just because you were running a model and you were trying to give a new experience to the customer the customer doesn't care so uh we'll look at couple of primary techniques uh that we were able to uh use uh through our time span uh for different kind of models spanning from uh segmentation models to object detection model to pose estimation to gaze estimation and other kind of things in a general sense what are the techniques what the landscape looks like and what kind of performance you can expect out of them improvements that you can get out of them on the key metrics so the very first one is neural architecture search you are beginning from uh maybe you already have a model

1:12:11 you are trying to design a model so first thing if you understand your Hardware well you know what kind of layers it supports for example if I have a mirex it has a CNN accelerator if I know it cannot support Transformer layers I'll try to avoid them I'll try to to figure out what are the Alternatives maybe it accelerates CNN better maybe it consumes a lot of memory like how nari mentioned for the skip connections it's consuming a lot of memory try to reduce my channel so these all the things that I'm talking about these all come as part of your neural architecture design before uh even making a model you can just figure out using these latency constraints memory constraints or whatever uh key things that you're looking for you can look at them and you can figure out okay these are my minimum and maximum constraints that I can play Within because this is what my Hardware supports I cannot go

1:12:56 beyond that because it's of no use you'll be just reading the efforts so given that you have say a search space where you know okay convolution is a good layer ghost convolution is another layer that my Hardware can accelerate I can have uh maybe uh depthwise convolution that are faster maybe I can have uh partial convolutions different kind of things so these are my layer levels in terms of kernels okay 7 by seven works pretty well uh 5x5 is good 3x3 is good maybe rest all don't work so I'll have this as my Sur space uh in terms of uh maybe the depths I can go okay 32 is good uh like in terms of the width of my layers 32 is good 64 is good 128 maybe 512 but depending upon how far or close I want to be I don't want too many features if it's causing me too much trouble for memory coping uh in case of unit for example then I'll try to limit it to say 64 so once you have a search space you

1:13:48 can use like a controller it could be like a reinforcement learning algorithm it could be genetic algorithm that you can go through and then it'll reiterate through all of these things and it'll give you a architecture initial architecture now uh there are two ways one is that you can look at all the layers and you can look at the latency of layers individually and you can figure out okay these layers are at this run time a combined model might give me certain amount of runtime plus and minus some amount of time so this should be a good enough architecture or second is you can have a model that you get out of that and then you can have Hardware in the loop and you can get your latency estimates for the entire end to end model then uh maybe you are not happy with the performance of the model in terms of accuracy in terms of the latency in terms of memory bandwidth so here we are working with two uh

1:14:32 different kind of trade offs accuracy versus memory bandwidth accuracy versus latency so maybe I'm not too happy maybe I can optimize a bit more maybe I found a new paper that came up with some new kind of techniques new kind of layers maybe I can play some kind of Kernel tricks here so I'll try to go back and reiterate maybe change my SE space and come up with more architectures so I can reiterate over this process once I have a final optimal architecture I can then go ahead and I can do some kind of more optimization on top of that so in a very broad sense these are some of the key on these key metrics these are some of the uh values that you can expect in terms of size reductions accuracy power and energy consumption memory bandwidth of course these things can change depending upon what your Hardware supports and what kind of layers you choose but in a very very broad sense General sense

1:15:19 these are some of the things that you can expect another technique is so far we have been developing the model say for a image classification model what we usually do we have entire backbone and we wait for the final layer to give us an output but what if the intermediate layer has already learned some information that might be useful to make a classification or give me a result as it is at that point itself because say for example my uh background for an image uh for a person detector is not complex enough it's pretty simple so my model is able to give me the output say on the third exit itself I don't need to run the entire model because when I'm running the entire model say if I'm giving an input image I'm getting the final output that means I'm calculating activations for each of the layers until the final output what if I get output only at exit layer one that what means I was only able to calculate the

1:16:13 activations for first convolution second convolution and the exit layer and that's it I don't even need to process rest of them depending upon what's your accuracy requirements you can provide accuracy thresholds for each of the layer and you can get get a good amount of result right there so this is another kind of optimization that you can do at the architecture level itself while you designing your networks so in terms of early exits you can have some of these uh things where you have reduced latency depending upon which kind of exit has been taken uh you have lower energy consumption as effect of that and then you have adaptive computation depending upon how complex or easy or um scenario is on which you are doing your processing or input is so just to wake everyone up after lunch uh let's see a small demo video of uh how this thing works in action

1:17:27 you okay so what just happened here we saw this architecture this is the exact same architecture that's doing an inference it's basically very simple classifier saying whether a human is present in a scene or not so what you saw in the video is actually at some point of time if the background was simple enough it was taking exit one or exit two rather than going through all the model like through the final exit so what this means is my model performance was good my confidence was 98 99% which is way above my threshold of % that I set for each of the layers but at the same time I reduced my amount of computation and memory bandwidth that was required for me so this is a very simple use case but this is basically a similar concept that you can use across uh segmentation object detection and other kind of things in your backbones now uh looking at early exits here is another example on the videos

1:18:44 just to understand a little bit deeper on uh what happened and why it made a certain decision so we just visualized the uh activation Maps using grad cam so for all of these images you can see the confidence of a human presence is 92% with exit 2 93% with exit 2 for last one it took exit four with 95.9 person reason being in Exit 4 you cannot see the human face maybe the model actually tried to look at all the features of the human face to make sure that the person is present or not plus the person is a little bit farther as compared to uh others and the scene is not too complex but still it's comparatively different than what it might have seen and here are some of the activation maps that it saw so for exit 2 it was able to see the faces clearly and it was like okay I'm sure 98% sure yes take Exit 2 I'm done I have my confidence similarly for others exit 2 and exit 3 were kind of pretty

1:19:37 good but in the rest of it you see that it took it about Exit 3 and exit 4 to be more confident and say that okay I'm now more sure that there is a person present in this scene and it's not empty scene so uh overall in terms of key metric these are some of the values that you can expect to have another uh new technique that's coming from uh the scenario of uh large language models is mixture of depths which is equally applicable to our convolutional models or normal small models which helps us to uh again have Dynamic computation so that is called mixture of depths so this is like a architecture from recent paperback Google called mixture of depths where uh for a Transformer model they have uh dynamic compute allocation so what they have is given the importance of a token you can either pass it if it's too important you pass it through the entire uh self attention layer and multi-layer perceptron if it's not too important you

1:20:38 can just bypass it through uh the other layer so the core of this is the routing layer that's right there in a very general sense if you want to design it using say a convolution network what you can do is maybe have a convol layer with a linear layer that has softmax as output as your input to the model that softmax say gives you only two probabilities that take the exit or take the entire route based upon that you can compute the things and you can actually learn during training that if the input is important enough or the token is important enough to pass it through the entire model or just skip it all together and this again gives you Dynamic computer allocation so uh in this you have computation uh savings and you it also provides you with a static computational graph and again on the key metric you can have uh some of these numbers that you can achieve so we talked about neural architecture search early exits

1:21:31 and mixture of depths another technique that's there is Hardware aare design so we have talked a lot about since the afternoon that uh you need to understand your Hardware because it's a hardware software code design problem and not only a software or a hardware design problem so what this means is that if you know that normal convolution layers that I am using uh for doing say inferencing or making a model and using them during inferencing they might be consuming more amount of uh memory also we know that convolutions in convolution most of the filters contain redundant information which is that most of the filters might look the same so is there any way that we can reduce that amount of memory consumption because you have to store those filters so can we reduce that amount of memory and compute consumption because you have to compute all of those filters can we do that so one of the concept comes from this paper

1:22:21 run don't walk which is called partial convolutions where is instead of uh running your convolutions across all the feature Maps you have a strategy where you select only certain number of feature maps and you copy rest of the feature maps that you have computed you apply the next convolution only on that selected set of feature maps and then concatenate the rest of the feature Maps as it is so here you reduce the computation and the runtime memory bandwidth because you only did the computer on a certain number of uh feature maps that you extracted similarly as nari also uh mentioned about pixel Shuffle and un shuffling operations instead of uh using in terms of upsampling or down sampling using transpose convolutions which actually have a lot of compute you can use pixel Shuffle or un shuffling which basically is a rearrangement operation which does not cost you too much amount of compute

1:23:06 on the hardware similarly um one paper from Apple that came out uh sometime ago mobile nit uh mobile one sorry so uh that paper also uh follows the same similar techniques and it shows that how you can design neural networks if you understand your Hardware well and they were able to run their models on Ione 12 within 1 millisecond for normal classification problems on image net another design again from Apple paper called Fast vit so we all know Vision Transformers are heavy because uh selfer tension has a polinomial Time complexity and then also because of the feed forward Network that's right after that so all of those combin make it more computationally and memory intensive you want to reduce that complexity but you you cannot remove those layers but instead of that if you want to use some kind of like convolution layers what you can do you can overp parameterize your model during training so most of the

1:24:02 models that we train are overp parameterized because as we have seen from the papers that do pruning or lottery ticket hypothesis is that in the bigger model there are mult there are multiple smaller models that can actually perform at a similar or better level and you don't need all the weights so what this basically does is uses a reparameterization trick where during uh training you over parameterize your model during inference you can actually combine all the parallel convolution layers that you had in the branches and you can represent all of them using a single convolution layer and it still gives you the same performance while reducing amount of compute and latency and memory bandwidth during inference so in terms of uh performance again on key metrix we can expect some of these values uh on average now we have designed our model we understood the hardware we have the model ready what is The Next Step can we

1:24:55 do something more to optimize our model and reduce the size yes uh the technique is called knowledge distillation so we have this bigger model but this model is still over parameterized because there is some scope of optimizing it further now model distillation is a technique in very general sense where you have say a bigger teacher model and you want to train your student so in a class you have a teacher who knows about the concepts he or she wants to train those Concepts to the students so this is basically if somebody is teaching a course they are distilling the knowledge that they have gained over the years into the students in that certain amount of time during a semester right so it's something similar to that the teacher is a bigger model that's over parameterized that has been trained you can make a smaller version of the model that becomes the student model now what happens in knowledge islation is that

1:25:43 the teacher model is already trained on a certain task now you freeze that model for knowledge distillation and you only train the student model you pass in the same data through both the teacher and the student model the teacher model gives you the output Logics and the student model also gives you the output Logics which is the prediction now what you are trying to do is basically the teacher model has some sort of a weight distribution you are trying to bring the weight distribution of the student model as close as possible to the student model uh to the teacher model and the best law for doing that is scale Divergence laws so basically you apply the K Divergence laws and you only back propagate through the student model so that only the student model weights are being updated the teacher model weights are always Frozen so that way you can actually distill uh the knowledge of the weights that the St uh teacher has

1:26:33 learned into the student model one of the most prominent examples in Transformers is distal bird which was distal a distal version from the bigger bir model that you can use for a faster inference uh for text tasks and these are some of the key metrics that you can uh get on these tasks in general sense in production what I have seen personally when I use knowledge distillation if I'm training a model from scratch the student model from scratch usually using knowledge distillation gives me 1 to 5% additional benefits in terms of performance or accuracy as compared to training the model from scratch so this is like a really good technique which can be easily used now we have a model ready our model is smaller we optimize it for the hardware we have all the steps in what's next model is still over parameterized we can still prune the model there is still a smaller model hiding somewhere in there so model pruning is another

1:27:27 task that we can perform after this in model pruning you can prone it either based upon say if you have convolutions you can prune a single Channel or multiple channels you can prune the entire filters you can prune specific weights So based upon that you will end up uh with a smaller model uh which has been pruned in terms of Transformers you have to be a little bit more cautious in pruning as compared to General DNN or CNN because you can PR in two places one is the self attention layer and second is the feed forward Network self attention as you know in Transformers uh gives you say uh output probabilities and even if the probability is too low even then it's of atmost importance to figure out what's the next token in a text uh task so that means you don't want to just prune it naively because you might lose those very small values if you are doing say for example magnitude pruning where you prune only

1:28:20 the lowest values you don't want to do that so you need to be really careful there and similar thing applies to the feed forward Network in the Transformer layer so now our model we started from a bigger model we distilled it our model runs fine on the hardware we have the latency memory bandwidth everything done pruning is done my model is small what are the caveats uh I can do magnitude uh pruning I can do structured versus unstructured pruning I can either prune equally the channels across all the layers I can uh prune them separately like they might not be equal I can also do local versus global pruning like prune certain number of Weights in a certain layer or I can do it globally in terms of key metrics we can expect some of these values up to 90x smaller models while maintaining performance so what's the caveat here it all sounds so good right it depends upon what your Hardware supports so if now

1:29:19 with proning what we are saying is I have a certain kind of a DNN where I have ch out some connections or maybe I have set those weights to zero that means given an input it will be multiplied by zeros the output will be zeros my activation a zero so I'm not getting anything so basically what that means is that it's not part of my network for an inference point of view now if my Hardware is not smart enough to take care of those smart matrices or those prune matrices or sparse matrices what will happen is during inference I'm still multiplying zeros so it's it's unnecessary compute that I'm still doing although it might not consume too much memory but I'm still doing the compute which is of no use now what you want to understand here is that if your Hardware actually supports pruning and it can do sparse Matrix multiplications what in general s happens is you can take an example of like a00 gpus of Nvidia what happens in

1:30:12 those gpus is like if you have a spar Matrix where most of the values in the your Matrix are zeros at run time basically it can bring all the non-zero values together in a matrix do just a computation on that and then uh provide you the final Matrix out of that so how that helps is that you are not doing additional compute you're saving on the memory because you're not considering the zeros and similarly on the infront side the hard new Hardwares are smart enough that they can automatically do these kind of things so you only want to do the proning if your Hardware supports it otherwise it won't bring you the benefits that you want in terms of model size and latency and other kind of things my model is small enough but it's still at fp32 floating Point 32 or floating Point 16 depending upon if I train in full Precision or mixed precision what else can I do I can do model quantization so quantization in

1:30:58 very basic sense is I have a model at floating Point 32 of floating 16 but it's still a huge range my model weights are high how does that matter it consumes my memory bandwidth I don't have that memory what do I do reduce the memory bandwidth how much can I do it whatever the hardware supports and whatever my framework supports if my Hardware supports only inate I can only go to inate so basically what happens is from fp32 uh in that range I can go down to a range of inate say between 0 to uh 256 or uh 127 to 12- 127 to 128 something like that but what happens here you end up with quantization errors why I'm am taking values that are distributed across a very wide range and I'm trying to bring them down into very small range so what that means if two values are kind of close enough in a wide range they will map to the same value in a low range so when I'm trying to get back from the low range to the

1:31:55 high range I will have additional errors which is my quantization error but in general sense how quantisation helps is that if you see on the image on the right if I'm doing a 8bit addition so our Hardware does basic operation at a very basic level is multiply and accumulate and it's doing multiplication and addition so input multiply the weights do the additions get the sum apply activations right the 8bit addition will consume this much amount of energy which basically reduces your amount of power profile and you can maybe increase the size of the S so maybe you can add more amount of memory or processing elements and you can make your product cheaper as well but if I run it at 32bit then that means my model weights first of all won't fit on the device I will have to go off the device to some flash memory and bring the weights in so that adds to my memory bandwidth that adds to my latency that adds to my additional

1:32:52 power because every time I'm going off the chip to bring the weights in or activations in I'm adding to my memory and latency so that's consuming my additional power similarly on the area cost if my area can be reduced like if I say my Hardware will only support 8 bit activation 8 bit weights and activations what that means is I can reduce my memory area and as a reason of that I can increase my processing elements so if I increase my processing elements and my model is small because of 8bit activations and weights I can maybe process multiple models of I can have better parallelization across all the processing elements I can increase my throughput so if you remember earlier we discussed about the rline model where we mentioned about the throughput and the latency and the memory bandwidth so memory bound versus compute bound so if you have more processing elements you can be more on the compute bound side

1:33:40 that's where we love to be in terms of quantization in general sense you can do it in two ways uh either you can do Post training quation your model is trained after that you can provide a calibration data set which for each layer will calculate the minimum and maximum values in which you want to quantize and then it will automatically quantize your models into like int8 or in four or whatever you want or you can do quantization aware training where during training time you tell the model weights that hey I want during inference time for you to be at inate so train it like that so it will add fake quantization and dequantization steps and it'll keep the weights uh during the for pass at int8 or in four but during uh back back propagation you still want to keep them at floating bone 32 otherwise you will run into scaling issues because your gradients might become zero another way of how quantization is

1:34:31 apply to Transformers is low rank adapters or quantize low rank adapters uh where you can uh freeze most of the model and just uh set the model weights to in4 and the activations to 8 bit or 16bit reason being why can we not set the activations to 4bit as well as we can do in convolution to 8bit or 4bit is because in activations that we see the self attention layer for example in Transformers their values can go all the way from very high values to very low values so that means the activations actually have outliers and when you map the outliers you end up with higher quantization errors which propagate even more in Transformers so due to that you want to avoid keeping the activations to a very low bit width you want to keep it at eight or 16 bit but the weights are still fine you can still keep them at int four so that's what you see nowadays when you look at Transformers that are running on device this is what's going

1:35:22 on behind the scenes so in uh overall sense these are the key metrics that you can achieve we have quantizer model we have prun our model we have a very small model so let's look at a quick summary neural architecture search you can automate your model development you can optimize the model you can uh have a very efficient model which is with the hardware in the loop that runs perfectly on Hardware then we have Hardware aare design you know which layers are supported or maybe there are certain layer that your model can Hardware can accelerate better so you use that as part of your model development cycle and it can adapt easily for your Hardware then once you have designed the model you do perform knowledge distillation you want the model to be smaller because it's still over parameterized so you can make it smaller transfer the knowledge makes keeps it still efficient maybe have one to 2% or 5% more performance than

1:36:11 training it from scratch then my model is still over parameterized PR the model if the hardware supports it make it even smaller or make it run faster less amount of memory bandwidth lower latency finally it's still in fp32 quantize it to inate or lower if your Hardware supports it and you have even a smaller model which is ready to run now looking at a case study we'll look at two case studies here one is how we went from uh looking at a video which is 180° field of view uh from a camera like this or a panas 50 or a panas 50 VBS and what are the challenges of developing a s simple model as simple as an object detector or a gaze estimation or things like that now when we talk about object detection it seems like a very general like a simple problem it's already a solved problem right so what could go wrong interesting well uh try putting an object detection model in production uh where people do

1:37:16 all sorts of things there are all sorts of like plants and different kind of animals and things walking around in terms of false positives and you'll be amazed what kind of things it can detect as a person actually another uh problem that comes in is that you have this 180° field of view that's almost like a 329 aspect ratio you are passing this higher resolution input uh maybe at say 1080p into your model so as we saw earlier the bigger your input the bigger your activation even though your model weights are fixed so uh these become like a big issue another thing is I want to detect people who are in medium small to medium rooms that means I could be detecting people who are maybe say 1 to 20 people in a room sitting at anywhere between uh say 5 to 8 feet to all the way to 20 ft as people go farther away they become smaller in size in terms of pixels so it becomes a really hard problem to differentiate between the

1:38:15 false positive and a false negative to detect a person so like nari mentioned in the previous presentation let's look at at what are the requirements that we were looking at and then what were the options that we had and how we went about deploying the models so in terms of Intel midex uh since this model is deployed across two devices right now Intel mirex and a qualcom processor so we'll talk about both of them briefly how the life cycle looks like in terms of development rest of the techniques are kind of common but wherever uh some amount of optimization apply I'll talk about them for Qualcomm in midex you know uh entire system is like three cameras for this big sensor going in eight microphones going in audio has a separate path video is there both are sharing memory in this s so we have this small part and even in this small part we have this small dot that is what is available to me to run

1:39:10 this model like nari mentioned he only got out of six shaven slices maybe one shaven slice I'm lucky I got two so the compute is limited memory is limited latency again is Li lied because instead of a single camera in that case you are now streaming uh three cameras which are getting sted and pre and postprocessing and all those things are happening bigger frame resolution more amount of memory so uh latency is the issue it needs to detect people as far as like 20 ft even in this room it should be able to detect it people can deploy the camera anywhere irrespective of whatever you specify it for uh Precision needs to be good so people don't care uh they're buying a costly product the Precision needs to be really good uh the trade-off again is accuracy versus latency and accuracy versus memory bandwidth and the number of detected people as it talked about if we look at a Qualcomm architecture from a very high level view

1:40:07 because every architecture has some specifics you are looking at a CPU which is a bit more powerful as compared to mirex mirex is just U kind of a like a CPU with like couple of threads with couple of acceleration Parts you have a separate dedicated GPU that can support floating Point 16 operations and then you have a DSP and there are couple of other options in the DSP uh one could be for application based one could be for compute based but here we are just talking about the compute based that is specifically for um image processing and computational photography so it's meant to accelerate convolutions and other kind of operations well so in this also you have two kind of different things that might be available across products which could be an hvx and HTA processor the hvx is basically a compute DSP Hardware Vector extensions so this supports int8 quantize models that you can run but again the kind of layers

1:41:01 that are supported may or may not be limited depending upon when you're developing the models because like for example I was developing a model and trying to deploy it on that and silu activation layer were not available on this or not supported so I had two choices either I decompose that silu activation into some simpler operations or I fall back to a Rel activation that adds sparcity and it's easily available across the board but then I lose performance by around 5% then you have a AIP which is the hardware tensor accelerator which is like a very very specific Matrix accelerator uh with two specific uh threads that provide Hardware compression and uh that actually has a subset of layers that cdsp supports so that means you are going even lower but now you have uh inate and floating Point 16 support it can run a little bit faster if you can put all the model layers on this now here's the caveat if

1:41:58 I am not able to run all my model layers either on AIP or on cdsp those layers will fall back on the CPU or on the GPU now for each of the fall back layers there will be additional read and write operation so added memory added latency right so that becomes an issue Precision will change CPU does not support inate maybe so what do you do floating Point 32 Precision changes my model accuracy is gone additional amount of memory jumps will have to be done so those are like some of the issues now once the model has done processing you apply non Max separation or whatever you get your outputs bounding boxes classes pass that back to the GPU or the CPU for rendering if you want so in general sense if you have a video input you get a video output bounding boxes something like this will uh it will look something like this in terms of workflow uh for meridex NY mentioned about uh how to go from like py or onx to deploy in them and

1:42:55 fish will be talking about that a bit more but here in very general sense these are like some of the things that it supports uh so you basically have uh your model trained in py toch uh tensorflow caras or whatever you convert that into either Onyx or Snappy framework if you are using it directly supports all the models like uh if you are using pyos you'll have to export it to Tor script and things like that once you have that you pass them through the Frameworks you get a specific binary uh Snappy gives you like a DLC file so what this compiler is doing basically it knows that what layers are supported you will Define a runtime where you want to run it cdsp AIP whatever you define that based upon that it'll select us a couple of layers it will make the optimization again kernel fusion and other kind of optimizations replace the layers with some other things if it has to do some kind of graph

1:43:44 optimizations do like uh inate optimizations uh using uh some kind of buil-in quantizations if you haven't done it already if you don't provide a quanti quanti model it'll do the quantization for you using a and other things and then basically finally you end up with a binary file that you can deploy on the specific Hardware that you defined for the execution so memory Pand we already talked about earlier uh what it looks like for each of the layers you have intermediate activations the model weights are constant inputs and outputs so all of this comprises of the memory bandwidth that's the first challenge across both the devices memory is the issue model latency is another issue like Nar mentioned about the skip connections if I want to have any kind of skip connections to pass the features forward uh it uh gives you good advantage in terms of uh additional regularization and things like that but

1:44:33 then you have issues while running that on inference uh you'll of course you don't want higher latency you have a latency budget across both the devices whether it's inate or fp16 you have little bit of uh more flexibility on the int side but then accuracy also kind of decreases if you are not cautious there while converting a model from fp32 int8 so the best way there is to train it as close as possible to your inference runtime for mirex for example floating Point 16 is only available we try to train our models with mix uh mixed Precision or half Precision so that when we are exporting the model there is no to very minimal loss in accuracy and that's what we have seen in production similarly for uh qualcom Hardware int is supported train and half Precision then deploy to inted or use quantization aware uh training so that when you deploy model when you do the uh conversion for the model your model still runs to

1:45:27 approximately the same accuracy that you see in the bigger model that you have trained another challenge model accuracy so synthetic data set that nari was talking about this is something that it looks like for the panagas 50 you can see like people could be far enough people could be have doing like different kind of gestures in terms of input resolution it looks kind of like this 180° field of view as compared to the smaller one that that will offer so the Bigg input resolution you have to look across the entire image you have to detect the people do segmentation pose estimation whatever so all the problems apply data set limitation of course everyone has heard about that and everyone has it because data set collection and annotation is the costliest Pro process so just to give you a basic sense of uh what the challenges look like uh this is what it says so in a meeting room or while standing or

1:46:17 wherever the camera is deployed Healthcare setting meeting room setting any other setting people people could be facing sideways uh people could have long hair where you don't even see the face if the model has learned to extract just the facial features how do you detect them so you need some kind of temporal consistency or variation in the data set there uh people could be facing away from the camera where you don't even see the face you only see back of the head and if the person is sitting with the black jacket and the black long hair the model might think it's just a black blog and it will say it's not a person it has happened to us actually in production and in testing in office person is far away from the camera person is too far away the pixels are too small we want to avoid false positives as much as possible so a better way is to not detect the person or to use temporal consistency to keep

1:47:05 the person there when they're moving uh farther than to have a false positive there because in the end experience the false positive hurts more than not detecting or using temporal consistency for us another people might be partially occluded what if you are making a decision for framing a person and you don't even see the head and if you're using just the head for uh framing decisions that person is basically invisible to you so what's the problem impact these are the actual impacts that we have seen in production when I joined uh for this uh smaller device uh we had an issue where the model was having some kind of false positive we sent it to Microsoft for certification they said there are false positive you need to fix this I spend my entire weekend and the next week just collecting more data from different sources from commercial uh sources as well as generating synthetic data going

1:48:00 through all the annotation making sure annotations are clean training and fine tuning and iterating through the models making sure the accuracy is good and it does not have the same false positives there is a reduction in that because we have a hold out test set for all of those things at the same time making sure I'm not changing the architecture too much or the latency or any other thing so I need to take so many things into consideration before I can make a decision that okay the model is ready to go out that's just certification it's not even even going out to production to people this is just initial internal testing similar thing happened on panas 50 and panas 50 VPS uh we were using a older model uh before what happens uh you pass in this big input now depending upon if the model is big or not if you have more number of people that means you are having more number of outputs and there is some additional load there

1:48:49 you run out of memory at some point of time if you run out of memory and the camera is running you basically run into frame Corruptions your frame starts corrupting you see like blobs coming up in the frames you're like what's happening to the video it's a really costly camera or otherwise the camera might just crash that's the worst case scenario so in both the cases we don't want that but that actually happened so for all of these things that's why we reiterate memory bandwidth latency performance all of them are equally important we cannot just have one and the other so the goals are uh input sizes fix as as nari mentioned for midex floating Point 16 is fixed in the hardware can't do anything about that model parameter rediction is something I can do but again sparity is not supported on madex so can't do pruning there um Precision should remain the same less amount of false positive and

1:49:38 false negatives so again tradeoff between memory bandwidth and latency so a lot of things in terms of model designing as I mentioned before half Precision training so that as a go to fp16 model deployment on mirex I don't have loss in precision and similarly ization aware training uh for qualcom Hardware so that I don't lose Precision when I go to int it uh in terms of convolution use efficient uh convolution layers like we discussed before ghost convolutions and U partial convolutions depthwise convolutions actually add a lot of memory uh bandwidth the reason being for each of you are taking each of the separate channels and in doing convolution with that so you have to read and write a lot of those and keep them in memory separately as compared to normal convolution so that did not help a lot uh feature size as nari mentioned we have this bck of a Frum which is even bigger than what the panagas 20 actually

1:50:31 has so the biggest thing is uh take all of that make it small as soon as possible by using a bigger kernel size or using like space to depth uh kind of layer so that all the spatial Dimensions go into the depth without losing the features you at least make the input resolution as small as possible how does that help you can process it faster your activation size will be smaller as you go above and your memory bandwidth and latency are in control uh space to depth uh pixel shuffling architecture make sure everything runs on nce and same for Qualcomm all the layers are supported and we don't have to hopefully write any custom Ops so just some solution in detail as I mentioned space to depth uh I have this big image which has RGB channels so pushing them across depth so that my spatial resolution goes down into the depth without losing the features in uh terms of uh the performance metrics what

1:51:28 kind of things do I get if I use a convolution 2d with a batch n and a reu I get something around 352 bytes in parameters for space to depth I get only 150 it's almost half so this actually helps me a lot when I'm doing that in the feature size because this is saving me both in terms of compute and latency and the memory bandwidth similarly uh like Nasi mentioned using dens net uh kind of layers of the architecture where instead of adding all the features at the end you are concatenating them as well as you are calculating all the features and you reusing the features so when you are calculating a feature for each layer a different feature and then you're reusing that and concatenating at the end that means you reducing your memory bandwidth because you don't want to recompute all the things whatever you computed you are reusing at the end and concatenating them so you have all the same features while the in between

1:52:22 features so that way you are reducing your me memory bandwidth for additional feature extraction there while maintaining the performance so what do we get here you have a dense feature block uh where in comparison to convolution batch Nom reu you have 18.5 6K parameter whereas you can reduce that to 15.5 uh K parameters ghost convolution as we talked earlier uh convolutions uh have a lot of feature uh repres uh repetitions so as you can see in the image the red ones are basically kind of look similar the green one look similar the blue one look kind of similar so you can basically reduce most of them and reduce computation and the memory bandwidth so use uh some kind of ghost convolution where you reduce the uh number of output channels for the first convolution and then use cheaper operations to calculate rest of the features using maybe depthwise or partial convolution and things like that where you can save on

1:53:16 compute and memory and what do we get out of this again good amount of reduction in terms of Max and number of parameters across the board for the same number of input and output channels we talked about partial convolutions as well uh we get some kind of optimization here we are just to remind you we are just copy pasting half of the channels uh based upon a metric where we are just m masking out rest of the channels uh which are not useful features or we don't want to recompute them which are repetitive we just uh perform convolution on a certain set of layers and don't do it on on the rest of the features we just copy paste them to the next layer and in this we have a good amount of parameter reduction so for normal convolution for 9.28 uh K parameters we go down to 742 bytes that's a good amount of um reduction in terms of both compute memory bandwidth and the latency one caveat here in partial convolutions uh

1:54:14 this number is there when you have same input and output because you are concatenating the channels so that means you cannot change the output dimension it will not work there so if you want to concatenate them that's where you will come closer to the convolution uh number of parameters and Mac operations because you'll have to add additional uh maybe one by one convolutions there for adding um increasing the number of channels for the layers that you copied then replacing transpose col Evolution that we uh talked about again uh using pixel shuffer or unshuffle which is basically kind of a no op uh you don't do any operations you are taking just everything from the depth and pushing that into the spal Dimension and you are getting the output in terms of instead of doing transpose convolution for up sampling or down sampling so you are saving compute there and what we see in the numbers uh the

1:55:02 execution time is uh less overall on the mirex hardware so in terms of milliseconds you are at only 228 milliseconds as compared to 2.98 milliseconds so as narc showed that for the model itself we only had like 18 milliseconds and N to end pipeline 33 milliseconds so every millisecond counts so what was the impact of the choices in terms of uh the object detection itself overall in terms of the latency we achieved around 20% 30% 40% 15% reduction across the board uh using these choices in terms of the memory bandwidth we had uh these kind of improvements across the board and finally the models are all deployed they're all in production and yes there are always little bit complaints here and there about longtail problem because there is always an h and everyone knows about the long tail where you can fix most of the things through data set for most of the part in a distribution but you always have a long tail where there

1:56:00 might be some edge cases that you always have to solve for so that's where you need to add data and re it a little bit but overall we're pretty happy with the performance so here's two videos uh one of object detector running on that camera and another one running on our panas 50 and panas 50 VPS so here you can see the bounding box looks kind of pretty stable and that's because we also added temporal stability to this in terms of temporal stability we Tred two approaches so the architecture basically looks something like a yellow architecture where we are utilizing all of these layers combined in terms of temporal consistency we added it tried it two ways we added a Comm Gru layer at the backbone before passing it to the head where you do upsampling for individual scales and we have three scales at the output small medium and large we tried another approach where we tried uh using LSM layers uh but that

1:56:52 was was like individual lstm layers for each of the scales like for each of the hits that we have as the output and try to add there so what we notice is that that adds too much amount of latency and compute so adding them just at the backbone Works equally fine for us it reduces the performance by a little bit but given the tradeoff it's still good enough so here you can see the person moves her head up or down goes close by is far away and it's totally fine we don't see any issues all the results are above 90% in terms of performance temporal stability is good so we don't miss a frame if you just rain on images you see the flickering issues sometimes in temporality you see much more smoothness kind of similarly we have panas 50 and panas 50 VBS with 180° field of view so this is what a room would look like in a 180 degree field of view and we need to detect every person from each end to each end

1:57:41 and everyone in between we cannot miss anyone irrespective of partial occlusions or any kind of uh artifacts or any things that they are wearing maybe like headphones or other kind of thing that might look like a person and based on this feature uh we have made a model that's good enough but now it become a little bit too good because uh as after Co offices started opening uh the it guys thought it's a good idea to have more open spaces so they started adding Glass Walls uh transparent glass walls to the meeting rooms now the model uh if you deploy the same detector in a small medium uh meeting room scenario it works well but if somebody's walking from from outside it will still detect them and it will frame them as well if it's a say intelligent Zoom feature where it's zooming onto the people or just framing them separately so given this problem uh we had to figure out a solution and we came

1:58:34 up with the solution uh using we okay the thing is we don't have additional compute on mirex we don't have additional memory we cannot put additional model we cannot use a depth model uh we cannot add anything extra that could help us so what do we do we came up with a clever trick using an algorithm that's patent pending but we'll tell you about how the thing looks like so on the video on the left what you see is people sitting in a room and you can see people sitting outside so the intelligent Zoom is framing the people sitting outside as well because it's able to detect those people as well right so you see uh the experience is not that great because the camera is moving around it's more distracting at the bottom what you see is using the intelligent meeting spaces that you can Define it could be either a room or it could just be this place where you can just Define your virtual meeting space

1:59:24 the camera knows okay these are my limits it will not detect people outside but it will not frame them so it knows whoever are there it knows where the people are but it won't frame them so that's the intelligence part of it okay awesome enough of me any questions so far otherwise I'll give it next to febrio yes sir observability of so many different TW one the other how do I know what lot of like statistics and lot ofel so what is I the tooling or the way you actually get the avability of yes is there any like recommendation around thate so there is no recipe that fits everyone uh that's what I'm saying since the beginning it's a software Hardware code design problem first thing starts from understanding the hardware because writing the model is the easiest part you can train the model write the model data set collection annotation is the hard part understanding the hardware is the hard part because in most of the

2:00:29 companies either that process will be separated across teams and then there's a like gap between the communication gap and that's where issues happen so for me I was lucky enough because I was working across both the things by myself so first step I would say that understand the hardware and what tooling does your like if you don't make the hardware whatever whoever is the vendor what are they providing go through documentation documentation provides you a good amount of things go through some of the examples go through some of the sdks how they implement the kernels do you have the flexibility of implementing kernels if you have maybe you might be good with the binarized neural network as well so you need to understand that first once you understand that you have some idea around the search space that you have from there it's alterative process which is comparatively easier because you can

2:01:13 just do like a grid search or as like a knife search or you can do neural architecture search and things like that for the model part and that's the easy part understanding what things are available to you what are the limitation or the bounds that you have to work for within for example what's your bitwidth maximum what's the minimum maximum bit we does it support mix Precision does it support int what does it support what kind of layers does it support am I using those layers or not what kind of performance difference do I see if I use this layer versus that layer what kind of Frameworks does it support some some of them might not even support all the Frameworks so you might like even that will decide what framework you support it's not the other way around like you write your framework and after after that you'll have to deal with a ton of things where you go through model optimization and conversion just to

2:01:56 bring it closer to deployment on the hardware and there in between something might happen and you might lose some performance so there are like a lot of things and it's iterative process uh that you have to go through by understanding the hardware and then going through the software and having a code design there yes yeah

2:02:32 so we do keep it to original aspect ratio U because if we change the aspect ratio what we have seen that it might distort the people or what the models are learning and that causes a lot of issues because we want to keep our training data set and the training thing as close as possible to the test time so whatever the model is seeing is the entire video that then downscales and runs through the model so we try to keep the aspect ratio to the same but we have tried uh different kind of um like augmentations around that like using uh mixup and other kind of augmentation that we have tried to see if that works or not but we had we had tried with the aspect ratios but it didn't give us too many like good results we also tried like training with a very small size and then going up uh from there we did see some amount of improvements but maintaining the aspect ratio actually helped a lot okay uh uh in fact we I take it

2:03:31 later yes we we will stay here um even after 5:30 to answer all the questions okay then let's move on because uh we have a lot of presentation yet I Elizabeth and Shem uh in my case I'm going to change a bit of perspective here and I will try to show you in practice how to deploy models into the MedEx device in my case what I'm going to show to you it is in terms of gaze correction okay the idea here it is that we keep the eye contact from a model and this model is going to be executed into the open V uh MedEx I'm going to show you the steps but basically what I want it is as once I have the trained model the idea is that I converted this for Onyx and they have the last part of in terms of optimization then you can deploy directly into the camera then the the basically the pipeline it is uh I'm going to have here uh just a moment I need this resource now yes the model development is the one

2:04:50 that you usually work the day by day okay developing a model neuron network using Pyar using tensil floor then this is the one that probably you you you have uh performed um at least once in your life the idea it is once you have the model we're going to convert this for one Onyx version and Onyx is ideal because it's a universal format if you have the Onyx at least the expected to convert for Onyx it's quite easy to convert to The Blob and the blop is the format that we are going to deploy into the camera and open Vino it basically sometimes we call this is open V another time MedEx is the same ship set but Intel provides to us the compiler tool that help us to convert from Onyx to The Blob okay once you have the blob uh converted and it's vpu based The Blob we can apply one inference and deploy the module into the camera uh I'm going to give you more details about the model later but this

2:05:55 is the architecture the basic architecture from my Jabra I correction okay and we have a mix here in terms of knowledge then let's say this blue uh rectangles are the uh the neuron network uh splited in three parts and these are the one that I'm going to focus today in terms of how to deploy into the camera but I also have some steps that involves engineer and computer vision okay but the idea is that I convert this entire Pipeline and can execute into the camera okay but I'm going to consider like only the col model for the moment once I have the model trained the first step it is uh converted to Onyx then uh the one that I use more in the company it is the py T sometimes we need we must Implement also in tens floor but I'm going to focus in Pyon and and the library itself provid to you uh different methods you can use the export okay and the export uh it was replaced by this tor d uh and

2:07:06 how it works my pipeline I have everything implemented using DVC then it's only executed the DVC every time that I increase for example the database then I started training the entirey process my DVC it has the entirey pipeline that it's going to download the data set preprocess apply the data cleaning the data set started the training process uh once it's achieved the the maximum number of epox or a better accuracy I'm going to evaluate the model and in the end I will deploy the model convert this to Onyx okay once I have the onx convert to the blop and this is the entire automatic process then I can deploy into to our uh experience on okay then everything we uh we implemented this step uh automatically and is going to be executed only using the DVC but in terms of code you are going to say that's quite simple okay then the code that I'm going to show to you it's a simple py of code and as you can see sometimes two or

2:08:13 three lines of code to install like the onx to export or to apply the inference it's only basically two peopleall then these are the libraries it's needed in your uh python environment my case I use the cond then once I install this very short process I can start like deploy the model uh in my pipeline after training the modu I have uh the check uh the checkpoints ready I can like create the model itself as one object and this is only to Define okay this is the format of my tensor the input t I'm going to feed the model it could be any Rand data because the most important here for me it is the shape once I have this I can only include this line of code and it's a simple line of code that is going to evaluate the model and convert it these to Onyx once you have the Onyx object in your hands it's on P you want save and Define what is the path you'd like to save your onx and the best here it's going to be Universal as

2:09:19 I said to you before then I can start like deploy for uh different platforms even in python as you can see three lines of code it's only import Onyx and this is the runtime I can load again the same Onyx and start using the Onyx as a model okay as you can see it's quite it's quite trivial they St in terms of python uh development uh probably you know but we have this nro application uh and nro application can help you to see okay how is the overview of my model you can go into uh this node and see what are the weights and they have basically one overview about the modu and in my case here this is the same architecture from that uh color model I said to you before and what I've made here as you can see in parallel I execute the same model twice why because I have the left and the right eye okay the only thing that happened with me here it is I need to include this part because to convert to Onyx it works

2:10:29 quite well to have like two inputs and two outputs I can feed my model with the left and right I image and have the output with the transformed image as the left and right output the problem it is when I converted from Onyx to the block I lose this information then what was the idea I keep the same weights in ter of my architecture from my module I only included here this step and what is that I'm going to like before feeding the module I'm emerging okay in terms of channel the left and right I image I feed this as only a single input inside the model it's only split and in parallel I'm going to execute the left and right eye image at the end you can see it's only one concatenation I concatenate the result and I have like basically a single input and a single out outut this was the alternative that I found the easiest one and I'm not going to change and I don't need to retrain the model it's only one is small

2:11:30 adjust I had here in my model is that clear this part then I have the Onyx Onyx Universal what is the next step is the blob and the blob conversion I'm going to use uh open V 2 Kit and no to is going to provide for you uh all uh the tools needed to apply one of the techniques explained by ano then I have the post training quantization okay two steps only the first one is the model Optimizer and this op model optimizers get the onx that we had before and convert into fires one XML that basically describe the architecture of my neural network and the second one that is the bean okay once I have this it's only feed again these two uh file and I can generate like the blob and this is the one that we try to achieve and I need to specify okay this blob what's the type of the block it could be vpu and the vpu it's for the MedEx executing here this is the important one but it could be also CPU it could be GPU

2:12:38 okay you can convert in different formats but the one that I'm going to try here it's converted this into the MedEx this is only the diagram as I said to you it's the onx then I apply first the optimization and after the the uh modu compiler okay the block is the one that we are going to install directly into the camera okay deploying into the camera in that case I'm going to use this camera that I brought today it's the look sonis uh I'm not sure if you know but this o a k it means open CV AI kit okay then this is another another name for for the schem thank you okay in terms of the optimizer it's everything that you saw before that uh narcy and anui presented for you the first one it's in terms of the type of data that we have we are going to execute uh in the media de then the format here is going to be FP 16 okay when you deploy this uh when you convert this into a blop it's not 32 anymore

2:13:48 it's going to be converted for FP uh 16 we also have uh a normalization because the same way that you work with open CV open CV provide for you one image as one npire array and the range of these image is from 0 to 2 55 in most of the models you need to normalize between zero and one sometime minus one and one minus 0.5 plus 0.5 doesn't matter but you you have the is uh normalization step you can include the normalization step directly when you convert the block then you can feed like the N array that represent your image the same format as the one used by open CV and the model itself has the first layer to apply the normalization okay then you can decide if you want to uh feed asigned 8 or even uh fp6 okay this is something that you decide another point it is you must take care I guess if you had this experience before to work with buff P Tor and tensil floor you know that sometimes the channel came

2:15:00 in first and the channel came in the last part then you need to see how is the format if Channel it came first in the last one and another problem for us that work with open CV sometimes you will see that we executing BGR but it could be the model training in RGB then it's something that we must take care as well when we apply the optimization in practice as you can see one line of code once you have the P installed and you have access to the Onyx how to convert the Onyx into the B and the XML file the first step in terms of optimization then the post training uh quantization it's this line of code here you can open the terminal and execute in a couple of milliseconds you have the first two files and it it is optimized to be executed here uh in the media de okay of course you can apply before all the techniques explained by by Ani like Bruning but in this case I'm only apply the the the simplest uh optimization and

2:16:06 it's only a single line of code okay then it's going to save the beam for me and also the XML what I need now it's basically uh use the compiler tool and the compiler tool it is the one where you are going to Define okay what I want as input layer I want with or without normalization then you can have this flag here and this flag basically tell to me okay I'm going to use an signed 8 okay to power8 I have 256 representation these are the gray scale levels that represent my image then I can provide this information or if you are going to normalize by yourself okay with the range defined during the training modu you can also specify that this is without normalization what you have consider it is the ship set provided by Intel this media de it has like 16 shapes okay for default when you compile from uh the b or the XML file into the block usually be that you are going to execute in two shaves or two threads okay but you can

2:17:16 go up to uh 16 shapes e because in our case as I said to you the the engineer team from Jabra sometimes give it to us only one shave that we need to adapt it for the requirements to the process okay but in general the maximum number of Chaves you have is this and as I'm mention for you it's not possible to increase it like number of shave the thread of the memory available in the hardware okay we always need to work over this limitation nice to compile nowadays it's quite easy because have three Alternatives the first one it's using the blob convert that is a library available on on on python okay and basically it's going to in practice use the second one that it's the the the website uh called blob convert okay I'm going to get the the Onyx send it to a server it's going to be like converted to a blob and return for you the result the third alternative that I have here is to use the local compilation okay we

2:18:26 have in that case some security aspects here to consider if you use the first two options here what's going to happen you are going to send the model for some server and you don't know where is the server and maybe you don't want to execute this most of the case and it's the case that we work at Jabra we apply the local compilation another problem that I'm going to f at that least I face it in the day by day it is this is my company's machine and this is uh MacBook use mu1 processor use the mu1 processor to deploy the block or to to work with OP V it's really a headache okay I really have a second machine this is my personal machine and it's still uh Intel based then when you work with the Intel based processor you usually work very fast F but if you are going to the M1 probably you are going to face some problems hey I would say that it takes a day that until you compile and have this machine ready to start deploying uh

2:19:35 models in terms of the the first option you will see that it is easy nowadays you don't need in fact to compile it from the beam and XML to after that have the block you can get the Onyx directly okay Define it the type Define the number of shaves and this code here it's going to generate as output for you the block directly okay because internally in the website it applies for you uh the optimization then after that the Second Step then you don't need to apply all the Step at the same time the only thing that you need to remember it is this mod is going to be sent for for some server on the internet right okay second one it's use the website the website it's available in that link all the slides uh you can have access I I included like the link then you can click on on the QR code to access the the website or or at least the links that it is related to the the slide but in that case the

2:20:41 website it's quite simple and what you can notice it is the version that I have used here it was uh 20 22.1 we have now like 20241 and why is not available here in the website because now it's deprecated the vpu okay you don't have the access to the media deck anymore and ittel doesn't Provide support anymore okay to to uh in the newest open V to Kit you can deploy for GPU you can deploy for CPU normally but if you want to execute in the media de and especially in in the looks on camera you must go up to the version uh 22.1 then we have this limitation here okay you can select the option and you also have here like if it is open Vino open vom me the bean and the XML file or if you want to like deploy from the onx this is only one option and what I'm going to say to you here I have included this part here the XML and the second one the be okay it's only specify what are the parameters I have is the same

2:21:54 unsigned because I like the layer uh with uh normalization then it's only press it's going to process in the S and send to you download automatically for your develop then you can start using as you can see also quite a trivial this step and the last one it is the alternative when you compile uh in your computer as I uh if you try to find most of the documentation it shows to you that you you need to use the the compiler tool then it's a command in your terminal but in fact you have the python code and the python code here only works because this device name as mediad or media deex it's only available up to the versions I said to you before uh 20221 if you try to compile with the newest one you are going to get an arrow here because this option is not available anymore okay then let's think about the camera itself looks on quite good camera in terms of prototyping or at least uh apply in practice computer vision for

2:23:05 Edge AI I strongly advise you if you have opportunity to work with one of the cameras and we have different models I bought two of these that is it is the luon is O K1 Max because it has the same architecture as our camera from jabber I'm talking about the same architecture okay we don't have what the same image quality it's far away I mean the quality of the Jabra cameras we have really better quality in terms of poster process in terms of image quality in terms of even the quality of the lenses okay but the the architecture is normal the same but when I execute like the Gaz correction use the look on and use the Jabra camera you can see that the ey image produced by the Jabra camera it's really better okay but let's talk about the architecture at least it's one camera you have uh a video streaming and you can start processing so mod directly to the camera what you need to understand it is in the camera I must

2:24:15 create a pipeline and the pipeline is composed for different nodes and each node you can find different inputs and outputs and it's only a matter to connect as you have in your hand now the block the most the uh the principal mode node you are going to use is the neuron network node okay because this is the node where you can uh load the blob from your computer and your computer is act as a host and it's going to send the blop into the camera and this is could be as exed and when I mention a pipeline you need to imagine one diagram like this this came from the official documentation this part here it depends on your application okay in my case my G correction I needed to create one pipeline similar to the diagram I showed to you before and I'm going I'm going to show you also at least two more times okay but I need to connect this one is quite simple because it's only the camera the camera is like open CV always

2:25:20 provide to you a video stream you can collect a frame by frame Bas on the frame rate and it are going to send messages okay what is the message is basically the current frame that I just grabb from the camera and I can send it to the neuron Network the new networ is going to perform so what I don't know it could be uh body detector body segmentation gaze correction Ling Mark detector this is the technique that most of you produce in the day byday the neuron Network provideed also one output sometimes it could be one image sometimes One n array depends on the application and you can get that one let's say in this case it's only uh one image and I can show as one output then you need to understand this is the idea here I remove most of the um uh the source code because you can find the source code in the documentation then it's quite uh easy to understand and read Because this is python code okay

2:26:22 but in the perspective let's say the neuron Network it could be one Edge detector in this case here this node has like two inputs one input that is provided by one image and this is quite clear for you you can connect with the output from the camera once the camera provides to you uh imag and you can also have some configuration and this is the second uh uh input from your note and in the and you are going to have one output image with some modification provided by the neuron Network then what I've made as I said to you is the same architecture I try to convert like this architecture from my uh neural network based on gaze correction into the pipeline for the camera and basically I reproduce like this architecture here with the pipeline okay then um this is the part uh the fourth part I'm going to start the last one where we're going to see to show you different modalities different interaction

2:27:31 modalities that we implemented in our camera as um one experiences only inside the video tech team at jabber don't worry in the end I'll open for the question and you can ask this questions regarding to uh the development

2:27:58 here in terms of multim modality uh for Edge AI in our team at Jabra we try to think now about multimodality interaction perception and the meaning it is we try to identify like the needs from the user monitor the user needs and we can use like the eyes gestures voice what the user can interact directly to our business product because as you can imagine one situation you saw that especially this pacast camera it is used in a uh meeting room and sometimes only the host has connected like the computer into the camera and the host is the one responsible to interact with the uh communication platform like Zoom or teams and my colleagues how can for example can activate and mute and mute the microphone or enable disable the camera or at least a practice with the most basically commments available in the platform they need to ask me but we can use like different Alternatives that all the participants from the meeting

2:29:16 room can interact with the camera and can interact with the environment only use different Alternatives of interaction and we try to bring like most of these Technologies into our camera in my case I work with gaze correction as I said to you the beginning I'm one I track specialist my master my PhD was working with a tracking most of the time for uh to identify the Gaz um uh the coordinates that the user is look at on on on the screen that we called as gaze estimation or use the eyes to interact uh for disabled people that doesn't have movement for the arms and cannot control the computer with the alternativ uh device but at Jabra uh my mission it was Implement one gaze correction why as you know in the computer when you are in a met uh in a virtual meeting we we are going to lose the eye contact because the camera is not placed exactly in the center of the the screen then if you remember most of

2:30:26 the time when you have uh one virtual meeting or or hybrid meeting most of the participant is looking down because the participant is not look directly into the camera but around the screen then what we miss we miss some engagement and that what I'd like to improve here to have better engagement between the uh the user to increase their attention then I'd like to uh uh emerge the the user more as a natural a natural conversation in the day by day as you have a person in person okay then the idea it is this is a normal video okay then I'm look around the screen and as you can see I'm not so engaged in in in that meeting what is the idea here that I can identify the I regions and the process on the I region to Recon struct the eyes what I have used here any video Mark scene this is amazing tool okay as you can see it's the same video as before but now I have more eye contact and see that I'm really engaged with the

2:31:35 person in the other side during the online meeting I brought to you like frame by frame and you can compare it you as you can see if you compare like the background and even the head movement it's exactly the same frame I only not I in that case Maxine construct like the eyes all right I have several problems we here regarding to the Maxin for example it's not possible to execute my Mac I need to execute this in a GPU and as you can imagine it's a very heavy process I cannot get the Maxine install directly into the camera because the limitations from the camera then what I've iMed here that because these work can execute even before the Maxine but the the focus it was executing this modu in real timeing okay because I had this challenge real time to execute here directly into the camera I need to think uh think about how to optimize the modu that it could be good enough to execute in in in our camera I need to also think

2:32:46 about diversity because I have sever I colors I shape skin colors then I need to implement one model that it could be adapted for any person that using our our camera and at the end it's looking for the integration how to integrate this directly into uh the camera and the video conference platforms I still have some problems because this is the original image but what happened if I even apply one gaze correction I can have one result like this okay it could be strange even if you use most of the the application noway you will face problems like this okay not only occlusion even uh uh different light is condition I can uh mention for you here most of the problems that you face in Comm um uh video based uh applications what I tried then it was looking for Implement again uh or warping and basically again can like regenerate the entire image from the eyes I have Advantage here especially in terms of the eye shape it

2:33:56 produce a very high quality shape in terms of the iris to reproduce the eyelids and to reproduce uh the eye image or can apply the warp that get the semi pixels and apply some movements in the pixel until I reshape the eye okay ideally here uh in this case it's better because in terms of color composition I can see clear that I keep like the same color in terms of the eyes in terms of the the skin color it's going to keep like the same like two years ago I started like my project by applying one systematic review and I found like this uh uh papers and these are the papers that fits better with our needs when we think about executing for Ed Ai and my paper my my my um model it was inspired uh in most of these techniques and the techniques itself here are inspired for each other because uh the composition from the model is basically the same sometimes CNN sometimes unet but you see is slightly difference but it's always

2:35:06 basically the same step one encoder uh one generator one warp and in the Second Step the color composition because I need to correct the color to bring back my eye image into the final result in term of jabber G correction model for AI as I mentioned for you the first step it will think about the speed I need to process one image in real time not only one but at least two images as I have the left and the right eye this should be executed in real time otherwise it not going to be useful or it's not going to provide a good use experience for my for our users the second one it was the eye shape because I need to reconstruct the eye but keeps the same shape in of Iris pupil eyelids okay and this one it was the best provided by the again the third one it was the color composition and this was the fight between the G and the warp because G provide for me better ey shape and warp provide better color

2:36:17 composition why I selected Warp because the result in my case I need to cut the ice using like media pipe to identify what are the Eyes located when I cut it and process and come back you will see some exactly the window that it was cuted all this area here then it was quite strange Al the quality it was quite quite good when I only compare like the image the process the input image and process one you didn't see any difference the difference appear only when it return back to the original pH then because of this I decide to use the warp based in terms how to building uh my gaze correction model I started by defining one data set all there are data sets available but as we think about uh using um in a company I needed to produce my own data set and it was a data set using synthetic and also real uh image then it was a long study comparing different implementations in terms of gain and warping uh then I

2:37:24 decided here that the one that we are going to use in our cameras it is the warping basic okay then training with the p lightning and Us in DVC to control the training Pro process and these are the two last step the one that I'm going to show to you it was the optimization before and now you will have the opportunity to test this because I deployed in both camera the look Onis and also um uh the Jabra camera okay the data set I used I'm not sure if you know this two that link is for the two it's not for my data set but I have generated here 1.5 million images okay then it's combination I create one specific user like then I move the eyes for 100 position and repeat this process until I achieve one point uh five million images the second data set it was a collaboration between Jabra and ifsp its Federal Institute of s Paul in Brazil then we collected data with 28 people and the best here because Brazil we have

2:38:36 diversity then we had a very huge diversity data set I don't remember how many frames but it was like 208 using two cameras looking for 35 position around the screen plus uh two times each camera and has collecting data for like two seconds then we had a very huge one and use uh images in in full uh HD then this is a high data set and I'm working on this paper and I'm going to publish this paper and the DAT set it's going to be available okay ah of course think about privacy here I'm going to also Pro like information about the face because basically you can have some interpretation what happened with the user face but the practice here the most important at least for my application are the ice region okay and all The annotation is going to be available for you as well I explained this before this is the pipeline if you compare with most of the papers there are slightly difference because some use

2:39:43 anchor map and then anchor map is basically get L marks around the IE leads to Define a new layer from your input image some of the some papers use the encoder others prefer to get the X and Y and create the tire to repeat this until concatenate with the original image you will see that there are is slly difference but in general like the architecture is going to be the same only two more slides these are some videos uh during my development process and as so you can see you can compare with the left one it is uh the one where I changed the eyes and the second one here it's the original image and as you can see most of the time the the right one uh it keeps like the eye contact of course it's not the sem quality as Maxine but please consider that I'm execute this in camera and this is a lower resolution image okay I can also combine with different models that we have in the company like purification okay I can get like

2:40:53 apply uh the G correction plus the purification Al of my mother say that the left image is more beautiful I cannot consider this but let's see you can see that I'm younger in this image we have different models that we can combine uh inside our experience okay the next one Elizabeth

2:41:29 it will be a rush presentation as they talk a lot I don't to go faster yes then and in general when we talking people about hand gers they are a bit skeptical because normally they think it's only four games or something like that but as we can see we have some uh Global reports that show this is a really huge Market that is growing faster and a lot than maybe it's something good to think about it's to develop something regarding hand gestures recognition and also tracking according to to the reports from these companies the marketing is growing most because of the Internet of Things device and also because the improvements from artificial intelligence and machine learning models that as we can see we can deploy also for AI and it opens a really huge Market to develop new ideas H and new products the main segment is SE and human human computer interactions that cover actually cover um almost all the

2:42:52 the products in general but virtual reality in gaming is also uh to Big um markets to explore um then why hand gors hand gors are everywhere not only in Virtual um games in virtual reality we can see different applications from the consumer um from people personal devices even for The Big Industry we have a huge of hand gestur applications for really serious application let's say something like that not not only for um entertainment then also from new auton Automotive for example uh using for interaction in general it's something that is really new and it's growing here I bring some um I find some examples of some commercial applications that uses also hand gestures or gesture recognition or gesture tracking or hand tracking that we can see on Market I don't know if you know all of them but for example my next acquisition probably will be this selfie drawn that is one thing to have and it's very interesting

2:44:22 it it works with hand Gest recognition only H just not for um control the camera but to start the process of using the the Drone to run away and also to take pictures then we have tracking also recognition this is example of interesting product that uses hand gester in in in commercial products uh the best thing about hand gestur I think it's that because it supports multimodality that nowadays is something really uh import important uh when we can integrate with other input methods for example we can integrate hand gestures with voice or with ey tracking and have a more versatil product when you can integrate those different kind of interactions then H with the multimodality we can start thinking a bit more and exploring the perceptual human interaction as Fabo told something about that is when the environment and or the devices are really perceptual with the the human what they are doing what they need what

2:45:38 next like contextual uh environments and it is it's something really interesting the hand hand based technology in general can be described normally in three kind of functionalities depending on on the application in the needs of the application what you can do you can use the only hand detection that can be enough for some uh kinds of applications for example Museum interactions for panels this kind of stuff or you can use hand tracking when you can you need to follow up the the movement of the hands what they are doing or like manipulating some kind of 3D objects this kind of stuff or then using handy um gesture recognition that is regarding to the gesture itself that can be a POS or a Comm for something that happens in the interface or you can combine more than one of those H functionalities let's say like that uh bringing the PIP the pipeline process from the literat of course you feel a bit of creative

2:46:51 Liberty here H some steps could be grouped or understood in different ways but here we can have a bit in general how the hand gesture recognition process occurs in a normal application like not exactly from Aji but we can we will see the differences from our general process from the aji in some minutes then for example from the data acquisition for interface feedback it can runs in the same computer or the same system or it can be distributed in different ways on depending of the always depending of the application I think is the best thing to describe any kind of system presented here then looking at the steps the data form for hand Gest recognition can be obtained from different types of sensors um that can be uh for applications like with hand gestur for St static gestures or for dynamic gestures H my focus is more on video based application as my research is applied to jabber collaboration products

2:48:10 as well and like for professionals camera professional cameras for virtual and hybrid meetings I'm trying to discover different ways of interacting with functional of the camera using hand gestures then I will I will use like the recognition of the gesture pose it could be static gesture or can be a dynamic gesture on depending of the H which kind of action I need the user performing on the camera we are exploring this right now uh the preprocessing for image based application we follow some uh different steps depending on the application we can have here some suggested steps that can be applied like different techniques to reduce the process time and the costs uh for the next steps probably normally we cut the image or size or identify the shape this kind of stuff to The Next Step be processed then uh the future structure step involves identify and struct different attributes from the data that

2:49:25 you be different according to the sensor and what you are doing H with the processed data H then here I see some common libraries or Frameworks that are normally combined together to extract this data according to the needs uh probably they well know here I don't know if someone works with hand gestures in general it's media pipe not not only for hand Gest but also for the body in general that uses geometric for example to H identify the the the pipeline of the hand gestur the hands General um this step is where the inference happens that uh a prediction classification or decision it will happens according to the the Unseen data inputs that you be process it some Frameworks are more recommended to static adjustors and then others are more related to the dynamic ones and this typ of inference model will be in different um location depending of the application in Ed AI it will be on device or in regular application it can

2:50:42 be in a server or it can be in a cloud on depends of the application again yeah yeah the postprocessing and gesture interpretation can be uh refined after the inference that the result can be like reclassified or that can be mapped in final final actions or Final commments in general H last but not least uh here is the feedback that the feedback step feedback step that is what the user is expecting H in in head in user head is what they paid for and what they need the best H feedback ever then like uh this is the important part normally it's another kind of team not the developer team that thinks about the user experience the user interface the user interaction and the result process how it will be showed to the user um but this is a think here that it's important to think and regarding to hand gestures it's important to show to the user what is happening if he the gesture was recognized properly or not because

2:51:59 sometimes the interface can be a bit stressed uh for the user here just H bringing and everything to the The Edge what happens uh probably in this this part the data acquisition and the interface feedback it you be together all together different of cloud AI or another kind of developing using servers for for example local Serv or whatever um how the steps are distributed here then the same at the same place and this is um how it works in general it will be dependent of the camera itself how is the power of the camera and all the the characteristics that it has how we are going to divide it sometimes it's better to use a kind of distribute architecture not exactly cloud cloud ji but we can like develop in different steps uh much progress has been made in general about Hest recognition over the years but some uh technical challeng is still remaining uh for example the data set not a problem but I I think that

2:53:18 you'll be more a challenge itself that is something hard to get new data even now because of DPR as you know it's diff it's hard sometimes to get new data that is focused on the application you really need um it's hard to get this data to training a new model for example we can we can see a lot of different data set available from internet but normally they will not cover exactly your what you need for your application then probably it's something that is really hard to work with it's found in new data H we have some collaborations with Institute of Technology from s Paulo to collect also a data data set of Hest for This research because in general I'm working also a bit related to which are the best gestures for the application because uh when you talk about um hybrid meetings for example it could be works very well in general the processing Etc the EDG model but if the gesture itself the pose I mean is not a good a good um

2:54:38 POS for all the people that I'm going it's not a good a good choice for for example to apply like we have people from all around the world interacting at the same meeting room for example and it's something uh related to okay this gesture me has a meaning for me but for people from another region there is another meaning then doesn't matter if I have a good a a a good designed process for AI and if the interaction process was not well thinking about when I think about the vocabulary of gestures that I'm going to apply then the gesture vocabulary is nowadays a problem when I think about there is no standard about hand gestures in general for example you can see different kind of devices doing the same action but one brain uses one Gest for example this one and another brain can use these Gest for the same action then this is a huge uh challenge like propose my intention is like propose a kind of study that can create

2:55:56 some standard regarding um vocabulary appropriated for this kind of products for meeting rooms in general Jabra for product collaboration because of course the gesture set depending of the context also depends of the application the pends of the device but also the context of use and we are going to discover this kind of standard if it exists or not how does it work and why this is a problem because and this is um hard to solve it's because we have a cultural problem not not a cultural problem a cultural characteristics between people the people H know something that they H are learning from another device sometimes they know something that he watch in a movie then this kind of stuff it concurs with the Cal that they have created with and when you think about the Cal is not related exactly to the country exactly you know there are a lot of um characterists that cting this Prisma of uh cultural meaning

2:57:14 interpretation Etc then another thing related to the Cross cutting problems that is a kind of transversal challenges is uh how can I how how can I discover that the user is understanding that gesture that they are going to use in the best way that you have a good um interface with that gesture sometimes we think that we need to to count with the user intuition but sometimes it's not enough sometimes we need to teach educate the users about some gestures then uh probably in general in my area ofation that is human computer interaction more related H it's really expected that we can count with the intuition of the user but sometimes it doesn't work at all and because of this the fluidity that it's like the flowing about the user even more related with the modality when you can use it combined characteristics combined um um modalities of interaction the same the fluidity like it's which works

2:58:31 properly and really natural in between for the user it's something really hard to reach maybe you have this experience with something um related then um for those who want to get a look I have applied Ed AI in a camera that is this this one it's look son One Max that is the same architecture from java the same and we use miror de BL format because actually it's a model that is still available on internet and it's already er um how can I say condensed and properly applied for the model and we use the media pip as pal detector in landm market then you can try after the the section all of them actually yeah that's it now Shen

2:59:49 how you guys doing first day of cvpr and it's already 540 I hope you still have energy for one last section so I'm going to be focusing on sound as a modality as they mentioned before at GN we are trying to look at multimodal perception and how we could the people at the far end could feel like being in the same room as where the camera is recording and that's where we are looking at audio or sound as a modality so what was the motivation behind it uh we have the video we have an understanding of what the person looks like on the camera but we also want to know who's speaking at what time and if it is a bigger room and lot of people are speaking at different times the person speaking last is given as much importance as possible and that's wherein we are using uh modality of sound to make sure the person who's speaking is given as much importance and uh for to do that today I will talk first uh what sound is as modality and

3:00:56 what are the different audio features as it is a computer vision uh conference I would try I will explain the different features that are commonly used in audio and then show you a completely audio based model that we have deployed on our panacast p50 and as well we will I will also show some purely multi model models that we are looking into so what is sound sound is uh produced by vibration of an object as you can see in this where isor as you can see in the small uh animation that I have created the air molecules that are oscillating over time that creates this compressions and rare refractions and this is a simple uh animation of what it looks like as a sine wave so this is basically sound and it needs a medium to propagate and that is air now how do we as you can see in the video uh sorry animation before it is a continuous system it is a analog sorry analogous sound is analogous but when we

3:02:05 have to do it on a digital domain we have to do it in a digital Way by not being continuous but discrete and that's where sampling comes in sampling is capturing and converting audio signals basically into a digital domain and it is quite similar to I would say how pixels more number of pixels in an image gives better quality similarly higher sampling rate or higher sampling gives a better quality audio now this is a simple representation of uh an audio in the time domain as well as in the frequency domain as you can see this part those are the where the amplitude is increasing and when it's uh there's no amplitude it's a flat line and this can be converted into the frequency domain by a simple mathematical function called The Frequency fast Foria transform and this is the representation in the uh uh frequency domain so now we have time we have in the frequency domain but still in reality we need a continuous we know

3:03:11 when the frequency what the different kind of frequency is available at a particular time but we don't know how is changing over time that is where we create the spectrograms so spectrograms is basically a representation of audio in a Time frequency domain so we have the full waveform over here what we do is we divide these into chunks and each chunk is a frame and that frame is similar to how you have in uh video so we have different frames and you choose the frame size and then next thing you do is the window so now that you have a chunk like that at the end you can have you window it down so that there is less spectral leakage and then you do how you decide whether you want to jump completely or how much overlap should be there between different frames and that is the Hop size and finally you get this image at the bottom where which is representation in the time frequency domain but

3:04:25 perception of sound is quite different for humans as it's shown for example if the frequency between 210 frequency uh Kilz and 220 khz is not easily understood by humans sorry is easily understood from 200 to 210 but from 2000 to 2010 is not that easily understood this scale change in sound is quite uh logarithmic and it's not really uh perceptual of perception of sound is quite different for human for the human ear so that is where we use something called as a Mel uh Mel scale which converts the whole spectrogram into a Mel fre sorry Mel filter Banks and then also does a logarithmic scale on it and we get something called as the log M spectrogram this spectrogram is using a linear scale and it's also using Mel scale which converts a spectrogram and if you can see on top I have the uh uh stf spectogram and down is the log Mill spectrogram as you can see because of that we can have a better

3:05:30 much better representation in the lower frequencies and more more better understanding as humans because the logarithmic scale is taken care of and now this gives more information and it is for obviously our models can understand better the representation of sound

3:05:54 and then the next uh step is once we have the uh log Mill spectrogram and another feature that is commonly used nowadays especially in spe speech recognition is msccs so it is once you have the waveform you get these log mil spectrogram and the next you come to the discrete cosine uh transform and then you get an mscc what mscc does is it's very good at representing your vocal tract and especially your speech and that it also reduces dimensionality quite a bit that only the important parts of the whole logmill spectogram is remaining and that's what helps uh the models learn better about speech

3:06:44 recognition that's all fine and Dy right but we have a lot of this mathematical Concepts and mathematical stuff and the Transformations but how do we do that that is where this package of librosa comes in you can think of it like open CV but purely for audio all you you have librosa dolo similar to CV2 imre you have the whole uh displaying of a wave show is as simple as a single line and you can also get the whole mfcc just by librosa features for mfcc you don't have to do the whole spectr log Bill spectogram all that math is dumbed down into one single line of code using

3:07:29 liosa but now this is the one model that was uh deployed into our panas p50 model which is purely audio based the original model was taking Azimuth is this angle like this and elevation is like this so what we have is a 180 degree field of view along the x-axis and we divide that 180° field of view into 37 different uh parts and uh from minus 90 to Plus 90 we have all the the divided into resolutions of 5 degrees each and that's how we get to see uh which if a sound is coming we have two different probabilities one is to see if it's speech or not and to probability to see which part part of this resolution is the sound coming from and for this I explained the different features because over here we use spectrograms and this is what was used for training the model and now this is the next model that we looking at now this was complete the previous one was completely audio based so this is sound localization

3:08:38 which is using both audio and images so this is the paper which we looking at it has an image encoder for clip and audio encoder called we to clip and what it does is the uh embedding the all the it is put into a single embedding space and it learns how to put uh the similar pairs into uh closer and the non-similar pairs more further apart and that's how it contrastive learning is used to uh to see where the sound is actually coming from and this one uses raw audio wave for 1 two three this is a test 1 2

3:09:27 3 as you can see over here we are looking at where exactly the sound is coming from and what are the which even the snap of a finger can be localized and we get to know okay this is the area of interest from where the sound is learning from and it's a completely inent model that is learning this contrastive way but now we need a more different approach one of the big difficulties we had with our DOA was the further away the people are then the resolution is smaller it makes it more difficult to see who's speaking at what time we already have an inbuilt B which Anu helped with the head body detector so from that head we get only the cropped faces of the head and we get the audio input we convert that into mfccs and we have the visual encoder and audio encoder and together we can get an active speaker detction we can see if this person is speaking or not hello one two three checking checking can you hear me

3:10:29 localize hello hello so this so this as you can see is better at seeing speech because the mfccs they used and the lip is being used to be able to see if the person at present in the bounding box is talking is doing active speaking or not this is from one perspective that was both of these videos were taken in the panacast P20 now I'll show you a video which was taken in a video p50 with a lot of people talk uh a lot of people present in the room and people talking uh but it's a it's a one to one yeah and if I am not mistaken I think the cost or something like it's in thousands to do the whole thing but it it is a good experience but how how long would it take them from being a one to one where is a group to group yeah we had a opportunity so that was a brief demo of the active speaker detection that we have been looking at and yes so this is more in the next Pipeline and hopefully similarly we'll be having this deployed

3:11:37 in our next products which we'll be hopefully showing next year also and uh we have after this the demos and questions we can take right yeah thank you then fact it's only to finalize here in terms of demo we're going to set up now uh at least this four demos the G correction where you can test here body segmentation hand gestures uh recognition and also the sound location it basically we have all the time that can you can try and test and ask more questions it's only one small uh information regarding to the Jabra collaboration business as you saw we are different research work in different projects we sometimes collaborate with each other but we have one specific project each of us the moment that we finish the project we get the uh uh the model uh the best mod that we have and we deploy for this experience then the experience sometimes we don't have conent inside the Jabra if the name is

3:12:50 XP or XP then we still decide but uh I usually ask talk this uh name this as xpy and the XY it's basically on experience once you have the module one interaction module ready we create a filter and this filter as you can see it's a pipeline similar to the one that you saw in the camera then it's only connect like as you can see here this is the person segmentation this is the one it was implemented by narcy and it has some input and output and we can combine different interactions then I can also demonstrate for you this but this is not public this is a private us it in our vide Tech at least we our Market team can use the experience that we implemented to have meetings with other companies then uh in fact as I said we are like like uh 24 minutes behind but now we have plenty of timeing to to answer your questions okay regard to uh the second part of this presentation or the first or the first

3:14:02 yeah or the first yes feel free to ask a question already yes sir um so what we want the model to learn as much as possible from the audio features and uh that get that can be done by mimicking the human perception as much as possible so logarithmic human pitch we perceive pitch on a logarith logarithmic scale whereas the foror transform is on a linear scale so converting that into log uh Mel spectrogram helps understand better in that for a human P perceptive level and having the features in the human perception level helps the model also learn

3:15:15 better I mean the whatever uh models and stuff that has been trained on it has been shown that the better features or better especially with speech recognition when you get mfccs the model learns a lot better and the performance of these models are also improving so it depends on your use case so I'm explaining the difference between spectrogram log Mill spectrogram and mfccs but all different all features are used for different models for example the we used only spectrograms on our DOA but mfccs was the use for the active speak detection and raw audio waveform was used for the sound localization so it depends on the use case in in itself but because uh speech recognition and the talking of a person is much more best mimicked with an mfcc that is what is used

3:16:21 uh in the sound localization one model you're asking yeah so the image encoder uh has the embedding space and the audio encoder has an embedding space and the contrastive learning they bought all the uh embeddings are brought into a mutual space where in it is learned to uh differentiate between by contrastive learning to say if it is me talking and my voice then bring those linear linearly closer using cosine similarity and if not bring them further away using cosine

3:17:11 similarity machine Lear for a noise reduction the it's really big resarch with the because maybe I can add perspec also yeah for the Audio I can add a little bit so the model architecture that you saw there was a video aspect of that there's a video incoder that's uh basically a clip encoder given an image uh you get a certain kind of embeddings out of that then you have an AUD audio encoder given an audio sample you get certain set of emings now we uh apply a cross attention layer on on top of that so what that does is basically it's attending to both audio and the uh video features in sync and trying to figure out that where the audio is coming from so it's basically trained end to end like that so that's how you can get both the person uh location as well as the audio information combine them together and learn and attend to both of those and then based upon that you can say okay the audio is maybe coming from here so

3:18:39 that's how the training was done for that I guess like if I'm understanding this correctly so we where you have the heat map as to where that inference is actually based on your video as well as your audio yes both the things are both the things are in parallel you require both audio and video yes information solve the one problem because two years ago we tried to identify

3:19:16 why because we can like the without need to and why because if didn't had like this sound someone the background so knock the door any kind of situation if they use only the in of the movement because still it's only part like

3:19:53 that we Tri still for it's like moment how much it takes now it takes almost one second but we' like to reduce this for let's say 10 milliseconds or then we can also have and depends on the perception as well watch made basically like noise that the and you like that provid possibility

3:20:40 topec yeah so another issue with the lip B system is that uh if you are just sitting in the front of the the camera you have plus - 30° field of view where it will work after that you'll lose the lip land marks so it won't be

3:21:02 scalable are ining the aim and elevation yeah so no no the original paper does asimuth and elevation in our model we are doing only Asim yeah only as only as because most of for us use cases everyone is more is sitting so we don't elevation does not recreate that much of a difference it's all mainly as so what do you mean when say you are using the as so you have a micro like One mic we don't have one microphone the panas 350 has eight eight microphones that's what was the first line which I think I didn't mention okay so we are getting spectrograms from all the microphones okay and they are distributed in a line yeah and we have the elev sorry asth fromus 90 to 90° and the prob we have divided into 37 sections with 5 de resolution each so these microphones are giving the feedback from which able toate yeah yes and even on the microphones you have beam forming and stuff happening so it helps with the better voice pick up

3:22:02 and reducing background Nos and things like that so that makes it more cleaner before reprocess yeah m I mean in the end it is similar if you see a spectrogram or a log spectrogram or mfcc in the end it's like an image right and we have temporal aspect because we getting each frame each spectogram of each frame so in the end how you would do with computer vision for image or s a video it's a similar uh pipeline that would be for audio as well so basically once I convert my audio signals from raw audio to ffts to M spectrograms it becomes like a image that captures both the time and the frequency domain as well as the harmonics that you see as the high and low level values now it becomes like a image classification problem now I can train like a simple convol neural network to classify that what this audio is for example Bird versus human versus something else or some other kind of things and I can use it in regression

3:23:11 classification other kind of tasks so it becomes a image classification problem or image regression problem basically instead of a Time domain based frequency domain based thing that's that's what this is helping us to to my research yeah resar

3:23:44 on yes so just whatever Shan told you here yeah so whatever Shan told you in terms of the presentation the steps it's basically what you have to do same thing yes oh Glass Walls yes okay so so that is still patent bending but okay uh it's basically a simple kind of an algorithm uh we don't have we can't run depth model anything so basically what we are doing is in a very naive way what you can do I'll tell that then um if you know the box sizes and if you know the ground Ro distance kind of So based upon that if you have different people at different distances from the camera and if you know the bounding box sizes you can collect that data and you can train as simple as a linear regressor and that will give you the estimates of where the people are and now uh for each of the people who are outside the boundary you can tell that okay this is above the distance from the camera and reject them so basically

3:25:10 it'll eliminate like move them out of the boundary in very basic sense that's it it's a little bit more complex on the system but it's pent pending so I cannot talk more about the details yet we have to close sorry yeah so we are analyzing depths but that's like a offline parameter we are not running any depth model we don't have the compute on the existing ones it's a offline process if we can have one last I think the guy is asking us to vate to leave stay here but we we hereis our cameras work on USB laptops we can just put it outside yeah we can yes please yes question

3:26:10 conm or you just ensure that the output for every frame is no the loss Remains the Same it's just that how you pass in the data set so uh in image based processing your um input Dimension is NCW now uh bat size Channel height and width now when you have the temporal aspect it becomes NT chw where you have the time s which is the number of time frames that you're taking and then during model training you multiply TNN and that becomes your bigger bat size and that's how you feed in frames continuously to train your gr model layers yeah yeah yeah uh for just the gr list but it trains the whole model like that then on videos if you have okay thank you all uh yes for be part of this we're very satisfied with the final results but to have more question we will be here outside and let's see how we can manage this thank you I will upload today yeah tonight in this in our website it's going to be

3:27:20 there okay thank you thank you but your backone is bigger until hi okay yeah thank yes please yes yeah let me add you because I don't have I saw you gu I forgot just yes connected but we can Hello nice to meet you

3:29:03 because we are open for collab we are open for research for publication wece we have patents of course we have more secrets that we but we have part that it's inspired in another research I cannot it's not a secret about for example because I was inspired from different applications the only thing that made different was get a very big model and deploy into this hard level and Bas it's like to show us as a research team okay although we work a private company yes but for example I was a teacher in a university I got like funded for her scholarship she's is my student at the univers and work with us the company then we have a perspective of researching as well and basic because of this is because we tried doing similar

3:30:19 also withing didn't know that what I'm here it's for this you didn't know that is denark most of people didn't know about the Jabra maybe people doesn't know about the I guess adver as well it was a lot of money because as you can see the team is quite also Leon it's not part from this group but also from ja this the that we and and second part

3:31:10 is our website it it yes I forgot to mention because I saw people taking photos okay it's nice then I'm I noted I'm not going to let people but make and finally the last one I'm from Intel so then you know a lot

3:31:42 about no because it is the one in our when I we are going to we also have of then it's something intern we only follow so basically you had it it was justment after doing this whole thing I'm not sure how this

3:32:39 St we have different experience that we could show here we only brought like because Elizabeth now for example Today weed an email from AG C and she received from the best paper and she's G receive reward in first of

3:33:14 July yeah let's go I'd like to stay here for more time sorry for that head out to then FAA oh yeah thank you narcy thanks guys bye yeah bye bye no I'm not I just want to say a quick thanks oh

Questions about this episode

Who speaks in "23714 Edge AI in Action Practical Approaches to Developing and Deploying Optimized Models"?

Shan Ahmed Shafi (Speaker / Team member (audio & multimodality)).

What is discussed in "23714 Edge AI in Action Practical Approaches to Developing and Deploying Optimized Models"?

In order: 0:00 GN, Edge AI, Jabra, Panagas 50; 15:00 Edge AI; 30:00 P20; 45:00 Panagas 50; 1:00:00 Qualcomm; 1:15:00 Apple Inc.; 1:30:00 Qualcomm; 1:45:00 Qualcomm, Panagas 50; 2:00:00 Onyx, DVC, Gaze Correction; 2:15:00 Gaze Correction, Edge AI; 2:30:00 Gaze Correction, Artificial Intelligence.

What is mentioned in this ComputerVisionFoundation Videos episode?

17 people, works and ideas are mentioned, including Edge AI, Jabra, Artificial Intelligence, Qualcomm, GN, MedEx. Each one on this page links to its own page with every episode that mentions it.

How long is "23714 Edge AI in Action Practical Approaches to Developing and Deploying Optimized Models"?

The episode runs 3 h 34 min; the transcript is about 34,149 words in 213 timestamped sections.

More from ComputerVisionFoundation Videos

• All analyzed episodes
• People mentioned on ComputerVisionFoundation Videos
• Topics mentioned on ComputerVisionFoundation Videos