Transcript

Ultra Ethernet for next-generation AI and HPC workloads

From Microsoft Research · Episode page · Watch on YouTube

· 1 h 56 min · 22,837 words · 164 timestamped sections

This is the full, timestamped transcript of "Ultra Ethernet for next-generation AI and HPC workloads", an episode of Microsoft Research published October 17, 2025 (1 h 56 min, about 22,837 words). It mentions 16 people, works and ideas, including UltraEthernet, Torstton Hoffler, Artificial Intelligence, Sarab, Azure.

What this episode is about

UltraEthernet is advancing as a critical standard for AI and HPC workloads, highlighting Microsoft’s hardware-focused progress, system architecture planning, and collaboration with academia and industry to drive the platform shift in AI infrastructure. The discussion covers leadership, ongoing development, and the strategic direction of UltraEthernet within Azure and broader research efforts.

Key takeaways

  • AI-centric data centers: expansion, efficiency, and AI workloads redefine scale; extraordinary capex signals strategic bets, not mere hype.
  • Middle innings metaphor lands: early chaos fades into decisive, incremental gains in performance, reliability, and cost per AI task.
  • Diversity pays off: mixing AMD and Maya architectures buffers risk, accelerates innovation, and avoids vendor lock-in for ML workloads.
  • Liquid cooling shift matters: megawatts moving from air to liquid unlocks sustainability at scale, with tighter operation discipline.
  • AI networks as a bottleneck focus: inter-data-center AI fabric (AI Van) turns latency into global throughput and resilience.

Who speaks in this episode?

What is discussed, and when

  1. – 10:00: Artificial Intelligence, UltraEthernet, Abdul Kabani, Satya
  2. – 20:00: Satya
  3. – 30:00: UltraEthernet, OCP, Artificial Intelligence, Azure
  4. – 40:00: UltraEthernet, Azure, TCP
  5. – 50:00: UltraEthernet

Notable quotes from "Ultra Ethernet for next-generation AI and HPC workloads"

  • Hi there. I'm Jacob Nelson. I'm part of the systems research group here at MSR and I'm delighted to welcome all of you to this mini workshop on UltraEthernet today. we're very…
    about UltraEthernet · at
  • Afternoon good evening if people are joining from across the world. I'm Sarab and like I said I lead the systems planning and architecture for Azure hardware systems and being doing this for the last seven to eight years.
    about Azure · at
  • One is that we are really being an AI first data center. Like the data centers that are being built are designed with AI in mind. And if you look at one of our you know AI data center that's.
    about Artificial Intelligence · at
  • Today and we'll be running CH GPT on that infrastructure itself. But along with that we are continuing to focus on diversity. So we continue to deploy AMD and our Maya systems.…
    about AMD · at
  • Use that tool chain. And what we also have we have and this is now getting towards Ultra Ethernet. We have a tracer and a simulator that is just published.
    about SC24 · at
  • Give you one number actually stole this number from another friend Rambalaga at Broadcom so he gives us this number at 2023 in OCP 600 million ports Ethernet ports are shipped annually okay so Ethernet itself is actually a.
    about OCP · at
  • Traffic we cause as humans are Ethernet frames. But there are very different ones like TCP IP which is what we are using right now to communicate and there is Rocky RDMA over…
    about TCP · at
  • Which is what Google does or what we at Microsoft in DCQCN or prior to that in DCTCP has done we took a very good combination with all the lessons learned and put them in a very meaningful manner.
    about DCTCP · at

Who and what is mentioned in this episode?

People

  • Torstton Hoffler — Person · Main speaker (academic + Microsoft consultant; UltraEthernet technical presenter) · 5 mentions
    Delivers the deep technical overview of UltraEthernet (UEC): motivation, workload classes, transport design (lossy/best-effort, out-of-order delivery), load balancing, congestion control framing, and security model.
  • Sarab — Person · Main speaker (Microsoft/Azure executive) · 5 mentions
    Introduces Microsoft’s AI-infrastructure priorities (datacenter expansion, liquid cooling, networking) and frames UltraEthernet as a key platform shift for AI infrastructure.
  • Abdul Kabani — Person · Main speaker (Microsoft network hardware architect; congestion/load-balancing presenter) · 1 mention · first at
    Presented as a principal network hardware architect and later takes the stage to explain UltraEthernet transport congestion control and path load balancing mechanisms.
    Professor at ETHZurich as well as a consultant for Microsoft. I've been a fan of his work for years. and Abdul Kabani is a principal network hardware architect with a long…
  • Satya — Person · Referenced executive (vision/strategy setter) · 2 mentions · first at
    Cited as setting Microsoft’s mission/vision for AI infrastructure scale and cost; referenced for the “middle innings of AI” framing and “world’s computer / AI supercomputer” narrative.
    But I think one thing that really stands out is as Satya has laid out this mission and vision for us to deliver the highest scale infrastructure at the lowest cost and driven…
  • Laura Anne Nelson — Person · Host / workshop moderator (introductory speaker) · 1 mention · first at
    Opens the mini-workshop, identifies himself as part of Microsoft Research (MSR) Systems Research Group, and introduces the speakers and their roles.
    Hi there. I'm Jacob Nelson. I'm part of the systems research group here at MSR and I'm delighted to welcome all of you to this mini workshop on UltraEthernet today. we're very excited about.

Companies & orgs

  • AMD — Company · Referenced company · 1 mention · first at
    AMD is mentioned as part of Microsoft's diverse silicon deployment strategy.
  • ETH Zurich — Company · Referenced institution · 1 mention · first at
    ETHZurich is mentioned as an affiliation of a consultant for Microsoft.
    Is the CVP who's responsible for system planning and architecture in Azure hardware. Torstston Hoffler is a professor at ETHZurich as well as a consultant for Microsoft. I've…

Products & tech

  • Azure — Product · Business focus · 2 mentions · first at
    Azure is Microsoft's cloud platform, heavily involved in AI infrastructure and UltraEthernet.
  • GB200 — Product · Referenced product · 1 mention · first at
    GB200 is mentioned as part of Microsoft's infrastructure for AI workloads.
    Is the silicon and the rack infrastructure. We were the first one to go into the world and the fastest ramp of the GB200 infrastructure which is now leading the performance and…
  • GitHub Copilot — Product · Referenced product · 1 mention · first at
    GitHub Copilot is mentioned as an example of AI workload scaling on Microsoft's infrastructure.
    With it and the ability to now build a holistic you know infrastructure that canonical workloads like Chad GPT or even GitHub copilots and all of these that are scaling can…

Concepts & topics

  • UltraEthernet — Concept · Core topic · 6 mentions · first at
    UltraEthernet is a new standard being discussed for high-performance networking in AI workloads.
  • Artificial Intelligence — Concept · Core topic · 3 mentions · first at
    AI is the central theme of the discussion, focusing on infrastructure and network requirements.
  • TCP — Concept · Referenced concept · 2 mentions · first at
    TCP is mentioned as a foundational protocol in networking discussions.
  • DCTCP — Concept · Referenced concept · 2 mentions
    DCTCP is mentioned in the context of network protocols.

Places & events

  • OCP — Event · Referenced event · 2 mentions · first at
    OCP is mentioned as a platform where Microsoft discusses its vision for AI infrastructure.
  • SC24 — Event · Referenced event · 4 mentions
    SC25 is mentioned as an upcoming conference where new research will be presented.

Full transcript with timestamps

Hi there. I'm Jacob Nelson. I'm part of the systems research group here at MSR and I'm delighted to welcome all of you to this mini workshop on UltraEthernet today. Um we're very excited about uh everything that's happening with UltraEthernet and all of the uh input that Microsoft has had in driving this new standard. uh and so I'm excited to welcome a number of people to tell us what Microsoft has been doing and and how the standard is progressing. Uh Sarb is the CVP who's responsible for system planning and architecture in in Azure hardware. Uh Torstston Hoffler uh is a professor at ETHZurich as well as a consultant for Microsoft. I've been a fan of his work for years. uh and Abdul Kabani uh is uh a uh principal network hardware architect with a long research career as well and so with that I will turn it over to Serb to get us started.

Good. Thanks. Thanks. And I do have a mic so hopefully people can hear me. And uh so firstly good morning and good afternoon good evening if people are joining from across the world. Uh I'm Sarab and uh like I said I lead the systems planning and architecture for Azure hardware systems and being doing this for the last seven to eight years. So when uh Torst and Abdul kind of asked me to come and say a few words I said oh this is really an exciting time and I I'll take any opportunity to talk about this you know tectonic shift or the platform shift that we are seeing in AI.

Uh Satya likes to call it I think he said in build it's like the middle innings of AI and uh being a Formula 1 fan I always reflect on what does it mean and if people watch races they always say the first lap is quite chaotic like it's like there's a lot of drama happening around just the first corner and then when it gets into the middle stretch if you ask any principal uh uh team lead he'll always say you know I think races are won in the middle inning sometimes and it's really uh you know They may not be as flashy, but they're more decisive. And um here we are in in kind of the middleings of AI and sitting in the infrastructure side of things does really feel that you know we moving from really now delivering consistence, high performance but also you know lap after lap giving the performance, the quality, the availability, the efficiency and doing innovation at the same time. This is really the middle innings of AI for

AI infrastructure and I think I just want to go through some of these shifts like what's happening under the hood maybe in a few minutes before I hand it off to tors and Abdul to go into the details of you know the technical talk but I think one one thing that really stands out is uh as Satya has laid out this mission and vision for us to deliver the highest scale infrastructure at the lowest cost and driven by the workload that you have to fall in love with which is AI today Right. And and so he just we just announced about what 2 gawatt of data center expansion in the last year alone. And that's more than what we had in Azure just three or four years ago. So significant expansion going on $80 billion of capex being spent and what's happening under the hood is is really around you know three or four macro trends on infrastructure.

One is that we are really being an AI first data center. Like the data centers that are being built are designed with AI in mind. And if you look at one of our um you know AI data center that's going to perform 10x the the performance of the fastest leading super computer. But it's not just AI, it's also the compute and the storage that goes along with it and the ability to now build a holistic you know infrastructure that canonical workloads like Chad GPT or even GitHub copilots and all of these that are scaling can truly lead and deliver the experience. So that's the expansion piece of it. Another expansion I would say there are three layers that we're continuing to focus on. First is on you know the unit of deployment which is the silicon and the rack infrastructure. We were the first one to go uh into the world and the fastest ramp of the GB200 infrastructure which is now leading uh uh the performance and

benchmarks of everything that we've run today and we'll be running CH GPT on that infrastructure itself. But along with that we are continuing to focus on diversity. So we continue to deploy AMD and our Maya systems. And that's the first layer of the silicon and the rack infrastructure that we're building. The second shift that's really happening is around you know the liquid cooling and the sustainability efforts that goes along with building such you know high massive expansion comes with a lot of responsibility and so we're seeing a massive shift of you know megawatts moving through air to liquid over the next few years and how do we kind of continue to operate that at the highest scale but at the same time with the you know best level of efficiency and operational excellence is is the second shift and the third one which I think the if the crowd here is really want to talk about is the networks right I mean

we've got clear you know networks that are being beating heart of the infrastructure today and whether it's within the data center whether it's outside the data center we have announced the AI van which is one of the most highest performance scalable AI network uh connecting multiple data centers together so that the job cannot just be run in one data center but is interconnected across multiple data centers we have work going on inside the data center itself with both on the front-end networks as well as the scaleup networks. So I think the network aspect of it is is is kind of one of the third and the most exciting personally to see some of the advancements going on. So not only are we delivering on the diversity of silicon we are doing this massive shift towards air to liquid and the operational you know excellence and rigor that goes with it and then we are reimagining the networks for this AI

workload. Now UC I mean I've been following the journey for UC for the past uh few years personally and just uh you know the need for something like this has been so critical right now especially when we look at you know the legacy of the Ethernet and the changing requirements that AI brings in and the principles that UC was established on are kind of foundational to this innovation. the principle of delivering high performance that you need. This is the backbone of the network that's going to happen and the scalability that goes with that performance but at the same time you're building on top of a known Ethernet stack that can make it interoperable that can make it open for all of the you know companies to innovate on. And then the third piece of it is the economies of scale. Right?

That's so again those three design principles of high performance and scalability, the openness and interoperability of Ethernet and the ability to drive the economies of scale and the ability to use a lot of the learnings we have on Ethernet to be able to then apply them to this next generation workload. That's what personally is exciting to me. So, so again, thank you for all of the work that this community has done to do, you know, publish that first 1.0 spec. I'm sure that that itself is is is a lot of u exciting innovation that are going to help us in years to come. And I think there's some great work going on in the road map as well as we look forward to how uh the UC and the Ethernet um ecosystem stacks up for AI. So again, thank you Tristan and Abdul for bringing in. Thank you everybody for you know participating in the workshop and I'm uh excited to hear how the rest of the day

goes. So Ten, I'm going to hand it off to you. Thank you. Thank you. Turn this on.

Okay. Thank you very much Saraf. I'm I'm assuming I'm audible um online as well. And so this was a wonderful introduction just framing the idea and what we are doing and and what we are doing at Microsoft but also what we are doing specifically to Alra Ethernet. Um, by the way, I'm supposed to remind everybody who is uh a Microsoft employee or Microsoft ND8 in the room, this is a public meeting. So in in this is going to be recorded. This is going to end up on on YouTube. Uh so it means whatever question you're asking here, do not include internal information that question. There will be a a private meeting right after this meeting uh where you can stay in the room or call into the team's call teams link that you've seen where we can then discuss uh confidential issues. Okay. But let me start right away uh with with Ultra Ethernet. Um thank you so much again.

This is a large effort. This is an effort with more than 100 companies involved. Actually many many people. Um there is the whole specification. There are two numbers on this uh on this paper abstract that I want to emphasize here. So this specification is 562 pages. Um most of you will probably not enjoy reading those pages because it's a specification. It is really really really uh not written in a way that makes it super interesting. However, it has a lot of really nice features in it. And so the authors of this uh spec basically have gotten together to extract the really nice features into a much smaller kind of scientifically minded document which is this document here. So you can see here these are the the key editors of the of the pieces of the specification at all the different layers like like ID is in the lower layers and then whole layers. I don't have enough time to tell everybody. you

you see that you can there's a good mix of companies also involved and but why did we actually do this? That may be another interesting question. It's also on this on this paper because the there's a 100x multiplier that ch sorry 1,00x multiplier that changed from the last time when we specified such an interconnect which was infiniband nearly 30 years ago until now. And that 1,00 times multiplier that changed is really the computational efficiency per moved bit. So Mo's law has been moving very very fast exponentially every 18 months doubling and networking unfortunately the number of bits moved or the bandwidth they have not been keeping up with this. So the gap between the compute capabilities and the network bandwidth has increased significantly which is positive. Right? So many people see this as negative because there's a problem if you're a system architect so you're like oh my god how do I now get

this bandwidth I need more and more nicks right which is what we do but from a network design perspective it's positive because suddenly you can employ more complex hardware on the nick itself more complex logic on the nick itself because that's literally coming for free right and that is a big design point in altra ethernet because infiniban at the time was designed for an extremely weak computation with respect to the network bandwidth so they had to make all kinds trade-offs was a great design at the time. They had to make all kinds of trade-offs like go back end and very very simple in order delivery and all of that. And now with a thousand times more compute per bandwidth, we can relax all of this. We can do full out of order delivery, full dynamic adaptive routing.

We can do everything on the nick. We can build a really silly network, a really high performance network that just switches packets in the switch. dump like many people say but I call it high performance because it's simple very simple pipelines and put a lot of complexity a lot of computation into the nick and this is the philosophy of ultra ethernet today just as a general general opener I would recommend all of you to look through this paper if you're interested because that really summarizes all of the talk nicely it is much longer than the talk has much more details than the talk um but again you can also look at the full spec where you find all the details but they may be really well hidden so they're all in there but they may be well hidden but First I want to talk a little bit about the workload classes because we're all here to do real computations and I want to arrange them along two dimensions.

One is the bandwidth sensitivity which we measure in flops per bite essentially. So how much computation per bite do we have in that uh application. So going from low to high and the second one I mean this is an easy one because the bandwidth is just a matter of area and cost. So we can just add more nicks and we get more bandwidth. That's a very very easy uh thing to do. Well, if you have money, the second one is quite hard is the latency sensitivity now which goes from high to low again in this scale. Now we have four quadrants that we can talk about. Unfortunately, the latency sensitivity is not latency is not very hard to very easy to beat, right? Because we have the speed of light. We there is a lower bound if you have long cables. Inter data center communication is an especially bad example of this. But the latency sensitivity itself is extremely hard to measure for an application. Even here

for bandwidth sensitivity, we just measure flops per bite. It's easy. You just do it. Latency sensitivity is not so clear as we'll see in a couple of minutes. And it's extremely hard to improve as I mentioned because we have limits with the speed of light. And so when you design a network, you really need to be aware how important latency versus bandwidth is because there are very often trade-offs you have to make between those two um things. And and it's it's very painful. It can be extremely painful. Right now at the le we're discussing um this outside of ultra ethnon is basically the base of ultra ethnet where Ethernet is there.

concatenated fck and things like this heavier f to enable higher bandwidth which is going to increase the latency who bad is it bad we don't know right this is why I want to spend a couple of minutes on workload classification to just remind us that there is a trade-off between those two so now in the low bandwidth sensitivity high latency sensitivity area there's not too much actually it's pretty funny so some graph computations are that the graph 500 is kind of in this area not not our workload that we have to love now so which is okay um in the high latency sensitivity and higher bandwidth sensitivity. If you go back to the more traditional field of high performance computing, modeling and simulation, we find a lot of strong scaling workloads there. If we scale to very large clusters, that's the case. We find weather and climate simulations typically there are great, but hey, aren't we here to talk about AI?

Exactly. LLM inference is squarely in that quadrant. It's it's really sensitive to everything and it's actually getting worse as we'll see in a couple of minutes. So then bandwidth sensitivity and low latency sensitivity is weak scaling HPC workloads. So workflows where I have a lot of work per u per item done and of course LLM generative LLM training and and AI training so that doesn't need a lot of latency sensitivity but it needs a lot of bandwidth okay then you wonder what's here like low bandwidth sensitivity and low so basically network doesn't matter quadrant yes um independent jobs large scale HPC but that may actually be a major business so so we don't want to ignore those too much but we are really living on on this site here somehow and what is even getting worse slightly.

What is slightly annoying, the LLM inference side is pushing to the top right with growing context lengths with all these new things like test time compute and wonderful reinforcement learning I'm doing at runtime. Oh, this is just really bad. There's more bandwidth and more latency. Uh I mean lower latency, higher sensitivity in both dimensions and really badly in many ways. Um okay but now the question is how do we assess the latency sensitivity of a real application as I mentioned that is not so simple you cannot simply compute the bytes the the flops per bite or or bytes per flop uh that you use because it is a problem for example large me if I have a large message kind of by definition the latency doesn't matter so much because if I'm sending a megabyte I don't care about the microcond latency because I'm hiding that in the in just the bandwidth term of the message transmission okay so now

it depends on our average message size. Well, that may be in a very simple metric. If you would just have the average message, no problem. I can totally measure that. But no, it's even worse because I can pipeline and overlap even small messages. So, you could say as a first heristic, small messages are latency sensitive. No, small messages have the potential to be latency sensitive. If you overlap them with computation and if you schedule them well, they're actually not causing latency sensitivity. And now it gets problematic because you now have to worry about the critical path in your application. Okay? So are are these small messages actually if they're large we don't care but if they're small are they on the critical path yes no not so simple but what you can do is you could look at the execution graph of this application you could actually look in detail what the application does uh

we've done this in our research group so you can take an NPI program or a nickel program you can trace it as a trace you can build these execution graphs and you can then actually use a pretty cool technique that I just want to quickly mention this is not this not about this talk is not about this technique I just want to mention that you can do it you can run a linear solve over and you can determine your latency sensitivity at the end that was published last year at SC24 was the best paper um um achieve the best paper and what did it actually show and this is quite interesting also for the design of next level architectures so it showed that for HPC applications and now we have new data for AI applications uh but not in that slide yet that you can determine you can grow the network latency right and you can determine when does it slow down in this case with the red line by more than um 5% %. So here in this case, if I'm

adding 22 microsconds to my network latency, this particular application MLC and 128 processes is going to go 5% slower. If I'm adding more, then it's going of course even slower. So you can now look at this this as a sensitivity metric, right? So you can look at another application. Oh, that application can take 65 here in this this red one can take 65 microsconds before it slows down more than 5%. Oh, that's great. Another application can actually take 600 microsconds before it slows down because that overlaps really well. That's a climate application. Oh, nice. Uh but not so nice because these applications they look pretty much the same from a from a bandwidth per bite uh sorry from from a flops per bite perspective. So now it's not so simple.

So we basically find that it varies by nearly two orders of magnitude at least one one to two orders of magnitude the latency sensitivity of applications. So we would need to understand how that goes. So now if you're in an operation like Azure for example, how do how do we do this? How do we find out for our applications what we do? Well, we could use that tool chain. We are actually collaborating with people in Azure to use that tool chain. And what we also have we we have and this is now getting towards Ultra Ethernet. We have a tracer and a simulator that is just published.

It's actually going to appear at SC25. It is the best paper candidate. So we'll see if if we again manage to get the best best paper for this. The idea here is that we trace nickel and MPI applications and we have three different classes of applications. High performance computing applica uh sorry first AI of course I'm sorry HPC comes second. Um first AIDL lama mistral traces up to 256 gracehoppers again we are working on larger traces together with uh with Azure. Um we have HPC applications and we actually have the public Azure direct drive protocol. So, so the the the version that has been publicly released, there's a YouTube talk on explaining how it works in detail. We have that full protocol implemented in the tracer and you can literally run um read write traces through this and it'll translate it to network transactions because that protocol is not very simple because it replicates to three different nodes and

does all of that. So now we have a full tracer and it goes into the standard gold trace format which is by the way used in Microsoft internally pretty heavily at this point. So there are multiple different orcs using that trace format and it feeds into the official UEC simulation environment. So now we are coming to UC. So that's HTSim. So you can now actually take your workload, run it through the tracer and run it on the official UEC simulator that simulates I mean that that is that is um approved by the committee so to say that is the official thing. So that's quite nice. Uh we can do this. But now I want to make one step forward. Going back to to Saraf's original statement at 2022.

Satya said we're really building the world's computer. Like this is what mic this is what what the goal he set out for Microsoft. Actually I've heard this before 2022 even but 2022 it went public. So so keep that that in mind. So he charged the whole company to build that machine. And now oh this is how it looked like a while ago. We have 60 regions, 300 data centers. Great, great, great. But then what happened then? Well, then Saraf came and and changed that slightly because as we all know, we are now going towards the workload we all love. We are not building the world's computer. Actually, we're building the world's AI supercomputer. And where did I steal this slide from Saraf's OCP talk? So, you should you should watch that talk.

The the link is down here and you can see more details on on how Microsoft envisions that going forward. So that is really a wonderful uh visionary way to to look into how do we build the future of Azure or the the future of computing in some sense and now it moved from the important piece from computer to a supercomput and that is quite a big of a change because how do we move from a normal computer an old good old data center like we used to run office and I don't know what to a machine that can actually enable AI everywhere AI in office AI is all about performance because these workloads are extremely demanding So now we need to be performancecentric and performance critical all through the stack at every single network drop. We want high performance. Now right this this gets very interesting. The first observation is that the network itself is really the cardiovascular system including the heart of any data center.

You think about it connects the units. If the network doesn't work, we are dead. Azure networking knows this very well. So they're going for high reliability. Typically you have to be extremely reliable. But the new goal is not just to be highly reliable but also highly performant and deliver reliable performance at the same time because nobody wants to wait for their query. So that is interesting. Um we've actually in in an academic context with some industry partners we've pointed this out later earlier also about 2022 that we really want to converge hypers scale data centers and high performance computing networks to something that is a single interconnect. Right? And now it is obvious from the market as sorry I've also mentioned that this has to be Ethernet. There's just no choice and this is the ecosystem that's there. It's a gigantic ecosystem. We cannot change too much about this. So I would

recommend reading this if you want to know about the the details of of things because it outlines what the operational differences between high performance networks and ethnet were at the time and now Ultra Ethan is converging those. This was 2022. Ultra ethon that was founded slightly later. Um so that's but that's something you you can understand as a motivation. I don't have enough time to talk about all the details but what is now this Ethernet ecosystem what does that mean the Ethernet ecosystem it's a very abstract term just want to give you one number actually stole this number from from another friend Rambalaga at Broadcom um so he gives us this number at uh 2023 in OCP 600 million ports Ethernet ports are shipped annually okay so Ethernet itself is actually a high performance deployment because we deploy 20 ports per second 20 40 60 80 100 ports deployed just while I'm speaking here. So think about this. This

is completely crazy. That is the ecosystem that makes Ethernet. But now what we have to realize that Ethernet is not Ethernet. The Ethernet is is is a very I mean actually Ethernet is even used for the wireless network that we are using here. Ethernet frames on this wireless. Basically everything is an Ethernet frame today. Like 99.99% of the traffic we we cause as humans are Ethernet frames. But there are very different ones like TCP IP which is what we are using right now to communicate and there is Rocky RDMA over converged Ethernet which in my view should really be called infiniband over Ethernet because Rocky for everybody who doesn't know it in detail is really just take an Infiniband header and duct tape it to to an Ethernet header and then ship it over the wire and with all its inefficiencies or efficiencies in case you want it. So it's really it's really that it's really infinib over Ethernet. Okay. So let's go

one let's double click on this a little bit. In some sense, you could argue, well, RDMA over converged Ethernet is already a high performance version of Ethernet. And yes, it was a high performance version of Ethernet that was put together by heroic engineers based on great industry demand within 6 months. They had to have something. People said, well, we wanted Microsoft being one of them. We want a high performance interconnect, but we're not going we don't like Infiniban because it's too different from what we have.

Our ecosystem is all Ethernet. So, make it work. So, what do you Well, you you duct tape or you you you put this e this infiniband header with all the goodies on on top of an existing Ethernet header and then you ship. Uh it worked well, but it has some issues, especially at hypers scale, right? It was designed in around 2014ish. It was really a quick design. Great. Again, I don't have enough time to go into all the details what doesn't really work. I mean, there's a lot of buffering. The losslessness is an issue. We can talk about lossless versus lossy because losslessness has a lot of problems, especially if you go long lengths. Uh and so Ultra Ethan has a different philosophy and here already several industry people got together if you can look at HPE, Broadcom, Google and Microsoft discussing that we actually may want something else, right? That was 2023 sort of history and what happened

then shortly after well a subset of these companies founded the Ultra Ethernet consortium. Right? So you can see these the founding members here we we are one of them but many many other companies got together basically saying we want something new. We we need to fix this. We have a thousandx increase in in this uh computation per bit transmitted. Great. That's a great baseline and we have a lot of issues with the stuff we have today. But we want to use the Ethernet ecosystem and drive it forward into the great Chinese future. So what came out of it? Well, that that was about a period of nearly two years. Um so Ultra Ethernet specification was released u about 3 months ago. In fact just 4 days ago we released an update to this. This is not the latest anymore. So now we are at version 101 fixing several typos. So it's not it's not a great uh a great change. Uh the big change will be 1.1 when we actually

come out with new features. But I'm I cannot talk about these new features here. This will be part of the session that we were doing afterwards. It's limited to Microsoft. Um okay. Then there is this paper that appeared two months later basically making the spec a little bit more accessible to people. Yes. So the missing one in this list is Melanox Nvidia, right?

Yes. In the in the founding members. Okay. Today, well, sorry, Nvidia is the uh is is is a full member of Ultra Ethernet is contributing. They're on all calls. So, so yes, yes. So, this these founding members, I mean, you you can now uh argue with yourself why Nvidia may not have been part of the founding members, but it was a smaller group. It was quickly extended. Actually today we are more than 100 member companies. We'll we'll get to this. It's it's a very very large consortium. So the calls managing actually we go here. So the ecosystem is very quickly growing. Thanks for the question. Perfect introduction for this slide. We have 10 steering companies now. So it's it's more. It's not founding companies but steering companies. Uh we have 17 general members and 51 contributor members. So that's about 100 companies. So we do have physical meetings. Uh so we meet every now and then. This is a a picture of the

24 meeting. So, we had 60 members on site, but there's really 1,300 people signed up to the to the system and and the mailing lists and everything. Um, so I'm not on this picture because I'm the chair. So, I'm standing in front and and and talking to people. This is Karen Shrem. She's my wonderful co-chair for for the transport working group. So, we are kind of the biggest uh massive change working group that we're working on. You see lots of people here. Mark Hendley, Eric Spa, Costen, and and you you probably see many many familiar faces from the networking um time. Okay.

So this is this is the full list of companies now in case you want to to check again um many many many uh companies. Um it is it is a lot of fun uh having the weekly call with about 80 people online uh and trying to to herd them together to come to a common goal. This is not always easy. Lots of interesting drama going on. Ask me over a beer. Um so that's that's actually a lot of fun. Um so now what what is this about from a technical perspective? So we have we are now comparing it to classic RDMA and what we've been doing in ultraeon just from a very high level perspective and then I'll double click into those going extremely technical in a couple of minutes. Classic RDMA really means Rocky, but also other RDMA interconnects for um like Infiniband or or or Cray Slingshot that were designed for um HPC environments, right? So they are typically lossless, right? So if they're Ethernet based, they're typ

typically based on PFC, priority flow control, which is kind of violating a mechanism that was developed for something completely different into delivering um delivering lossless networking. It's it's not a great it's kind of a hack, but it but it does work. Um, CBFC is the more elegant version. You can actually save some buffer space, significant buffer space if you use CBFC instead of of PFC, but again, I'm not going to those details here. Um, UltraEthernet set out to completely function over lossy interconnect. Oh, oh, oh, I'm sorry. I should be careful.

We we we did we decided to rename lossy to best effort because it sounds it sounds more positive. Um, so I should probably update this. This comes from our official slides. Interesting. Um and we also support lossless operation but the the base assumption is that that we are a best effort network which means the switch may drop messages or or packets if the switch is overloaded at any time and there are several cool mechanisms how to deal with this like packet trimming for example but that's the general assumption okay then classic RDMA is typically in order transport and delivery this go back end mechanism like if I'm losing a packet all the packets that are on the wire that come after that packet are discarded and basically you retransmit that from the beginning so you lose full BDP at every single packet loss which is a total disaster if you operate uh in a lossy environment because every packet that's lost loses

all the BDP coming after it. BDP is bandwidth delay product. That's basically the capacity of the wire for uh a specific a specific connection. So in ultraethernet we do out of order data delivery and out of order message delivery actually. So out of order data delivery is is how the data is connect committed into main memory like how it arrives that's fully out of order but also the messages may be out of order. that there's a now a communication going on with the higher level because in in the good old RDMA you just assume if I have a connection with somebody then the all the stuff arrives in order from that somebody right here we have four different transport modes rod is reliable order datagramgram which is basically in order that's the same as rocky as we have today just for backwards compatibility we do not recommend using it because it's not very good for reasons I just explained um the

real ones that Altra is using is RUD reliable unordered datagramgram or delivery. I'm sorry, not datagram is my infinite term. I need to get infinite out of my head. Delivery delivery. So, reliable order delivery, reliable unordered delivery. There is a small extension to um reliable unordered delivery in case it's an item potent operation. You can optimize the network a little bit more, but that's a detail. Let's not talk about this. And then there's fully unordered delivery. Okay. So, we have in the old days we have proprietary congestion control. So, basically everybody rolls their own congestion control. Azure networking knows this very well. It's a long document internally how to how to do this. Um and this is not compatible with anybody. Like if I'm building my own data center, I'm running my congestion control and nobody else runs my congestion control. And so it's all different. Um UE is really standard

config configurable and flexible. And Abdul will talk a whole lot about this uh how this works and and how we've designed this. We have contributed quite some piece of that. Um, in the old uh classic RDMA single path routing, so no lo node balancing, we have problems with link polarization. If you have an ECMP issue, they they could all go to the same link and then you're just messed up um for for simple reasons. In UE, we do per packet multipping. That's very important. So actually every packet goes onto a different path. So it sounds like a total mess if you think about it because you know need to reorder and everything all paths at different speed.

But this is quite nice in the overall package. This actually works really really well. And we have a per packet load balancing as well. So we can actually send two packets on the same path. If that path happens to be twice as fast as the other path, right? So it's not just oblivious spraying. It's it's very very smart schemes. We have contributed a scheme that's called reps recycled entropy packet spraying here. And and again Abdul is going to uh shed a little bit more light on this. We have an extreme extremely large state per cuber. So kilobytes of nick memory fast expensive nick memory per pier. Like today um Nvidia's nick memory is is about a kilobyte or so. And you may know about thrashing. So you can actually they have a little cache on their neck and then they run out of that cache and then they they they move it into main memory and if you have a bad access pattern you thrash that cache and then

you move extremely slow there like a factor of 10 to 100 slower than you should be. Um Ultra Ethernet is a fully connectionless API. So so this actually like there's no connection. When I say connection is a misnomer. There's a context. There's a packet delivery context. Really nice complete ephemeral zero state RT reliability state. And I'll talk about how that works in details in a couple of minutes. we have security that was added later. So, so the original Rocky and the original HPC hander connects, they were not designed for security because the the model there was you lock your machine, you put a you put a lock in front of it and and nobody gets in and it's all fine. Well, the model we run at hyperdata centers is a little bit different because there may be a cable going through through the street for example and then this must be encrypted or there actually many companies provide services that whenever

anything leaves the rack it's encrypted even inside the rack may be encrypted. So this wasn't this was added later which is a bit of a problem because it was really shoehorned onto the existing uh framework and has a lot of issues. So it's it's n square context. So you have if you have nodes you have n square security const one for every endpoint at every endpoint. Um we my research group actually wrote two papers how to how to attack it. So they're pretty bad attacks on rocky. So you could have illegal memory accesses into any uh expos exposed memory if you share the same VM.

This was quite quite funky. And in ultra ethernet this is all built in from the beginning. Like we have security discussions from day one. We Eric Sparta is driving this. I'm I'm part of this. We do really cool stuff like clust cluster bite re keying zero state replay protection and again I'll talk a little bit about this in a couple of minutes. So this is the overview. This is the menu what we're going to see in a couple of minutes. At the end we are aiming at workloads that are NPI open schme and CCLs. CCLs are collective communication libraries. In ultra ethernet we call those star CCL because there are so many and there's not just nickel like Nvidia CCL but there's actually a Microsoft there are multiple Microsoft CCLs not just one. Um there is an AC there are multiple ACCL's one for AMD another one for accelerated. Um there is about 40 different CCLs that that you can you can look at. Um so this is why we just say

star CCL all all kind of put them into the same bucket. Okay. So now what do we have? The application layer is basically aimed at primarily star CCLs then MPI and then open schme. Open Shrem is gaining popularity with Nvidia's NVME uh pushing it out and now we can have device initiated communication but that also all runs over ultra Ethernet no problem. We we put those on top of an interface that's called lip fabric. There's a pre-existing interface. It is not bound to infin. The other option would have been something that is bound to infinite. There are many of those like like um UCX I think and um and IB verbs. All of those were driven by Q pairs and not connectionless. We wanted an an interface that does not give the abstraction of a connection to the user because the moment you give the abstraction of a connection to the user there come all kinds of expectations with a with a connection like in TCP IP

it's in order and all of these things. LIP fabric is fully connectionless. You literally just establish a universe of things, a universe of of of accelerators or whatever. And then you can read and write and whatever. There's no you don't see any connections, right? That that's the importance at the very bottom. It's Ethernet. It's your standard run-of-the-mill well actually high performance Ethernet. It's 100G lane speed or 200G lane speed. But these are standard E. We have a Leos with the LE E at that at the FI layer where we basically look at what they are doing.

sometimes scared when you look at at all this concatenated fact we're a little bit nervous it's like is this a good idea is this is not so we're trying to influence them uh for for our and they're good um but this is not specified within the ultraethernet consortium so so we have we have a leas there then the data link layer is also partially in the e it's also e 802.3 we have optional link level retry and credit based extensions so we actually use the good old e headers by the way this is the ethernet that header that we've been using for 30 more years, right? So, it has a preamble. Oh my god.

And a and a start bite, destination MAC address, source MAC address, a type, and then it comes the data, and then comes a frame check sequence and the interframe gap. If you want to you have to round to 8 byt uh multiples of eight byt unfortunately. So this is this looks very inefficient, but there are ways that we can talk about later to make this much much much more efficient. It's not uh public yet. Um so there's credit based flow control that we add at the data link layer in the 1.0 spec. Basically you can turn off PFC like many vendors have this today actually I don't want to name names but many vendors you can buy this from today already. There is link level retry. So this is kind of an extension for PFC to enable optimizations or not extension replacement really it is it is not PFC it's very different and you can have link level retry which is a mechanism where you can retry on the link

basically not end to end but to to amp the reliability of the whole thing like to amp the reliability of a single link in the environment then there comes the network layer the IP layer which has not been changed at all that that is everything that's gray here is is kind of already specified or specified in another body. Everything that's green is an ultra ethernet thing. Okay. So now the network layers of course sits in the in the data packet. Um it has you're using standard IP. So either IPv4 or IPv6. The good old headers they look again humongous for an HPC person like 20 byt header right there plus optional things or 40 byt header if you use IPv6.

And then comes somewhere the UEE payload. Um UE was designed at a time when we did not really consider latency super critical. Actually, it also turns out, oh, I'm sorry. Packet efficiency is super critical because it turns out that at the speeds today, if you add a 100 bte header, that's a nancond. Doesn't matter. If you're running at a 200g per lane, header space is no issue anymore. So, you don't need to optimize the headers to death. The only problem is if you send very very small packets, then you have an issue. So, it's not a latency issue a large header. It is it is a packet efficiency issue. if you run at very small packets. Ultra Ethernet is designed for AI workloads that usually run relatively large packets. Um, okay.

So, we have that and now let's go a little bit more into detail of the transport layer. I mentioned that I'm the co-chair of the transport working group, which is really the massive thing out of the 560 pages. It's it's 60% or so maybe 70% is a transport layer. So, here it gets a little bit more complicated. So, IP this is the transport uh sorry, this is the internet protocol. So, it's addressing IP addresses. Then there is an optional UDP header which you may be using. We got an Ayanna assigned port now 4793 in case you're you're you're a port geek. That is uh Rocky the Rocky port plus++.

So it's twice above it because the next one was already taken by Huawei. So we couldn't get that but it's but it's twice better. So okay. So then we have an optional security layer. So there's an optional security header which which has an ICV at the end because it's it's of course security. So you can see the bite numbers here. That's the transport security sub layer that enables O of one key state per endpoint. Um we use key derivation function for rekeying. Super super cool stuff. And we we have this full replay protection. That's actually super nice as well. Um congestion management sub layer is really the window based center and optimized receiver congestion control that Abdul is going to talk about including flexible load balancing. Uh CC or CMS doesn't have separate headers because the CMS headers are in the PDS layer.

The PDS is the packet delivery subsystem that is kind of you can see a whole layer structure including nested layers. So we call these things in the transport here. By the way, these SS are subsystems or sub layers. These are layers and then there are sub layers in here. Okay. So the packet delivery sub layer really enables this zero RT context build uh request response control packets flexible loss detection all of those things that I'm going to talk about later. You see I'm going more and more into detail over time but trying to make it in a way that you can follow easily. And then there is on top of this is the semantic layer which really goes to lip fabric and that's the interface to the top that has then the concept of addresses send receive messages deferable send RDMA or RMA reads and writes so the these are translated this is messages this is packets we are then at the packet level

here we are still at the packet level this is security and now down here we're at bytes right so this is how the stack goes it's very it's a very standard stack we just extended all these sub layers um okay wonderful And then let's zoom in a little bit. Let's zoom into one very important aspect of of ultraethernet which is really the its load balancing philosophy. It's load balancing philosophy has is very different from what everybody has been doing in HPC before but it's very similar and compatible to with what people have done in the data center mega data center environments and and good old Ethernet.

So many of you are familiar with ECMP equal cost multipaththing. It was actually the good old days was designed for failover links, right? So you don't you wanted alternative paths in in case something fails you just go another path. But you can use that equal cost multia to enable multiple different paths potentially for every single packet in each multiple different packets go into multiple different paths because what it does is very simple. It uses some you it it applies a hash function to the header of the packet and you can configure which part of the header and based and that the output of that hash function. So the input is the the the header or part of parts of the header. The output is an index into a port group. So it's like an output port.

So if you look at this switch for example here. So we have certain down ports. So this thing has eight ports. It has four up ports and four down ports. Let's just focus on the upport. So we are coming in with a packet here. Then the the ECMP applies a hash function to the packet and selects one of those four ports based on the hash output volume. So if all packets have the same header then they will go all the same port because the hash function is deterministic. Ignore failures. If you have failures it goes different but ignore failures. So now but what I can do is I can introduce something that we call an entropy value somewhere in the packet header. The endpoint can decide to change a value or not change it. So now the the endpoint knows whether two packets go on the same path or whether they are likely not going on the same path. If the entropy value is the same, we are guaranteed that these two packets

are in the same path. If the entropy value is not the same, we have a high probability that the that the um packets are on not on the same path because they could still have a conflict. I mean the hash function is is having an n tom mapping and m is much smaller than n. So obviously there's a conflict in the hash function. So let's look at this a little bit in more detail. Um and and so this is all written here. So if we want to communicate from A to B in this network, we would use an EV equals A, right? So it selects that path and then it goes down. By the way, an EV you can in in a fat tree network like this or in a class network like this. You can essentially imagining an EV address is one of those top switches. The the way down is deterministic. There's only one path down. Okay. But we support different network types as well. So it's not not the only topology supported. EV equals B goes the different path. So we we hit a

different top switch but we also end at B. So you can see now how this works. The the injecting nick decides which path to take based on all kinds of complicated measures that Abdul is going to talk about. Um but this is the general philosophy. Now even if you're in the same group going from C to D, we have a similar thing. So we have like going here and we can we can go here. So this is the green EV and the the red EV. So just just for everybody to understand. So now going back where does this EV live now? Well, in in good old data center, the EV was encoded in the UDP source port because nobody needs the UDP source for port for anything. You can just like as long as it arrives at the right destination port, nobody's going to check because UDP is is is no context anyway. So you you can update the UDP source port or you can actually there is a mode for ultra ethernet where we are not using UDP. Okay. Um where

ultra ethernet directly sits on top of IP. We did not get an IP protocol number yet, but we are working on it. So, we have a UDP number and then you have a separate entropy field. The nice thing is um this will work over any switch you have today because it's at the exact same position as the UDP source port would sit, right? So, so it's just all backwards compat. It's Ethernet. You must have must make it all backwards compatible until the the end of time. So, this is a nice thing. Okay. So, now we know how to do load balancing. Now, I want to double click in each of those going into the transport layer in each of those areas. Okay, going from the top to the bottom semantic sub layer. So the semantic sub layer is very important that is compatible with existing applications that are written on top of lip fabric. And now you can argue well there are not too many applications written on top of lip fabric and you're

absolutely right but but those those that are there are compatible. Um so you may actually implement a little shim layer that you can link to existing older um IB verbs applications for example if you want to deport them easily. Um but again I can't talk about too much in that yet. Um so RDMA services we have cent RMA just standard stuff mostly optimized for CCL semantics scalable addressing to millions of endpoints. We'll see how that works. And then we have optimized extensions. So the one that I like most is deferable sense. We'll see how that works. We use rendevu which which has been done in in HPC for ages. I mean when I was a young student I was working on rendevu. And we have exact match as well to optimize the hardware in the nick. I don't I will not go into this too much. This is a very small thing but super important actually for for Nick architecture. Yes.

U just just go back to the semantics layer. You you you mentioned messages which is great. There's no concept of reliable stream like like TCP, right?

No, it's all message based like you have a message itself is a little stream of packets but it has a concrete start and end there. There's no infinite stream like in TCP IP where I'm just like reading and writing to sockets. Exactly. But you can if you want by the way you can easily implement sockets on top of the thing because you could emulate it. It's it's very much like the sockets direct protocol in if very very good question. So let me first and please keep asking questions if if there is anything unclear. So let me first talk about the the the protocols. So it's really all optimized for specific use cases. Ultra Ethernet has three so-called profiles. So when I'm building a nick, I may as a vendor say I'm supporting only the AI full profile for example or I'm supporting only the HPC profile or I'm supporting all of those.

Like this is now a specialization opportunity for Nick vendors to cut down on the nick silicon to build smaller, more efficient, more optimized things for these three different use cases. AI base, AI fold and HPC. The main difference between those three is going to come in a minute is the protocol they use for communication and the and and there's some some bells and whistles on the side and you can check the spec like some don't support certain atomics others don't support certain atomics basically HPC supports pretty much everything um AI full supports the subset that most people agree that is is great for AI and AI base it's a very small mini version in case you want to make a really really optimized AI nick minimal size but it has some some downsides because of course if you minimize things you often lose something so let's first talk about addressing addressing is very very simple you have

a an IP address I mentioned targeting scale up or scale out so this whole ultra ethernet 1.0 zero is aimed at scale out.

Scale out only. Correct. Scale out only. Um, however, you can use it in a scale out context if you want to, but it's just designed for for scale out, right? So, it's not optimized specifically for small distances, small domains, but we can talk about uh future plans in the in the uh closed session. Um, okay. So, now if we address things, so we have here a standard address. We call IP addresses fabric address because we have a fabric. We also have fabric endpoints and and so it's basically the same thing. Then there is a job ID that's new. So the job itself is part of the address which is very important because it enables the scalable addressing to millions of endpoints and a fast lookup. And then we have a thing that's called pit on fab process ID on fabric endpoint. A fabric endpoint is something like a logical nick. Imagine it like a logical nick. Um okay. And then we have a resource index

which you could imagine like a UDP port that is now on the logic on the process on this logical nick we have a port. So you see lots of lots of addressing because we want to address millions of endpoints. How does it go? We look at the fabric address. The network will route uh things with the fabric address to a particular nick, right? So that's now standard stuff. So I don't want to talk about this. But then this nick has a table of job ids like which jobs are running on that nick and these are parallel jobs that could be thousands of nodes comprising many many nicks. Then we have this pit on fat business here which you go from the job ID and then you find the particular process ID that is associated with that job right what's very important is we can have many many processes on the same node running with the same job right I mean it's pretty obvious I hope and then we have a resource index which now you can say

well there's an NPI service there's a CCL service on that particular job right this is how it works this is these these addressing components now you may wonder what if I want to build a storage system What if I want to build a system that doesn't have jobs, right? I want to have a client server system. Then we use what we call absolute addressing. So we have a fabric address, we have a job ID that is not used in that context and we have still pit on fab and resource index. So we just address directly into the pit on fab. We address into the resource index and um and that's how it goes. So now let me talk a little bit about the the the protocols that I was promising. So how do how am I now transporting messages over the network right? So first of all we have a sender and a receiver and I explain the standard HPC rendevu protocol because that's probably what most people are most familiar with.

So we have CPUs and nicks in these protocols. The the the job to to be done comes from the CPU. So the CPU says I want to send a message. This is a t send is a tact send but details doesn't matter. I want to send a message that has an address and attack to the other side. And then in a in a messaging protocol the receiver has to say I want to receive the message into this address and of course I'm waiting for that tag. Right? So now there are two cases. There is a case that is called expected message case when the receiver has posted that receive before the message arrives. Then there is a case where there's unexpected message where the receiver has not posted that receive. So the problem is the receive tells you which address to go to. So if I'm sending you a gigabyte and you have an expected message, you just put it there.

You know where to put it. If I'm sending you a gigabyte and you don't know where to put it, you can't do anything. I mean, imagine I'm coming to your house with with like a ton of of of rocks and and you don't know where to put them. So basically what you say you you you decline and you say, "No, no, no, come back later." Basically, okay, this is what this protocol is about. Expected case. How does it work in HPC typically? So this goes to the nick. The nick issues and an RNV send message, which is a smallish message, usually of size BDP, and says, "Okay, I have something for you, but I'm not going to send you the whole thing, right? because I'm I'm careful. Then on the other side, the nick will decide, well, you know, this is an expected case, so I actually know I just write this to the address. I've gotten that before. And then what it does then, of course, because the message may be bigger, it issues a RDMA

read. So here is metadata in there that says, "Oh, by the way, I have more for you." And you find it at that address with that key. And then the the receiver nick issues an RDMA read of that message that completes here. and then it gets the response and then it gets the whole like two tons of stones if it wants it. Okay, that's the expect case that's always great and then of course it writes it to this address and then it's done, right? The unexpected case is a little bit more annoying like you have this thing message comes over here. Oh my god, no, I don't know what to do with it. Right? So you may have very little buffering. So what you do then is you this protocol sends a message back. Oh no match I don't know what to do with it. Um okay and then comes the receive later, right? And then you can do the read. That's the that's the rendevous protocol in HBC and then it does all of that and then it completes everything

and everything is fine. Make sense? That's the old protocol. Cool. Then we have an AI based. So So this requires hardware support for this whole

first. Yes, please. I was going to ask if there's a time limit there otherwise you might deny the service.

You can Oh, this one is actually rather hard to deny service. I mean you you there's usually no time limit but in in the HPC Oh, let me let me go a little bit broader. in the HPC context um you trust your user.

Yeah. Yeah. Yeah. So you can you can probably run this against the wall if you want. Yes. If you just issue a whole lot of rendevous requests and then the buffer is overrunning at the other side and and you you made that lock in that case actually. Yes. Yes. Yes. Um I'm not sure you could introduce a time but it's not prescribed.

Yes. Yeah. Basically the same question. Same question. Yeah. You you you trust that system. Um okay. So then on the AI based pro but but the problem with this is that people saw is that you need this runs underneath this protocol. So you need hardware to to run this to to do this. People were concerned about this that in AI base oh my god I need too much hardware in AI base which really should be the trivial the most trivial nick you can imagine. So how does it work then? So there's the expected case you would basically you have the same thing right you you get this address but what you do then it's receiver driven.

So you get the address from the receive side. You do a one packet send to the sender. The sender doesn't do anything. The sender waits in software until it sees the one packet send and the one packet send contains simply the address and then you do an RMA right to the other side. A very simple protocol. It's strictly less powerful than this protocol because in this protocol I cannot have I'm I cannot have a receive operation that receives from anybody because I have to know who I'm receiving from otherwise I don't know where to send this message to right so this only works for CCLs while here in MPI it has a specification that you can wait for messages from anybody which you which you cannot do over here but for AI that's that's perfectly enough um okay so then the complete negative how does the unexpected case work well it's exactly the same because now it's always expected essentially Right. Cool. But

you already see the problem. Anybody sees the problem with that protocol? Yeah.

You have to wait on the sender side. Exly. Exactly. You always have to wait. In the worst case, you gain an additional RT which can be really really bad for small messages. Okay. This is why we came up with this deferable send actually contribution from Microsoft um where we have a sender and a receiver. Same thing. What we do is basically we have again the same thing. this expected message case, but instead of sending carefully only the start of the message, we just say, "Look, we just send the whole message. We start we we we pound the whole thing into the network." And then if that guy is ready, it'll just sync it. Okay, no problem. And it's called deferable send for a reason because if you do this, there may be the nick just says, "Oh, you know, I I can't take it anymore." But it says it's fully in hardware. So this is kind of a weird in my view weird protocol like it it uses hardware support but it doesn't

really use the hardware support. This uses the hardware support fully basically makes it part of flow control. It just message comes and say I don't know what to do with it. You you just stop it. This is this is a there are two tokens an initiator restart token a target restart token that you can identify these messages at the initiator and the target and then later when the receive comes you have an a ready to resume basically this is a stop message is a ready to resume message and then it goes like this has the big benefit that you don't have this back and forth here you just sync it in the expected case the nice thing about AI is that most of the time you're in this expected case because you can schedule your workload pretty well most of the time not always in HPC it's a little bit HPC are actually most of the time in the unexpected case unfortunately because it's not so good schedulable for for the

average application again there application difference okay cool so this is you're going high level summary this is standard in HPC we we know how to build it so lots of people very comfortable with this yes

so for the deferable send case uh so the protocol needs to do tracking at the packet level to resend the packet because it's out of order right

yes okay the all of ultra ethernet's tracking at the packet level. We use very simple bit maps for this typically. So you don't have to know anything about the packet besides whether it's zero or not. So it's a single bit. If you look at it, it's not that much because packets are 4 kilobyte in size or 8 kilobytes depending on your MTU. And then you you you can track a whole lot of packets very very cheaply. Like that was not possible 30 years ago. That is possible now because we have this thousandx improvement.

Yeah. But it's all per packet. Very good. How many packet are you track? So how many factor are you able to track at the maximum?

Um you you can you can compute it basically. I mean it depends on your nick resources but you can all you need is a BDP essentially if you think about it because you cannot have that much more than your BDP times the out of order factor and if you look at these networks the out of order factor hopefully doesn't grow that much. So it's a it's a small constant times the BDP that you need.

You have packet loss more than packet uh BDP right? Yes. Yeah. Yeah. So you have other reasons who spec is

that yeah exactly but that depends on your implementation how much you want to keep that that is not in the specification. So so you got

so the spec doesn't say that the minimum how many do you need to track the spec says you need to at least a BDP times a small constant factor I think it's 1.5 or so. Yeah the spec gives you a minimum but but a good vendor would probably use more probably have more tracking resources.

Mhm. Yeah. And in the deferable send, can the receiver uh buffer data before sending a response back saying it's up I have space on my side. I

this is this ml length thing. So if the receiver actually has memory, it can it can uh it can store it locally, but it can also say well you know I didn't I didn't have any memory and re-request the whole message. By the way, it's the same over here. This can also you can discard that message here if you wanted. You can reread the whole message if you want. So it's really flexible in terms of receiver resources investment. Okay. So that left side is very nice experience in industry but it has it has to send a fixed size packet here right a fixed size message sorry this is all AI optimized. So it can basically this can react this is better than the than the rendevu protocol because it can react to congestion window change while it's sending the first message. Here you have to decide what am I sending this round the voo send at what size am I sending and the congestion window sometimes they

go up and down and up and down. if you're a CC expert. So, and this is super simple logic because it doesn't require read like read is complicated to implement and then this yeah the main argument here it's simple to implement but it requires software to get active and it adds in one RT in the worst case so an RT can be big so but it's super simple to implement so it also has its place in some sense so if you're buying a nick there are these three different versions of the nick essentially so packet delivery so packet delivery what we have here is a dynamic ephemeral context so we have these services that I already talked We support all the cool stuff shared receive Q out of order packets and everything and selective acknowledgement as well. We just talked about packet tracking. So you can do a fully sac um unless you run rod. Rod forces you to do go back in but but everything else is is fully full

selective acknowledgement. Zero RT startup just to visualize it in the good old TCP days you do this right you you build a connection and then you send your data in in UE you basically just send your data right so how does that work in detail? You may be wondering oh that's not so easy. it is not so easy. Um we have two two connection state machines that I'm visualizing here. Actually one connection state machine but two different ways to visualize it. Um but the nice thing is you can drop a state whenever it's convenient. There is no overhead whatsoever to reestablish a state. I mean it's in the nancond. There is a little bit of overhead because you need to allocate local resources on the nick but if this is in hardware this is in the nanconds.

So so this makes it trivial to migrate a process to another place. Right. It's it's very nice. It's very nice because you don't have any state basically. Yes, exactly. You can just drop the state, migrate the process and rebuild at the new at the new position. You just need to make sure that you get the address translation right. I mean that the new address is addressed. But you can also move the address with a I mean we we know how to do this like multihoming and whatnot. Okay, that's all standard Ethernet stuff multihoming.

We would just use that on top. Just to give you a little bit of a of a of a view of that zero RT connection establishment machine because it's quite cool. Here's again initiator target. So if you wanted to start you always start at the initiate. I mean it's obvious that you start at the initiator because you know you want to send something. You allocate a local local uh packet delivery context ID. PDC ID is packet delivery context ID. You send your message over and now what you do uh this is the state machine is building here at the initiator at the target. And this is the flow diagram. Now what you do is you you have your PSN. Your PSN is you keep on a per pier but you can reset that PSN if you want. It doesn't actually matter that much. It's just it just has to be consistent within a single message, right? So let's say we start at four five, we set the SIM flag of this packet and now we say our source PDC ID is the

one we just put here, but instead of the destination PDC ID, we put an offset because we're putting multiple packets on the wire, right? And then on the on the other side, the other side will not have the the moment the first packet arrives and it it could be that the the first packet that arrives is the second packet send, right? So if the second packet arrives, let's say it'll see, okay, I have a source PDC ID Z 7. I have an offset of one and then I can look at the whole shebang in the packet and I can compute where it needs to go and I can build my PDC ID at the target which I'm sending back in the act but I'm sending all of those X that have the sun set. I'm sending all X back with the uh destination no sorry with the source PDC ID 19 now. So now the moment the first of those packet arrives, it doesn't matter which one first of those packet arrives, I can go here into established because I got the PDC ID from the other

side. And now I'm just moving forward. I'm also established the moment the first packet arrives back from the source. Uh it carries both correct identific identifiers. So the offset is is now the PDC ID and I can move on and then I move into established and then uh like tearing it down is complicated but it doesn't tearing down is not the performance critical thing, right? tearing it down requires a round trip. It's it's always the uh the initiator I think that says I want to kill it. I want to send it close to the other side.

I forgot it. Maybe the the destination as well, but it doesn't actually matter that much. Okay, so this is the full zero RT startup protocol is actually quite cool. I think it's the first one in industry that that I've seen that that really does it consistently. So you can build connections at at nancond granularity. Okay, cool. So then CMS, I'm not going to talk about this because that's Abdul's part. Uh we have NS network signal based congestion control and the receiver controls congestion control. And again I defer I think his hand.

Oh, Abdul has his hand up. Yes. No, sorry. Just a quick time check reminder. Yes. Yes. Yes. And so so this all works together. Exactly. That's why I'm skipping this. There is a lot of um uh lot of academic stuff going on here in the context with a lot of collaborators here that Abdul is going to talk about that I'm skipping. And then the last piece I want to talk about is the transport security sub layer. That's it's really really nice. But again, since we're on time check, I'll I'll keep it very short. So we have end to end AES encryption. Key derivation. In case you don't know what key derivation is, talk to me about it. Absolutely awesome because it changes the key for every single packet basically. It's super super nice. you have very nice replay protection which is which takes a while to explain. Um it's also written in the paper very cleanly. But how do you do replay protection just for your

brain? How do you do replay protection if you're not allowed to keep a per endpoint state? You don't know when it's replayed, right? Because you're not allowed to keep a per endpoint state because your PDC may vanish at any given moment. You'll find the answer in the paper. I can explain it later. Uh this is a really cool idea. I'm very proud of this. Um and scalable security domains and it's all the whole thing is optional by the way. It has additional headers. It has it has a little bit of overhead of course but it's all state-of-the-art IPSCAC PSP everything that we know and love. We can have extreme scalability.

So the improvement on IPSCAC and PSP is that we have group rekeying secure shared domains with jobs. So O of one key state generally like this scales until the end of time and we have a very strong isolation. Cool. With that I would like to come to the the end just overview. So we've talked about the motivation of ultra ethernet how it came into place. Um, it's a big ecosystem growing very quickly. And then I gave you a detailed overview of pretty much everything that I would find interesting. The most piece here is in the transport layer. And again, I'm I'm one of the co-chairs. So if you have any questions, I can explain anything in excruciating detail if that wasn't excruciating enough yet. Um, and then Abdul is going to talk about this. This is why I want to skip it. And if you want to find more information, uh, you can read this paper. I think it's like 30 pages or so. Um, but it's written for

a scientific audience about the the the cool pieces in it. So yeah, thank you very much. And I'd like to give the the stage to Abdul then to talk about congestion control, congestion management, unless there are any quick questions.

Okay, one last. So Abdul, you can set up while I'm Are the messages reassembled at at the application at the OS at the nick?

That's a great question. The message is reassembled into the buffer of I mean you have to have a receive buffer, right? that that that that's the fundamental principle and the message is assembled into that buffer directly unless it's an unexpected message but but then you can also ignore it and just wait for the expectation and it may completely arrive out of order into that buffer and the nick keeps a a bit map marking how many segments of this message are still outstanding I'm still waiting for and when the bitmap fills up to one you basically notify completion to the user so this is this can in terms of in order and out of order it can be complete craziness in some sense because they could fill up all at the same time like with random offsets and random things and then come these transport modes in you can force the protocol to deliver these messages to you in order if you want um but most often you don't

want to because you lose a lot of performance right then there is a really really cool let me just finish the last thought it's really cool mechanism how you can use matching to get in order message matching is parts of deferable send you can get in order message notification but fully out of order message delivery So, so they will still notify you in order, but they can arrive in any order and you can use matching a little matching trick um that I can explain offline. It's very very simple. You just put the message ID in the match in the match field. Cool. Yeah. Thank you. And Abdul, the stage is yours.

Thank you, Torson. Uh all right, great. Thanks everyone. uh I will uh be talking about the UE transport uh condition uh and load balancing which I am a co-editor for as a section uh for a brief time actually during the UEC inception I served as a coacher and I continue serving as a steering member u for UEC in the meanwhile uh one more thing is the the work is a combined effort with many many people uh many within Microsoft and of course some uh we work with outside and u just uh cannot fit all the names here. So uh before going too deep uh it would be a good idea to give a high level definition or hand wavy maybe explanation for these terms just because uh it might be carrying different meanings when you say load balancing is an overloaded term. Uh so uh we are referring to it in the context of path or routing load balancing. Basically if you look at the lower right uh of the slide you'll see a toy topology very

simple basic one uh you have at the very bottom a few sets of servers each is connected to a top of rack switch and these different topof rack switches are interconnected via the top layer aggregation switches between to one and tor n to go between these two uh you have three different routes in this example and basically load balancing is the mechanism by which you are attempting to spread your traffic as evenly as you can across the different viable routes. What you don't want is to hit the cap on one of them while the others are underutilized. It's too bad for your good condition control on the other end again in a very handwavy manner. It is the mechanism by which you're adjusting or controlling the sending rates uh cumulative sending rates of the perhaps several entities that are sharing uh one link in the network or perhaps uh one uh receiver.

Uh what you don't want again is to overload any bottleneck where you're sending more than it can drain at a rate that it cannot absorb at and consequently would end up filling buffers and causing massive queueing latency especially for entities that are just driving by and contribute nothing to that congestion. Uh this is uh in essence what uh Torson was mentioning in his talk too about the tradeoff between latency and throughput. You don't want to run too hot uh on the links and cause massive queuing latency. Neither do you want to be too draconian and bring the throughput way too low just in order to be happy with your latency.

And uh we choose here to talk about the two in conjunction uh essentially because despite them being often u actually most of the time studied independently uh we do see a very tight interaction in certain cases at least in the context we're working on. uh I will demonstrate to that in the next few slides uh to make sure it makes sense why we're addressing them together. So ECMP um Torson again mentioned about that at the very high level it is uh people exercise ECMP at the flow level. Uh if you think of a flow or a connection TCP connection or even a rocket cooper as an entity that has constant source and destination port numbers and source and destination IP addresses. uh switches hash on these things uh and uh they based on the hashing output decide on which of the viable forwarding options uh they will choose. In effect, it is equivalent to flipping a coin for a flow and saying you get one of the n viable routes to

hash to. And it's great when it comes to ensuring no packet delivery makes it easy for detecting lost packet. But uh the challenge with these kind of things is uh the extent of hashing collisions you would see. Uh that's again a toy example or simulation. We just wanted to demonstrate how bad the issue can be. In this case, we have 16 uplinks or ports 17 through 32 and you have 32 flows that are all of the same size. So, uh, and they're going from one tour to another. Basically, uh, what you're seeing in that time series is on the y-axis the number of flows mapped to each uh, of the uh, ports in the network. Uh, some of them are getting only one. I mean typically 32 divided by 16 uh means if things are ideal and perfect you'll get two flows on each port but u well it's random you cannot really control it that well so some of them are getting uh less than what they should and some of them

are getting 3x what they should in this example and consequently their completion time is u and 30 something millconds in this case as opposed to a bit less than 10. So you you have a 3x or more than 3x uh regression in performance just because you want to uh use that flow level hashing mechanism. Um showing this bar chart uh because it will come up in the next uh slide again. You know your 32 compared to 10 and um I'm actually going to show it here again just to compare to something we call random packet spraying RPS. Um RPS is similar to uh flow level uh ECMP uh in the sense that uh you also make it uh obviously and randomly but uh the main difference is you're not doing it anymore at the flow level you're doing it at the much finer granularity of packet level and therefore instead of having only 32 choices to choose from uh across 16 ports and yes the margin of error is much larger with the many many

more choices you're making uh for at the packet level uh each can hash randomly ly or route randomly. Uh you have relatively speaking much smaller gap throughput wise and that's why uh you are getting around 3x improvement. You're quite uh better but you're still like around 25% worse than the ideal completion time. Uh in that case remember that 12.8 uh for RPS in red. I'm going to actually zoom in here uh from a congestion control perspective to complete that picture. And again to tell you what I promised uh that congestion control mechanism we were using you see the labels on the x-axis was called HPCC high precision condition control which is a great scheme uh in the literature uh one of the very advanced ones actually and it benefits from inband switch telemetry. uh you get a lot more information about how many uh more buyers can you send to fill up uh your pipe uh P delay product basically or

your bat play product but inherently in the design is the assumption that well you know you're going to use one path therefore and and I'm going to optimize well for you that one path uh which is therefore I mean not not to blame HPCC it was never intended for multipassing uh the ironic thing is DCQCN which is you know an older generation solution um which doesn't have all the uh extra signals that HBCC has despite that actually is working much better in that uh in that context when you are doing package spraying or multia and uh and the reason is if you zoom into the time series of the different flows that uh were traversing the network with hpcc uh you'll see that their window size which is a proxy for their sending rates was too much like too volatile uh quite not very stable. Basically, one act acknowledgement packet uh coming back to the sender might have indicated that on the path the data packet that triggered

me there was uh uh so much x bit bytes available for you to jump and grab. uh the act right after which you know was triggered by a data packet traversing a different path might have indicated the very opposite saying well you know you you have to go down by y bias and therefore you get that sort of confusion jumping up up and down with the HPCC which is not exactly uh what you see with DCQCN despite the fact that maybe DCQCN was not initially designed or intended to work in that setting but uh it it ends up serving better that sort of environment uh you do get um more stability hence or consequently resulting in that better completion time as we showed earlier. Uh I'll show you later on uh the solution we have contributed uh how it will help it get even better uh when we talk about it.

Getting back to UET the transfer for UEC u and u a few things here just to narrow down the context and clarify what we're going to be designing for. Uh so UET was not really designed for any arbitrary topology or at least we wanted to initially narrow down the space and focus on what uh perhaps much or most of the industry is focused on fatted trees or uh you know folded claw network. Uh that being said our solution as uh we have been exploring independently few of us outside uh you know can be easily extrapolated. We have actually some good experience and already some uh unpublished results to that and where we we uh show how we can make it work for things like a a Firefly or a click or any of the of your favorite you know direct connect topologies.

uh but but in that context at least in the fat tree all routes have the same length between a pair of uh servers all all meaningful paths you want to traverse and therefore you know your low balancing mechanism uh is not burdened with the extra effort of knowing the dimension like should I use a short path or a non short path uh you will get away always working with the shortest path to minimize latency as well uh we chose to start uh by default at the full line rate as opposed to taking a TCP approach where you start too slow and gradually ramp up uh with what what is called slow start. Um we are not too concerned about uh bandwidth meltdown. Uh we have uh some very good mechanisms to very rapidly actually adjust the rate and uh quite accurately. Uh we'll talk about it quick adapt. Uh so that that's why we were also comfortable to say sure let's actually start at the line rate and work

our our way backwards if needed. And uh finally on the workloads uh we're not too crazy about incast I should say in uh in what we have done so far in UEC. So uh as a starting point we were focused on addressing the light uh extent of uh incast 10 to1. We did want to provide some uh cushion for adding more confidence and demonstrate that we can handle things like 100 to one. uh but again just like the first bullet I'd like to say here what we have come to design uh though it was not engineered for that but actually does work quite well for uh much larger incast degrees and that's why we we have uh suggested that this kind of work is more generally useful beyond just maybe an AI IML uh or HPC context and can be extrapolated for storage or cloud computing more broadly as we have also uh obtained some results to that So now I'll get to talk about um what we contributed to congestion control. Uh as

Dorson mentioned there are uh and by the way this is the foundation for the for the default uh mechanism for condition control which is the networking based uh one. There is an optional one uh which is a receiver based uh uh solution for better handling or for being optimized to handle incast. Uh so I'll focus on the network based one here and um uh in fact elaborating on the name will give you a good idea what we're talking about anyway before I go into more details. It's called smart and uh it stands for a sender based as opposed as I mentioned to a receiver based solution that relies on ECN marking hence the word marked ECN marking explicit condition notification that's a mechanism by which switches uh when they have massive Q or when they have Q build up they will set a bit in the data header and consequently the receiver will echo that big bit back to the to the sender to let it know that

you have congestion uh we use both actually ECN marking and as the last word in that name is saying timed or delay based or RT based measurements roundtrip time inflation is also a good proxy for indic for indicating condition and uh they're not exactly equivalent I should say ECN and RT in the literature there's been a lot of debate and arguments we chose to uh take an open-minded approach and actually to incorporate both uh knowing that each has its own uh um trade-off or downside and we want to get the best of each. Uh so we do utilize both signals given that they're readily available anyway today. Um and we also as I mentioned have a component a key one that allows smart to rapidly adapt it in both directions actually not just when you want to go down in incast but also when you have a lot of abundant bandwidth when a job finishes and you are in a position to ramp up quickly and grab remaining bandwidth. uh smart does

that very efficiently without significantly overshooting and causing massive issues. And the last uh word I'll mention or I'll comment about here is trimmed. Uh we do actually optionally uh we don't need it but we uh do utilize it when it is available. Packet trimming is an emerging switch feature which is a great way for uh signaling packet loss or actually identifying the lost packets more accurately. um much uh more efficient in recovering lost packets as opposed to timeouts in general. We uh did again openmindedly utilize it despite it being initially proposed in a in a receiver-based incast control. Uh we we earned very similar rewards with it and uh showed also how we can integrate it for adjusting your window and more than that.

So here's a more uh qualitative description uh of smart or at least descriptive uh for what it uh is doing. Uh in gray are the things that are not too different than what you might have seen in some of the schemes in the literature. So yes, we are sending based scheme. We're window based as opposed to rate based and uh we we are simple and very easy uh to deploy in in hardware. uh but we as I mentioned uh worked better on providing the overlay of both ECN and delay signals. We did recycle some of the work that was done uh I should be clear about that but uh we're very happy how we orchestrated it together whether uh things that were done in in some work like swift or uh which is what Google does or what uh we at Microsoft in DCQCN or prior to that in DCTCP has done we took a very good combination with all the lessons learned and put them in a very meaningful manner.

uh as I mentioned in the few slides prior to that when we when we were talking about uh the load balancing and how congestion control interplays with it uh we showed HPCC and uh random and DCQCN and demonstrated how the volatility of rate control was hurting the performance with smart from the get-go uh we were very keen on making it actually react in a gradual manner we don't want to be too crazy about jumping up or down and taking a big hit just because one of the many routes we're taking is congested. Uh it's very true when you get to look at the mechanism and read about it and and in fact uh one thing also to keep in mind is uh the the analogy I always like to make about condition and load balancing is that when you're driving on a freeway and you you're on one lane and there are many other lanes, your lane might have few cars ahead of you is a bit congested.

You can either choose to slow down and stick to your lane or change lanes uh hopefully without making anyone uh get into an accident uh or you can do a bit of both sometimes. So load balancing and congestion control are essentially trying to uh achieve these kind of uh tasks without having to be too confusing. What you don't want is to be like overreacting condition control and masking away the uh congestion for low balancing so it cannot anymore adapt for uh or when you have congestion vice versa. Uh so there's a very sort of nice game that goes in there and you want to balance it well balance it well. Uh and and again as I mentioned we worked very well on being very responsive especially on the way down when you have incast. uh but also as I said we do a quite good job on ramping up and we uh gave more attention to fairness uh essentially because some of the synchronized workloads in AI some of them not all of

them uh it matters if you avoid having extreme unfairness and essentially uh leaving few guys at the end of the story like just hanging out uh lonely and having difficulty uh building up the rate or losing some time before they have recovered the entire bandwidth. So I'll get you more details now into the mechanism. Um we have a paper here you can look at has a lot more details with a lot of evaluations. ECN and delay are the u common stars for uh the entire show. Uh but they're not the only ones. If you look at the truth table essentially at the bottom uh the rows are corresponding to whether or not you got an ECN mark. ECN uh for those who might not be too deep in in that space uh is actually a little bit more responsive when you get congestion because um you carry the congestion signal on the packet getting decued uh even though that the cute packet might not have been congested itself but it

helps you learn faster usually of the network. Uh delay signals or roundtrip times if you like to measure them as such are more granular. They're multi-bit and we use them as well here to check whether or not we above a certain target uh that we'd like to achieve in our queueing latency. So uh basically our choice is when you don't or when you do have congestion whether or not your uh delay is uh excessive we choose to slow down. But again depending on how much or or uh the fact if you're below or above uh the target threshold we choose to go either gently or more aggressively on uh slowing down. Same goes for uh the case when you don't have ECN we do increase but depending on the status uh as you would expect it it would either be a gentle fur increase we called it or something a bit more rapid called proportional increase. The far right and left columns are corresponding to the cases when you do want to have uh

rapid uh window adjustment whether again going down uh when you have a lot of ECN or when you have a massive inflation in your delay or packet trimming or an any any scary signal basically can be used to say hey I might want to check what I'm getting on the wire and sort of a reality check to avoid like having the delusion that my window is big and I should send it be sending a lot more. I'm going to go by what has been actually acknowledged. That's the ground truth. And uh the the price you pay for a false positive is actually not bad. It should be supposedly close to what you're getting. We have a lot of work and uh results in the paper demonstrating how well that is uh designed. And vice versa, when you have very mild or or really low like no delay, no queuing delay, no ECN, it's a clear signal that well, you know, you're not anymore flirting with that uh uh low Q occupancy. So, it's time for you to go

and grab more aggressively extra bandwidth. Um I mentioned a couple of slides ago that we are using the supererset of ECN and roundtrip mechanisms. In fact, we demonstrated very nicely in this uh evaluation. We were fairing the number of flaws u running many experiments of incast and as you change them uh there's a place where ECN usually does well but a place where RT afterwards with many flows can handle things a bit more gracefully and uh we're showing the two versions the pure ECN in uh in orange and the pure uh RT delay based one in green uh and then we're showing smart in blue how it's actually getting nicely uh the minimum of the two functions uh more or less and sticking to the to the design really we wanted uh to or the goal we wanted to achieve.

Uh that's a more elaborate tree decision way of describing the mechanism. Did I get ECM? I mean was it ECM marked for the act or was was it not? And then I'm checking the delay. Is it u high? So uh the arch being the delay here is is bad or is it not? And then I do like a gentle decrease not a aggressive one and so on. So we'll not go through the formulas and bore you to death here. But basically mechanism is super easy. Uh has been in fact already uh implemented in hardware. Uh we demonstrated that we evaluated it. It's actually very nicely also um uh easy to do in a programmable programmable nick. uh I'm highlighting the two things again that uh also are very special for uh smart especially on the fast convergence and when we trigger quick adapt uh let me show you some more or or some uh results to demonstrate uh now how well we're working um for for this and for the next section I'll be talking

about load balancing we we're using a high-speed u network in simulation 800 gig it's a three-stage set minimal uh roundtrip time which less than uh 10 microsconds. We purposely chose the buffer size to be uh conservatively small like only one BTP. We predict in the future we will be actually getting closer and closer to that sort of uh scenario. So we wanted to stress test that and demonstrate how we can work there. uh we chose the target uh delay uh to be uh appropriately leading to half a queueing or half a bdp on top of the permit latency and uh the ECN's uh the kayman and kmax when you start marking and when you reach the 100% marking they are symmetrically uh chosen around that target RT queuing wise uh we have a lot of experiments I'm going to only show two here uh in the interest of time. Basically that particular experiment or the two experiments we're showing here were permutations. We're varying the size of

the flows. Uh in one we're running 2 megabytes, the other is a larger 32 megabyte. And uh you see here like the CDF on the y-axis uh for the completion times uh in the you see a light gray dashed line in each figure. This is actually the ideal theoretical ideal uh line for what your performance would look like. If everything is perfect, you'll be aligned with that line. The closer to it, the better. And uh you see smart in purple is actually uh doing the best in both. Um in in the figure on the left, 2 megabytes, Rocky, which is using DCN, that's what again uh is now popular in many networks, is actually doing not a bad job there compared to smart. You see the regression happening more on the right and vice versa. MPDMA which is a nice scheme in the literature lightweight and so on does actually run in 32 megabyte but not so much in the 2 megabyte and then you have things like EQDS the receiverbased scheme uh which

uh in this case because the network is over subscribed cannot really uh reflect uh all the signals very accurately to the sender and to the rate control appropriately because well it is missing uh where the bottom neck is um and of course swift is what Google does again we show in one case it's is doing good in in another case not so great again in some other cases we've seen it to do twice as bad but the point here is like you know smart is supposed to consistently and we have seen it to consistently do well across all the cases uh regardless of uh what scenario we looked at.

So now before I go into load balancing again I'd like to uh uh pave the way again for that discussion uh and remind you with what we looked at earlier. Uh we showed in the past HPCC and DCQCN how the volatility in uh your condition control scheme was not so great for your performance and led to u less uh or longer completion times. Uh here's actually what a simplified version of smart looks like. We will not get into the details of that but basically we did it uh in a short period of time. Um and uh you can see the wiggliness of the time series or of the window sizes it is you know much better and that's why you will see here your flow completion time actually will get better.

Now um we are still what like you know four 5% or a bit more uh of the ideal time. We're not saying well you know we should pour the ocean to to grab it. uh but in fact uh we did start there thinking back then as I'm paving the way for talking about the load balancing we did start looking at like okay well I mean random pack spraying is pretty much uh like almost ideal perfect for these cases everything is symmetric all the routes have the same capacity so just flipping flipping the coin randomly for each flow is pretty good not exactly the best but it should should do quite well and it is uh in other cases um it might not actually the best and and here's why uh we needed to come up with a better uh load balancing solution. Uh random pack spraying it's not reactive. Uh if all your routers are having the same capacity and all your uh uh players in the network, all your connections are using uh random packet spraying it might

be fine. But if one link has a degraded speed or if you're thinking about it like if some of the capacity on the second half of your journey is in the patter tree uh is degraded because some links fail uh even if you can actually avoid black holes if you can avoid these broken links and uh uh or if things are still perfect but you know some traffic is doing regular old-fashioned flow style flow level ECMP and and you're trying to be the nice guy here and do packed level low balancing either one of these cases will will clearly show you that well you you're not going to get the best performance actually you will have some major disadvantage and uh and that's why uh in in such scenarios you you will need to have something a bit more adaptive and and this is what uh we wanted to do here. So when things are perfect, we we actually do want to get something similar to random packet spraying u great performance. But when

you have asymmetries in workload or in the topology or link failures and uh so on you you do want to get uh something much better and we'll show how bad things can be with random packet spraying. All of that should happen while having a quite straightforward implementation in hardware. Which brings us to talking about u recycled and pack spraying wraps. that's what we have contributed to uh also QEC uh it's one of the two uh schemes that are optionally supported and uh we'll actually compare to the other scheme here anyway uh so just as we did with condition control I'd like to give you a high level overview for what is like it's a sender based so the power is at the host uh the switches and the network doesn't need to do really anything beyond what it already does today with hashing uh in fact I should say uh though we have evaluated it basically with hashing for the convenience just to

to bring it more easily to everyone's eyes in the community and say look you know you don't have to immediately migrate to sourcebased routing but we know and we have seen it to work uh very well with source routing schemes uh so it does rely on the same uh signal we mentioned earlier for congestion control ECN again uh the analogy I made about driving the freeway congestion can help you both decide what to how how much you you should put on your brakes and whether or not you should jump off and go in a different lane. Uh we do have some nice separation by the way between the two. Uh basically at the high level we defer the first uh uh task of removing congestion to load balancing.

We we we say don't don't freak out and try to be too tough on uh your window adjustment. Give it a chance. Let's see if load balancing will resolve the issue and of course it all happens organically without like any of these sort of complicated uh mechanisms. So organically what will happen is initially you are being a bit gentle on condition control. Low balancing should come and do its part. If uh if if it's not a load balancing issue because maybe it's an incast well control will anyway kick in uh in the right way it should.

So both are very well orchestrated. uh we don't uh require any like in fact there's a minimal state it's more of a cache state only we retain or keep track of on the sender uh which is not proportional to your balance delay product it's not a function of your link speed it's more a function of perhaps your burstiness or how uh much you're coalesing your traffic uh that will be clear in a second once I give you the example on the mechanism uh basically uh the secret is we're keeping the stick on the wire and uh we only momentarily cache it on the host uh for a very short period of time and then burn it right away as we put it on the next packet to depart and therefore uh that's a nice way for uh adding all that complexity on the host. Um I said it already it does complement smart and uh works very well um in in that case but it also actually can work well across the board even if you're not doing smart we've seen it

work very well with DCTP DC and and a lot more schemes. So that's really a one slide explanation and an example on the mechanism. It's a very basic one but yet very powerful. So um basically as I said the sender maintains a tiny cache of entropies. Entropies as to mentioned are the proxies for the route that you're going to take when you're hashing based. And uh you don't really keep track of all the entropies or all what your acknowledgements are carrying. You only look at the good acts the acts that are indicating there was no congestion. No ECN marking. uh if you're into the RT delay based thing I didn't have an inflation no packet trimming no knack it's just a nice thing that indicates no problem it's it's a it's a good entropy that I would like to see again hence the word recycle I put it into my buffer uh into my cache and basically uh the next packet that will be leaving uh my uh

nick can recycle it and if I have recycled everything that I recently added and I I have no more easy. I just flip the coin again and uh I move on to the next entropy I have not explored. Uh so in other words, if you're unlucky and uh you got congestion again, we're going to flip the coin again. And there's that tendency or affinity for you always to recycle what is working. And for the times you're not very lucky, you will flip the coin perhaps few times. Ultimately you you should u have uh some sort of asmtotic convergence as we have demonstrated in the paper. Uh so here's an example to make it more clear. Let's say you are sending 10 packers and uh apparently you have got no axe yet.

You're just start starting to do your transmission now. Nothing in the cache. So basically you do um arbitrary choices or you can make it like consecutive choices of entropy 1 2 3 all the way to 10. Basically despite them being consecutive keep in mind the hashing function the switches is uh doing a pseudo random effect on them. Uh so they will not really like go consecutive uplinks in a round robin manner. It can be very different but you chose these 10 entropies no matter what. And then uh in number two uh later on you got the first act and it happens to come for entropy five the fifth packet actually because you sent them uh on different routes they don't really have to be acknowledged in the same order that you sent. It's a green nice packet. No ECN marking in this example. Uh so great.

I'm going to put it in my cache. But because I'm a window based and uh I have more to send. Well, great. U the inflight packets now are nine and I'm allowed to send an extra packet right away. Fantastic. I'm going to send the 11th packet right away with the same entropy that has just came back fresh and nice to tell me. Well, you know that works. Why don't you try it again? Same happens again. Uh the second act comes for two. Nice packet. nice act the 12th packet therefore is going to take it out from the cache right away and again state is being maintained on the wire except for the moment between getting the act and sending a new data packet that's when you need to cache it third packet or act uh in number four it comes back with entropy 9 but I don't care what the entropy is because it's ECN marked I'm not going to do anything about it I'm going to save it later on I happen again to be allowed to send a

13th packet I look in the cache I have nothing good anymore well I move on to the next entry to explore which is number 11. That's basically it. And uh as you can see, it's very simple and very intuitive as well. U I'll demonstrate here in the evaluation how well that actually works. A similar methodology to what we had. Um we're showing here also permutation, but I'll show you a lot more more than permutation in the next uh slide. In the permutation case here, similar to what we looked at with smart and condition, the dotted or dashed gray line in the middle is the ideal one. You'll overlap it if you're doing very well or perfect.

Um, reps is uh the closest in that case. And uh next comes uh random packet spraying the RPS one that as I said you know in an ideal case things are symmetric and uh well it it should be quite good. will not be that far from uh the ideal performance. Then you see things like ECMP which is as I said the flow level hashing one we know it's not going to be great and PLB uh which is what Google does in their Falcon transport happens to be something I have contributed uh but it's a flow level one uh we're doing much better at the packet level load balancing now again looking at the two uh tightest contesttors here reps and packet spraying ground packet spraying orange and red I want to show you the case where Well, you know, uh that shows kind of uh your true great potential if you do have actually link failures or if you have downgraded link speeds. In this case, we only downgraded the link speed

for some of the links. And uh we wanted to see how well will reps do in that case. Um um very interestingly it sort of self adjusts on the on the route that have lower capacity without any direct kind of uh explicit may you know signaling from the switches about like oh I have you have to go twice less likely on that path than the other path or whatever it it's going to dynamically adjust and pretty much send the same utilization or if you want it's going to match the capacity uh the weight if you want is matched by the capacity of each link remaining capacity And therefore it it gets you the best throughput close to the ideal dash line above. uh random pack spraying in orange uh which though does very well when things are symmetric in this case uh is going to again flip the coin evenly on all the routs including those that are running at a lower speed and as such they will hit the ceiling they will start getting

tremendous congestion and they will continue not learning the lesson and saying well I want to choose the different routes uniformly and and hence they pay that kind of price costwise. Um one thing I wanted to zoom in even in the symmetric case remember here the red and the orange are very tight when things are symmetric. But actually uh you know we were interested like you know why does it make a difference here like there was around 10% or 7% improvement with reps compared to random package spraying and uh we show here the time series of the cues how they were evolving over time and uh you have two bands like a right and the left y-axis the right one is showing the Q size left one is showing the draining rate of the uh the ports and uh you can see in the lower figure with reps you cues are very nicely stable. Uh they are not underutilized. They're not hitting zero.

So they're all running at line rate and that's evidenced from the output. Uh whereas with uh random package spraying or oblivious package spraying you can see these kind of wild fluctuations in the Q sizes and every single color you see here has at some point hit zero which means the link was underutilized which is evident again from the uh throughput plots on the top. Um that's again the microscopic behavior sometimes is uh easier to explain helps you understand what's going on even though in general on average well I'm sending the same number of packets on every port but every mistake I make on a short time scale actually adds up and will cost me the performance.

Um as I said we have a lot more evaluations. This is a very dense slide and I will not go over every single thing here but starting from the far left we tried different incast experiments different permutation ones different tornado ones they're microbenchmarks and we're showing uh many of these schemes including uh MPTCP one of the you know classical works in that area nice piece of work adaptive rocky what Nvidia uh you know uh what what some switches uh do today adaptive kind of uh load balancing in the switches. You see of course reps PLB the Google one MPDMA something again nice and lightweight bit mapap is actually what I mentioned uh has been uh contributed to the uh to to the UEC as another optional feature uh the one in the inverted triangle in blue uh slit is some nice work from Aliza's group in MIT and oblivious text spring and of course ECMP uh basically you might see in certain in one these figures like oh

well there's a tight contender somewhere but if you check one of the other figures you will you will surely find that oh well somewhere else it doesn't do very well and and that's the beauty of reps it it's actually uh dominating the performance throughout all the scenarios we have tried by by a good margin actually uh you know there's no scheme that will not have that people that way more than 30% regression to compared to reps uh if you look in at least one of the experiments you will find something like that somewhere even if it does Well, in some other cases, uh, as I'm going to wrap up actually in this slide or the one after, let me bring this to closure with a nice uh, add-on we did for uh, reps and uh, basically u given the very uh, nature of reps uh, with the tendency to recycle things that are working. Then comes the very nice point of link failures. How does it work when a link fails and it's

black hauling and your routing engine or SDN or whatever it is has not cleaned it up? You still have that entry saying, "Oh, well, for that destination, I'm allowed to send on that port even though it's a disaster to send on it." Well, the good thing is of course reps is not entirely amenable when that at the onset of of that link failure to completely avoided or to avoid going taking a dip into the black hole. But the nice thing is once it's taken a dip in it, it will not really go again and recycle it for the very simple fact that it's not getting anything back to recycle. So it will uh cleanly keep on using these that are still functional and working. Uh so that's uh on on the first order of business. Now the second thing we realize here is like okay great since we're recycling and we're taking chances uh when we are having nothing in the cache to recycle uh when we have any kind of worry that there's you know a

bad thing out like we're losing packets you know like a timeout or some sort of uh things like that are happening we actually fall back to a conservative recycling scheme that when you're not finding anything in the cache you might be better still taking onto something that is working even if congested as opposed to flipping the coin and taking yet another dip into the black hole. That's what we call freezing. Uh you you you stop being aggressive on exploration. We show here in the bottom how reps uh when one link fails uh in a time series here how it takes a little dip and then you know it gets back up and doesn't really keep on uh taking a dip. In contrast uh obl spraying above you see like a main uh degradation uh because uh the problem is with things anything that is oblivious uh like random spraying it's not just going to like you know get some packets stuck in the wire and then you know that's it. It

will keep on adding more and more and more and your window is only bound to that much. So you're getting more and more of your throughput getting stuck somewhere and never unlocked and that's why you see like the degradation despite uh the failure being presently speaking you know not that much but you lose 3x or you know 3/4 essentially of your uh performance and takes you a long time to get back up uh consequently resulting in more than like you know around 40 to 50% regression in your completion time compared to uh so that's my last slide.

uh I uh you know we we have uh demonstrated again for the UEC context uh how ref does very well uh and as I said it can work very well also for force routing it's not carved in stone to be working only for hashing base we have invented in hardware and in fact continue seeing the great gains that we have seen in simulations u and again because of how u malleable graphs is and the fact that it doesn't really assume everything should be uh doing graphs for this to succeed. You can go with very partial implementations and start seeing gains for both the traffic that is doing uh reps and for the traffic that is not doing reps driving by it will still benefit from the lower cues and condition that uh reps will pave the way for. Uh, with that, I'd like to thank you for uh, listening and um, I know we don't have maybe much time, but I'm happy to take any questions in the few remaining minutes.

All right. Thank you. Any questions? Yes, let's let's take questions. The mic. Uh I think I think we've got you in the in the

So was any testing done with um nonpacket spray flows traveling through the network at the same time

like legacy flows we we we do have that I should actually have put a pointer to to a paper with uh a lot more evaluations and we did exactly that we varied the fraction of traffic without that is not doing graphs but doing ECMP and uh we showed that well you So the rest being reps and remaining being ECMP how they interact and how both that are doing ECMP flow level ECMP and how those that are doing reps are both benefiting.

Thank you. Thank you. Other questions in the room or online? That's that's really nice work. I was I was curious about where congestion usually happens and why is it on the receiver? is it on the network and and a quick comparison to the receiver based uh congestion control to sender that you should uh yeah so it is highly dependent uh you know on the on the workload and uh uh if your question is like you know in in our network where we see it um I've seen well I think I'll defer that particular part to people who are you know more hands on on the on our network but I've seen in some other operator that I'm not going to say the name and in much of the literature uh to be really very frequently happening uh on the edges like the sending or the receiving tours and uh essentially again if you look at it from both perspectives this is where uh the first tier of over subscription is hitting you the front line of drops

will happen there vice versa on the on the last hub the receiving tour that's where like things can gang in together and cause some incast uh at the receiver where our workload in some places definitely with storage does actually uh you know essentially lead to incast. It's no secret. uh and uh and and the other reason is your entropy as far as like the number of flows competing for bandwidth at the edges is not as high or as great if you want to look at the information theoretically as in the core where you have a lot more flows crossing the core of the network and you you you know the cross traffic the extent of packets being interled the hashing function having much more diverse input all of these things make it a little bit better usually in the core once you have overcome all of these kind of tears one after the other of being over subtracted.

I'm curious uh the HBC community has looked at a lot of uh you know sophisticated adaptive routing protocols that involve a lot of state on the switch and communication between switches and I'm curious how this sort of simple approach compares with those. Is that something you guys have looked at? Yeah, we have so I think the uh so in this particular work we demonstrated uh this you know I think hexagon blue adaptive rocky that is actually a switchbased uh adaptive uh routing uh where think of it like in a way you're trying to send on the shortest skew uh when you know among the different routes you are having so you try to load balance as such and um I don't remember if we did also look ahead one step uh forward again because uh usually what what's well understood uh in the literature is uh if you only base your routing on your local information uh actually it might be a disaster and many

uh pieces have demonstrated that so people are now starting to say oh you know let me have a look ahead where I don't just get the status of my next uh hop on the forwarding port being good but I'm going to get some signal from my neighbors next doors to tell me you know if I send to you am I going to be in good hands or you have some problem so that way I can uh reduce the extent of issues uh but even if you have that one step look forward it's still going to have in general the same challenge that well when you are choosing your path you'd better make an end to end choice uh because uh as you know as an example in the fat trees you go up the tip till the root or the top of that fat tree and then on the way back you know you have only one path on the way back. So you might do a greater job getting up climbing, but then if you have a broken link on that one path coming back later on, it's too bad. You cannot really uh

deal with this. Uh or if you have a congested one because of some local traffic having congestion again. Um that forward path looking thing might not do anything for you.

Yeah, just just to add to this, that was exactly the the main point that Abdul pointed out. I mean these schemes turn out to be pretty brittle in the in the failure case and in mega data center providers you have a different tolerance for the network going down

than in HPC right so this is much more reliable any other questions online all right let's thank our speakers

Questions about this episode

Who speaks in "Ultra Ethernet for next-generation AI and HPC workloads"?

Jacob Nelson (Host), Sarb (Guest), Torstston Hoffler (Guest), Abdul Kabani (Guest), Ten (Host).

What is "Ultra Ethernet for next-generation AI and HPC workloads" about?

UltraEthernet is advancing as a critical standard for AI and HPC workloads, highlighting Microsoft’s hardware-focused progress, system architecture planning, and collaboration with academia and industry to drive the platform shift in AI infrastructure. The discussion covers leadership, ongoing development, and the strategic direction of UltraEthernet within Azure and broader research efforts.

What is mentioned in this Microsoft Research episode?

16 people, works and ideas are mentioned, including UltraEthernet, Torstton Hoffler, Artificial Intelligence, Sarab, Azure, Abdul Kabani. Each one on this page links to its own page with every episode that mentions it.

How long is "Ultra Ethernet for next-generation AI and HPC workloads"?

The episode runs 1 h 56 min; the transcript is about 22,837 words in 164 timestamped sections.

More from Microsoft Research

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated October 29, 2026.