Transcript

XO Ruby Austin 2025 - Rag Demystified by Landon Gray

From Confreaks · Episode page · Watch on YouTube

· 32 min · 6,012 words · 68 timestamped sections

This is the full, timestamped transcript of "XO Ruby Austin 2025 - Rag Demystified by Landon Gray", an episode of Confreaks published January 12, 2026 (32 min, about 6,012 words). It mentions 4 people, works and ideas, including Chad GBT, Landon Gray, Dealmaker, UT Austin.

What is discussed, and when

  1. – 5:00: Dealmaker, Landon Gray
  2. – 10:00: Chad GBT, UT Austin

Notable quotes from "XO Ruby Austin 2025 - Rag Demystified by Landon Gray"

  • I'm Landon Gray. and I'm a senior AI engineer at a company called Dealmaker.
    about Landon Gray · at
  • Really interested in all this like stuff. And like fast forward like 10 years, I started playing with Chad GBT and LLM and things like that. and I sort of had in the back of my…
    about Chad GBT · at
  • So what I did was I took did some course work at UT Austin. They had a certificate course. I didn't do the master's program. I did that for a while. and then I learned how to like train models. And like during that time.
    about UT Austin · at

Who and what is mentioned in this episode?

People

Companies & orgs

  • Dealmaker — Company · 1 mention · first at
    I'm Landon Gray. and I'm a senior AI engineer at a company called Dealmaker.

Concepts & topics

Places & events

Full transcript with timestamps

So, you'll notice the name of this talk is different than the name of the talk that I originally had, but I'm gonna I'm gonna actually throw another twist on you in like three minutes. So, um, but first off, so the name of this talk maybe potentially, I'm not even sure yet, is Rag Demystified. Uh and first off, I want to introduce myself. I'm Landon Gray. Uh, and I'm a senior AI engineer at a company called Dealmaker. Uh I just started working there about I think I'm in my my fifth week now um as an AI engineer and then I worked prior to that a previous company doing AI engineering work and then prior to that I ran my own consultancy doing software consultant and AI engineering work. Um and so today I would like to present you with an option.

So there there's two pills. There's the red pill. If you if you take the red pill, I will we'll forget this ever happened and I will give the normal talk that I was going to give on retrieval augmented generation. Um, but if we take the blue pill, we can kind of deviate into a different sort of like conversation where I'll stand up here and I'll I'll share a little bit about my experiences um and we can talk a little bit more about AI more generally. So like um you know how is AI affecting um the market? what have I seen, you know, working at two different companies in the past like year and trying to do hiring strategies that I use to like land jobs. Um, do I think AI is going to take away all our jobs now that I'm working on agents? What kind of agents am I working? Like I will I will share all these fun things if we talk if we take the blue pill and that will look like you all directing where the

conversation goes. So, you'll be asking questions and I will just be saying like this is what I've seen in interviews. This is what I've talked to from different companies. Or we can take the red pill and I'll just give my talk as normal. So, uh by a show of hands, I'm curious how this is going to go. Who would like to take the red pill? Okay. One, two, three, four, five, six. Is I count that right? Keep put them back up. I want to make sure I get it right. Three. Okay. Three, four, five, six, seven. So, seven are saying you want to take the red pill. So, I do my talk as normal. And then, who wants to take the blue pill?

Okay. So, I think I see more than six. So, I'm I'm going to take the blue pill. So, fun thing, if you wanted to take the red pill, I'll share a link with you where I give the red pill talk. But, we're going to take the blue pill and I'm going to turn off my laptop and it's going to be self-directed. Um, so I don't know, maybe I shouldn't have done that. I don't I don't know. Like I I don't know if that like messes up the thing, but So, okay, we are like detaching and we're like we've taken the blue pill, now we're in reality. There's all this like hocus pocus about like AI is going to take our jobs. Like developers are going to way away. And uh Jim and I when we were talking I think last Saturday or Sunday when he was driving through I forget where you were driving through we were talking about like is this just like a bubble sort of like the crypto bubble and it's just like all smoke and mirrors like Phil

this is you now like this is you took this is you're you're directing this like I'm I'm just here. So any questions hands and then I don't know if we need like a microphone or something. I can even take this off now. I can just yell and make make it more. Oh okay.

See, this is what happens when you take the blue pill. Like technology just breaks down and like psych.

I have uh one request. Would you mind me opening the laptop because it's like flashing like the behind us and just keeping that open.

Oh yeah, thank you very much. But um Landon that Yeah, Landon. So uh my name is Ariel and uh so I'm I'm uh very interested in hearing your story. Can you tell us about your story because you know you're here as a senior AI engineer but there were steps that happened before that you alluded to them.

Hey there's like the step before senior there's the you know inexperienced engineer

there's that how did you end up choosing this path or did this path choose you and can you walk us through that story? That's a scary question because it's like kind of like how far do you want to go back, you know, like when a mom and a dad love each other very like like like

Oh yeah. Like this could go this could go really far back. So I guess can can you give me more comments?

Let's frame it from the perspective of what got you into AI engineering and how did you uh break into it

and how has your career evolved as a result of it? Yeah. So, I guess I'll go back to college years. I'll go back to Marvel. So, like I watched some Marvel movies and like I always thought like Jarvis was really cool and then uh I went to college, got a CS degree. Um didn't do any CS like model training or anything like that. And I audited a class briefly after college, like one or two years after on like artificial intelligence. And I had these grand ideas of what it was. And I looked at it and I wasn't that interested. Like I was like, I just want to do software development. I'm not really interested in all this like stuff. And like fast forward like 10 years, I started playing with Chad GBT and LLM and things like that. Um, and I sort of had in the back of my mind like, well, from that AI class I took, I had in the back of my mind this idea of like, okay, all these people are doing these model trainings in Python and I

don't want to train models, but like why can't people do that in Ruby? And then fast forward, I started playing with Chat GBT. And I had this like loose idea of like I don't know, it was kind of like two things. I was like I had a I had a loose idea of like a project I wanted to do to be like maybe you could use Ruby for AI stuff and then I also had this experience where I was playing with Chad GBT from like 5:00 one night till like 4:00 a.m. in the morning and was like, "Oh my gosh, like everything's going to change and like I don't know what to do with my career and like I need to pivot so I can be in this space because I'm really interested in this now." Like like what I foresee us being able to do with it. And you've always you've all probably had that experience where you started chat with it. You're like you're seeing different use cases about what you can do with it. Um kind of spurred

you on and and you want to like get deeper. And so like for me like that was sort of like the interest point was sort of playing with this model and kind of being like I want to deepen my skills. So what I did was I took did some course work at UT Austin. They had a certificate course. I didn't do the master's program. Uh I did that for a while. Um and then I learned how to like train models. And like during that time I talked about there was like a question about where Ruby fit in that I didn't feel like anyone was addressing. Um and like so like during that time I submitted a conference talk that was basically like I think I can train a machine learning model in Ruby and I actually told one of my friends about this. We haven't talked in a while. It wasn't because of this but like actually after this we stopped talking but it wasn't because of this. I I I told them I was like, "Hey, I think you know I can

train a machine learning model and use Ruby and people will think it's really cool and like rubists need this." And they were like, "Why would you do that and everybody uses Python?" And I'm like, "Exactly, that's why I would do it because nobody thinks I would do that." And so I submitted this conference talk and then I end up dropping out of the program doing that and that kind of was like my bootstrap thing. Um, and I did a conference talk at like Ruby comp and a bunch of people came out and there was just so much interest in the Ruby community in terms of AI because a lot of work had been done in Python. And then from there then on out like I just kept trying to see if I could like be more in like the AI space but kind of really software adjacent. Um, and I end up like quitting my job and like starting my own company and then like taking on all these projects and then taking a W2. So that that was kind of

like my journey. Um, and now I'm at another company doing AI engineering work in Python, but we also have a Ruby Rails application. So it's like weirdly full circle. Um, I don't know if that answered your question, but

thank you. I'm gonna try to be more brief, too. I like really struggle, but I want I want to hear y'all a lot more than probably me to some extent.

I'm gonna go in a little different direction. Yeah. Like going back to uh a little bit of what maybe your Red Pill talk would have been about.

Yeah. Uh so the place that I work at, we built a like uh real basic rag pipeline like

god two years ago like when it when it like like ear like relatively early, right? And uh

did you do it in Ruby? Just curious. What's that? Did you do it in Ruby? Did not do it. No.

Yeah. Yeah. It was all in pipeline. Um and uh and so the uh so much of the like design and architecture of the pipeline was built around the at the time like relatively small context windows that you had available right?

Yeah. Uh, and so I'm curious what the explosion of the size of those context windows has like what what are the implications for when you're building and architecting a rag pipeline? Like what's what is different besides the fact you can just send more data to the AI before it like loses its mind?

Yeah. uh like is it is it just that or are there other things that have like shifted in the way that you would build or architect a rag pipeline now that we have much larger context windows?

Yeah, I think having larger context windows is a really great thing. I think it's great for reasons that aren't like super obvious. I think a lot of times people think, oh, larger context windows, we can be sloppy and we can just pass more data like instead of like maybe the old context windows. I actually have a slide. Um,

I'm sorry, Landon, before before you go any further in your answer, uh, can you uh can you define like what rag is and a rag pipeline for folks that might not know what it is?

Yeah, I could do that. Hey, hey, we're still on the blue pill, right? Uh, so RAG stands for retrieval augmented generation. uh and it's a way to augment the LLM's knowledge so that um with additional information so that I can answer your questions correctly. If you don't do this, you get hallucinations is what you were talking about. Um and so since the context window is limited, you can only pass so much data. So what we do is we like pass little bits of data that we think have the answer that we want the LM llm to respond with and it will probably respond correctly. Um but we have to be very specific about how we spend send what data we send. But now with larger context windows we can drop a whole book into the LM and be like hey let's have the LM like figure it out.

It'll just figure it out right. Uh so there's still like problems with um are you familiar with like the needle in the haystack? Um for those who aren't, you know, if there's like a needle in the haststack and you're trying to find a particular piece of information, it can be hard to sift through all that data and you're like forcing the LM to do that. Um it so so that's like one problem is like it may have issues finding that um needle or if there's multiple needles say it's a multivariable like like answer that you know like you're saying like um what happens if this and this and this it's trying to find three bits of information. Turns out like the more bits of information you're trying to find like the worse it does and like it it it's challenging based on like where the information is located in the documents too. They so they found out like if the information that the LM needs is in the beginning of a document

or the end of the document it'll find it really well but if it's in the middle of the document sometimes it gets a loss. So it's like we still have to get into this more like rag for me the big thing is the retrieval more than anything. It's like how do you pull you want to you want to only give it the relevant information and how do you give it that so it has the highest likelihood of success. Now, let's say like that didn't matter. Like let's say like it 100% would always get the right answer. We got so good we don't even have to worry about that. Infinite context windows. Um so we can we can pass the whole internet in one in one swoop. And um so infinite context mode is another point I wanted to hit on. Um and we'll always get it right. Infinite context mode. You can pass anything in and it will always get the right answer. You still have another problem, right? So, if you're doing a

chatbot, it's freakishly slow. Like, if you put in the whole internet and tell it to respond, it's going to be sifting through that all that information for a really, really, really long time. And I've like I've dealt with that. Like, if you do a chatbot, no customer is going to wait 10 seconds for most responses. Like nobody's going to wait for that except if you're doing deep research and that and like you're like going away for a while and then coming back for deep research. But even if you weren't doing research and you asked something simple, it might still search through stuff for a really long time. So the use case really matters with the rag and and the speed of the rag response, this retrieval architecture response like really really matters depending on what you're doing. Did that answer

your question? You know, it makes a ton of sense. And that that the the retrieval heart part of what we built was the thing that was the hardest and took the most.

It's the hardest part. Yeah. And so I was I was just curious like, you know, if

if we had waited two years if it would have been different, but it sound it sounds like it's still it's still the hardest part.

Yeah. So we're still doing like the retrieval stuff. So all the retrieval stuff, everybody's doing that. Google, that's all it is. It's retrieval. So, so, so I like to think about it this way is like you know that little like crappy box at the top of Google where it gives you that LLM response. So, all the good part about that actually isn't AI. All the good part about that is they're curating and picking which links to rank. There's a whole ranking step where it's like, you know, you actually might have five right answers in the document, but three of them might be better. And so you want to rank those above the other ones and drop the other two even though those are still true to get even better responses. So like if you think about a medical context, you know, there's a bunch of white papers on how to like treat lymphoma, but like some of those resources are going to be better.

How are you going to rank that? That's like really hard. Like the LM doesn't know how to rank that. Doctors know how to rank that. You have to hire doctors to like read through these white papers and be like, well, this one's older, but actually the insights are more relevant to the type of research we're doing or, you know, this researcher made a mistake and like we can't really count that paper. Like, this is all like human work. Like, there's going to be like I believe there's going to be a crap ton of human work. Like, we're all going to be spending our time like automating all this stuff we don't want to do. And then like the stuff that's like really really hard like the last 2% of the hard human problems everyone's going to be spending 100% on that trying to like break through with the assistance of LM and that I think the value creation that could happen there could be like you I'm

like tangenting but I'll stop there. Get in the mic so we can record it. I want to hear

Oh, I don't have a question. I just wanted you to go back to a previous slide. Yeah. Yeah. Which one?

Uh the one where you were going over rag. Yeah. This one. Yeah. Just so we're, you know, have that at the forefront when you're discussing it.

Definitely. Definitely. Yeah. I mean, I guess I could ask a question. Yeah. As far as um I'm a baby coder, so I'm just learning how to code.

You could drop the baby. You're just a full-blown coder. You're doing this. I saw you over there in the terminal. That's VS Code. Like, you're you got this.

I'm learning test code right now. But anyway, um for someone, let's say, can somebody give her a microphone?

I'm sorry. someone that's not in tech at all, right? A commoner, an everyday person. We know what chat GBT is, right? So, you just breaking down this framework helps me better understand what's going on in the background, right?

Yeah. How can someone that's not in tech at all use these frameworks or chatbts or whatever better or more effectively at at the stage we're at, right? not with the intense knowledge like if you had to give me one or two tips.

Yeah. So I have found the a really good way. So the reason I'm giving this talk is because people kept talking about rag and retrieval augmented generation and like I had this conversation with someone. I I thought the reason why I couldn't understand what was going on was because like people are very smart and they're they're like they're explaining things too high level and I'm not a very like I'm a very like simple-minded person. So like the way I explain things I try to like really break it down into like terms people understand and like maybe other people aren't doing that. I found that problem in college. They'll use these specific words. I'm like, but what does that word mean? They'll use another specific word that I don't know was that and then they'll define it and I'll be like, I'm lost. So I was like, well, it can't be this bad. So like what I ended up doing was I kept looking at repos and just reading code

and being like until I could like understand it in like a way I could try to explain it to a 5-year-old. Like and if I didn't understand it, I would go find someone else's implementation and just keep digging and keep digging. You keep digging and it's like anyone could do this like especially now because what will happen even if you're very very new to code coding you can just take the segment of code and be like I don't understand this. I do this at work. I don't understand what this code's doing. Can you explain more about what this library does? and it saves me time having to look it up and or I can or it might give me a link and I'll go look it up and read the documentation and just keep digging and digging and just being super curious. Um but but the but the challenge is like you got you got to like pick the things you're really interested in because it's like if you there's so much out there's like

so much to absorb. For me I just picked retrieval augmented generation. And I just spent like months and months just digging and digging and how are how is this work and how does this work and how do different people and you'll you'll start to like it'll start to like soak in slowly. So that's my recommendation. Also yell at me if I if time is like

since we're taking the blue blue pill. So, building on what you just talked about, were there any surprises that you had where you kept hearing about how something works and then you went to go look at how it works and you looked at the code and you're like, "Wait a minute, everyone thinks and says that it works this way, but when I read the code, when I actually dig in, it's actually that way." Like, were there any surprises or what or did it actually kind of line up? I think most of those surprises come from like assumptions and like libraries that like sort of extract abstract away like the internals of what's happening like that's like very problematic. And then the other problematic pieces was like sometimes people write code to be efficient to like work but not efficient to read. So I just got really tangled up and I started like trying to make an efficient to read thing. But like one of the big

surprises for me was I'll just show you the whole rag architecture. Oh, this is super efficient. Inefficient. Click faster faster faster faster faster faster. Okay, I'm not going to do that. I'm just going to go to the slide I want. See, sometimes you try to be lazy and it's actually like more work to be lazy than just to like do the thing that this is like a big takeaway from LMS. So, one of the things that was like not obvious to me was like there's actually like more than one model involved in rag. Like I thought it was just the LLM model like claude and GPT and all that stuff. And I'll send you a link to like the actual talk in case you want to like understand the whole architecture. But there's this model over here called like an embedding model that takes text and turns it into like an array of floats that have like semantic meaning. And like it was like really hard for me to like wrap my mind around the fact that

that was like a different type of model. Like I just thought it was an LLM and that was it. Like things like that were kind of kind of trippy for me. I'm trying to think of any other things that were kind of

I have a quick question. I guess this is more uh looking at your knowledge from the perspective of someone who's deeply involved in like building the foundation of kind of what the future will be like you're working on these projects. So my question is I guess what is your opinion on the future of where this could take us? like for instance looking at AI or large language models and say it's right now a village what happens when it becomes a big city what happens when the student becomes a master and thinking not from the perspective of a short or even like maybe few year time span like 20 years down the line when this becomes so deeply involved with you know large amounts of data and collecting over a period of time I guess the question is um is there a point where this becomes so advanced where the internet is no longer a place where we connect and explore and discover because it runs on

itself when it becomes integrated with hardware cuz where is your opinion on the long term?

I think if that world exists, I don't think it's AI's fault. I think it's like our problem. Like that's our fault for not connecting and trying to build like community like this. Like trying to

I'm trying. Yeah, you're you're trying. Yeah. I I I don't know. I I think say say your question one more say say can you say like say it one more time just so I can

sorry it was a bit of a loaded question I guess uh I'm saying if we zoom out a little bit and we think you know ultimately the design of what you're talking about is based on building more efficiency building more knowledge over time becoming more efficient and becoming smarter gaining larger data sets so eventually like I'm thinking in long-term application eventually this gets more and more integrated the hardware. We see it in phones. We see uh auditory prompts being able to be provided via devices. We see devices using this technology to run without the need like for instance dark factories that have no power on and it has a large language model just running the machinery. Right? So at what point is there a point that it becomes so efficient that the internet as we know it now is no longer a place where we connect and explore in the same way because these hardware have become so prevalent and so knowledgeable that we

interact in a different way or we no longer use it in the way we do today. I guess it's a loaded question but I'm just curious your opinion on the future.

Yeah, I don't know. I'm kind of like I don't know. I I kind of feel like sometimes efficiency isn't always what we want. Like, you know, I've listened to like AI generated music and it's like it can do all these really really cool things. Sometimes I want the backstory. Sometimes I want to know that like Bob Dylan sat down and wrote this song and met Aikoyoko or I forget her name, but like and did all these things and like that's what I'm going for. It's not so much like the efficiency. It's mostly like the efficiency exists because I I just don't want to do boring stuff. Like everything that I'm working on, I want to be the most fun thing I'm working on.

And if the most fun thing I'm working on could be automated away, like I'm actually not gonna tap on the L1 because I want to have fun. Like, I'm doing it for fun. Like, you're not automating your AI. You're not like, if you're playing video games, you're not automating your video game play. Like, all right, guys. You know, I mean, people were doing that in World of Warcraft and Runescape, but then as soon as the dragon freaking came, you're like, oh, I'm going to sit here for the dragon. But then you have to ask yourself like well and this is a this is like a this is something I'm worried about with programming. This is like actually this is going to be not so great. It's like if I'm like see I think a lot of us here may be a little bit lucky in that you've been programming for a while. So now you're using LLM, but like if you if you're coming out and you're just using LLM to code, the

thoughts that you're having about how to design code and how to design UI and how to design different architectures based on your experiences with friends and families and problems in the world are like super novel. And LM's not like thinking about those. It's just taking, remember, it's taking I'm going to average out all your responses and then a couple of data engineers are going to curate a couple of them and that's what's going to be the result. But there's so many other diverse set of ideas that could like worlds that could be built, you know, that AI can't really dig into all those different things.

Also, we're going to hit a like I think we basically hit a wall with Loom. Basically, I I've been super skeptical about a AGI in general in general and I'm terribly skeptical about AGI in terms of LLM. You're not going to get there like like contextually like this whole motion I'm doing with my hands. How do you represent that like in text, you know, like how do you have you seen um Goodwill Hunting where the guy's like counseling this other guy and he's like, I've read all these books about like the cyine chapel and dating in five minutes.

Cool. And and the guy was like, "Yeah, that's nice, but you've never like really smelled what the Systeine Chapel is. You've never really felt love. You never, you know, watched, you know, the love of your life like pass away and all these ideas." And it was like it doesn't understand like it's literally trying to predict the most likely thing that you will be like that's great, you know, and it's like that's not intelligence. Like that's not intelligence. And it's it's really hard to get more interesting things if we're just averaging all of human existence, curating a few things and and saying like, well, these are the answers. Like there are so many more. I mean, I don't even know if there's answers. There's so many more paths the world can kind of go in to. So

yeah, I just had a quick question about the step between embedding and vector. Can can you explain those like the how we get no

how we get how we get to those floats like what's what's the whole process or from this?

No, just from embedding to the vector. Okay. So from embedding to the vector like

how does the embedding model work or how does the embedding model work? So the embedding model basically is trained I I like and this is see there's more deep I I haven't done as deep of an into the embedding model but basically it's like that model's trained so that a lot of like the semantic um I had some slides semantic meaning is sort of oh there's one slide I don't know where it this maybe it's not here so that the semantic meaning is sort of uh maintained. So like if in this list here where you have cake, water, milk, cookie, you know that water and milk are drinks and that cakes and cookies are desserts but cakes and cookies are also food, right? But like how does the model know that? Do you just pass it to that model and it sort of knows? No, it has to be trained and someone has to classify and say like, you know, hamburgers, pizza, these are all food and in like this embedding vector space, there's like a float

percentage point that's like a representation of like those things like like trying to keep the meaning tied to those words, but the words themselves don't denote that like drink and things like that. You have to train a model to like know and kind of numerically map in a you know embedding space that certain words are closer to other words um numerically. And I'll just show this one thing before I jump off. This isn't actually like how it is, but you can think about it this way is like when you're trying to plot the words on a graph, you can kind of see that some of these things are related to each other more closely than others. And it it's trying to do that numerically. So like sun, moon, and star are mapped are over there. So they're celestial bodies. But you'll notice in the middle, this is like a fictitious example, but in the middle, you notice star, starfish, and star trek are all close to each other

because they have star. And you you have to imagine like a multi-dimensional space instead of like a two-dimensional space where everything is sitting next to another word. And the things that are closest to each other are the things that um you know when you when you go to search for like a word like um like if you say like uh what's a star word like if you say star day or something like that you could probably guess where in this thing will like that will get mapped and then you will say well give me the closest three words to star day and it wherever it lands on this plot you just have to pick the three closest things and those are what you're going to end up retrieving because those are mo most semantically similar. But it's not just the word star. It might be like the fact that like the groupings like it might be a food and someone trained to be like these things are foods or the fact that these are high carb. It might

semantically know it's high carb. And you can train the embedding model different ways depending on how you want to like have the data get grouped together when you pass it to your the next model. Um, but I will link you to the actual Red Pole talk another time. So, yeah. Thank you. Let Thanks for indulging me.

Questions about this episode

What is discussed in "XO Ruby Austin 2025 - Rag Demystified by Landon Gray"?

In order: 0:00 Dealmaker, Landon Gray; 5:00 Chad GBT, UT Austin.

What is mentioned in this Confreaks episode?

4 people, works and ideas are mentioned, including Chad GBT, Landon Gray, Dealmaker, UT Austin. Each one on this page links to its own page with every episode that mentions it.

How long is "XO Ruby Austin 2025 - Rag Demystified by Landon Gray"?

The episode runs 32 min; the transcript is about 6,012 words in 68 timestamped sections.

More from Confreaks

About this transcript: the episode's spoken words as captioned on YouTube, split into timestamped sections. Reclip.Me locates every person, work and idea in the text, keeps the time it is first named, and quotes the words around it verbatim. Last updated January 12, 2026.