Research Retrospectives: An interview with David Ha, Co-founder and CEO of Sakana AI

What this episode is about

David Ha, co-founder and CEO of Sakana AI, discusses his journey from a managing director at Goldman Sachs to a research scientist at Google, focusing on his interests in complex systems and creative machine learning applications. He also shares insights into his personal projects, including interactive web visualizations and experiments with AI, highlighting the inspiration behind his work and how he transitioned from traditional finance to cutting-edge technology.

Key takeaways

• Interactive web visualizations transform the typically dry world of AI research into a playground for exploration.
• From slime volleyball to neural networks, David proves that fun and learning can coexist in tech.
• Visual tools can reveal nuances in machine learning models that numbers alone might obscure—don’t just trust the score!
• Boredom can be the seed of innovation; David’s shift from finance to coding birthed fresh ideas in AI.
• In an era of static PDFs, engaging web demos breathe life into research findings, making them accessible and fun.

Who speaks in this episode?

• Rosanne (Host) · 17 min speaking
• David Ha (Guest) · 39 min speaking

What is discussed, and when

1. 0:00 – 5:00: David Ha, Artificial Intelligence, Google LLC, Twitter
2. 5:00 – 10:00: Google LLC, AI Conference, Doom
3. 15:00 – 20:00: Evolution Strategies
4. 20:00 – 25:00: Google LLC, Evo Jax
5. 25:00 – 30:00: Google LLC
6. 30:00 – 35:00: Twitter
7. 35:00 – 40:00: Twitter, Google LLC, AI Conference
8. 45:00 – 50:00: Twitter

Notable quotes from "Research Retrospectives: An interview with David Ha, Co-founder and CEO of Sakana AI"

• Include complex systems self-organization and creative applications of machine learning prior to joining Google he worked at Goldman sax as a managing director where he c-ran…

  about David Ha · at 1:00

• Been somewhat of a growth and decline of web Demos in the last seven years I was part of this and recently I gave a talk at a v AI conference about some of these themes that I…

  about AI Conference · at 5:38

• When I was working on these World models like back in 2017 I trained the recurrent neural network to emulate Doom level it kind of worked the first time but by being able to…

  about Doom · at 8:05

• Digits I don't know about you folks but that's not something that got me into AI I was excited about Evolution because it's able to simulate things at different levels of complexity with traditional machine learning you have to Define like.

  about Artificial Intelligence · at 10:03

• Could also help with Intergroup collaboration so could you talk a little bit more about this yeah well EV Jax is actually led by my team member Eugene Tang it's it's like a…

  about Evo Jax · at 22:24

• A toll so I wanted to try something else it's like it's like at Google when you have a perf and then people want to get EES and sees every term and finance it's like okay you…

  about Google LLC · at 27:49

• And that eventually like stuck and like I don't know I think Twitter is a useful platform for me I get a lot of value from using it I don't get any value from Facebook Instagram for like not just professional life but even like catching.

  about Twitter · at 32:59

Who and what is mentioned in this episode?

People

• David Ha — Person · Main guest · 2 mentions · first at 0:53
  Primary interview subject; discusses his career path (finance → Google Brain), research themes (complex systems, neuroevolution, visualization), and tools/demos used to communicate ML research.

Companies & orgs

• Google LLC — Company · Business focus · 9 mentions · first at 0:53
  David works at Google Brain in Japan, discussing his research and projects there.
• Goldman Sachs — Company · Referenced work · 4 mentions
  David's previous employer before joining Google.
  Include complex systems self-organization and creative applications of machine learning prior to joining Google he worked at Goldman sax as a managing director where he c-ran…

Products & tech

• Evo Jax — Product · Case study · 1 mention · first at 22:16
  A library for evolution algorithms implemented in Jax, discussed by David.

Concepts & topics

• Artificial Intelligence — Concept · Core topic · 1 mention · first at 2:09
  Central theme of the episode, discussing AI research, applications, and future directions.
• Evolution Strategies — Concept · Strategy · 3 mentions · first at 15:59
  David discusses his interest in evolution strategies for AI research.
  Digits I don't know about you folks but that's not something that got me into AI I was excited about Evolution because it's able to simulate things at different levels of complexity with traditional machine learning you have to Define like.

Places & events

• AI Conference — Event · Referenced work · 2 mentions · first at 5:47
  David mentions giving a talk at an AI conference about visualization themes.

Media & podcasts

• Twitter — Media · Referenced work · 11 mentions · first at 1:20
  David discusses his use of Twitter for sharing research and engaging with the ML community.
• Doom — Media · Referenced work · 1 mention · first at 8:24
  David mentions training a recurrent neural network to emulate Doom levels.

Full transcript with timestamps

0:03 I haven't really prepared any slides but since we might talk about some visual stuff I I have a bunch of Chrome browser tabs open that I can share later oh okay um I don't think anyone in this year have shared their screens or even Chrome tabs you're by far the best prepared guest that we've had thanks welcome everybody to uh this edition of the research retrospectives Rosanne and I are absolutely delighted that David har could be joining us um I am very excited to uh to have a chat with you to hear your answers to uh the questions that um that we prepared but just to give H I'm sure David actually doesn't need much introduction but by way of introduction David is a research scientist at Google he works in the brain team in Japan his rearch include complex systems self-organization uh and creative applications of machine learning prior to joining Google he worked at Goldman sax as a managing director where he

1:11 c-ran the fixed income trading business in Japan and he obtained undergraduate and graduate degrees in engineering science and applied math from the University of Toronto H he is also on Twitter under the handle hardmaru the thing that comes to mind when I think of David is slime volleyball uh and this is like the first way that I encountered David's work and perhaps this is uh this might resonate with other other people listening um and so David I I wanted to ask um you make a lot of interactive web visualizations for example like making neural network agents in the browser how did you get started was this a personal interest or fixation yeah well it it kind of all started as a hobby personal hobby around uh six or seven years ago I was getting bored of my previous job as you mentioned in your intro and I wanted to do a bit of programming because I studied uh engineering before and had

2:09 some background uh I also want to learn a bit about AI as a hobby something that has always fascinated me but because like you know I was seeing numbers every day and I wanted to do something visual so I ended up learning uh learning a visual programming language called processing which also worked on the JavaScript and when I was learning about AI the the best thing for me is to learn by building things so some of my earliest projects here were like simple neural network experiments uh that I created using some of these tools and I hosted them on a Blog I started doing Simple Things uh here's like a like blog with some creatures uh I started doing some simple things like uh like controlling this uh chaotic double pendulum you know in in the web browser uh has to be able to to B B in some some like uh some cre random States uh that's that's like very unstable and you can the thing about visual thing is you can

3:08 make it move move and play with it and then as you pointed out I created this uh this clone of a slime volleyball game that I really like in high school but these were created control using neural network agents I ended up doing things like U like Evolution so we created these slime agents that evolved to avoid these killer walls and planks which is really fun and then ultimately I went to learn about some machine learning like so actually using using gradient descent to train things and back then I made these uh experiments to generate fake Chinese characters using recurrent NE networks this was kind of popular back then know thinking back it's all it's all trivial now in 2022 but around seven years ago these were all kind of new and it really got people excited uh I I was uh Happy back then because what made me happy was uh someone without any formal design education or background uh my work was

4:12 uh hosted or like I was invited to to present or have a solo exhibition of these Works in in a design gallery in Korea so I don't understand what they're saying but they made a poster from work and and like there was this bus station where they had a poster know in Korea and there's like some some boy band next to them who I don't know who they are that that was really cool and then I decided to like it was time to to do some something new at that point I find it so fascinating that you know throughout this what seven eight years of you doing research sort of like demo and visualization is a consistent theme up until today you're the only guest that are you know talking with this sharing screen in an interview because you need that to a your introduction so that's so interesting also this is quite an achievement thanks yeah downhill from there I think uh visualization as I did more like a like a proper research after

5:16 joining Google I thought it was quite important discipline and it's not just me there's other groups like peir who who are deeply study visualization but I think the thing that I I try to do the most is to communicate research findings using visual Tools in particular the web browser there has been uh somewhat of a growth and decline of web Demos in the last seven years uh I was part of this and recently I gave a talk at a v AI conference about some of these themes uh that I I think what what I learn over the years is in machine learning is not just about the the numbers uh but also about you know how to interpret those results and it's for me it's always good to be able to uh to play with the models look at what they're generating rather than just looking at the performance number because sometimes they they tell us a lot of things like a robot that beats the previous score might still not be

6:18 able to walk right so it's able to like I was I like creating these uh these like some of the the the over the years created some of these like models and put some effort into making them usable like so that we can interact with them and I think uh this is an important theme and sometimes we can get some research intuition just by playing around with the model yeah I'm so glad that these days every ml project has a web page that you know shows more than just static PDFs back in the days every ml project is just a PDF and even robotic projects like in in this figure one they just like snapshots of the robots but you don't see them moving at all is really not showing up like the best part of the work so yeah I think like you in the right um train of thought that that research is better presented with moving things yeah yeah I think uh I mean not not all research can be done this way like there's important lines of research

7:18 like like ml Theory there are many fields where this doesn't apply I just happen to work on the areas where it's possible to present some of the results visually uh and and it does take a lot of work as well like because like we're all very busy working on the papers and so it is a trade-off between like working on presenting research especially on the web uh versus just uh you know like working on a PDF of the paper and then moving on so I'm I'm also curious do you ever find that you learn more about what your policies have learned or are capable of doing through this work to visualize them or to like create and communicate it yeah yeah I do like for for instance when when I was working on these World models like back in 2017 I I trained the recurrent neural network to emulate Doom level it kind of worked the first time but by being able to interact with it I was able to to learn things like uh adversarial RL like

8:24 how the agents are able to to cheat uh in the game and interactively what I did back then was uh adding some new knobs to the to the to the inter to the experiments we can do things like a increase the the temperature or the uncertainty of these simulations by doing these interactions uh you can figure that out that the the agents can can learn better in in a higher temperature setting and without this interaction I may not have discovered or or come across these results and these findings have also made their way into the publication as well so so there there is like feedback loop uh between the the the interaction and and also the research and I was lucky to be able to do these inter interactive experiments where I was able to find things from it but I don't think uh in general that's the case though let's talk about new Evolution what made you first um interested in New Evolution what made you hate

9:31 gradients I don't hate gradients though I I I love like you know machine learning I even did did when I started like Tred to learn a bit of machine learning I think so back back then when when I got into to AI what what excited me was was about you know like making neural networks control things like play games control robots you know Control Systems I came from a control systems background and this was before deep RL became a thing like it's hard to get excited about like classifying mless digits I don't know about you you folks but that that's not something that got me into AI I was excited about Evolution because it's able to to simulate things at different levels of complexity with uh traditional machine learning you have to Define like an entire pipeline uh of of systems your your system has to be relatively well defined engineered but at the same time I felt that might limit the creativity

10:34 like I'm I'm interested in looking at uh when we're looking at things like a complex systems there could be multiple levels uh of of hierarchies even the machine Learning Systems you have to have like a know hyper parameters and everything but with with something like a blackbox Optimizer like Evolution you you can you can just think about the problem at hand and think of creative ways to solve it without thinking about all the gradients uh involved uh to put everything together so I think uh I I when I when I started to research I I I like to use these Evolution tools because they were just they was just easy uh to to try things out like I I worked on implementing some old Evolution algorithms uh called a the nits algorithm it was sort of a neuron Network morphology algorithm this was before AR neuro Arch Ure search so so I created like some web demos that can evolve like these classification

11:36 networks uh and and eventually I think back then the Jeff Dean saw the blog and I think that they played a factor in in me getting hired uh because that that may have inspired the work in 2016 on neural architecture search uh so I I do like to uh use a lot of ideas from evolution uh in the research I I like uh like for for instance even when I started to learn about like a a bit about machine learning when we learn about Mist I was not really interested in learning how to do the classification I'm more interested in generative aspect like maybe building Gans or Vees on mless right so one one of the early projects I did was to combine Concepts From Evolution research like in particular these cppn networks that can generate really high resolution images and combined it with with mist uh you know like a generative model so so I think there's there's many interesting directions in in evolution artificial

12:43 life and complex systems and and I think some of my Works uh were combining some of the best ideas from these FiOS with with machine learning to try to push push the needle in terms of what can be done yeah um for those who don't know um CBN and all the new evolutionary work it's quite fascinating as to remember this blog post that you had this giant endless digits maybe this is maybe this is one of them because in cppn there's no such thing as resolution really you can train on small things but then when you generate you can use arbitrary resolutions so yeah yeah I think a I have a slide here yeah so this was one of the early slides that got me into AI like is 2013 uh it was about one of these is cppn generating all these images on on the rights and the others are controlling finless robots and computer games and I ended up like creating this as you mentioned this NS and cppn networks you

13:46 can see on the rights this particular Network defines this spaceship thing on on the left uh this this algorithm was created by Ken Stanley who you and I know and and as a learning exercise back then uh I I was working on uh this uh the same algorithm called a pick breeder but with a recurrent connection so these these uh we can mutates these uh these drawings and you get different levels of complexity at every generation of having me as a human on the loop defining what's good and and what's what's not good and uh then if you can see like these simple networks you can get these uh these kind of cool designs and eventually I created like a bunch of interesting designs and hosted them on a gallery so I think the C cppn tools are are back then um they they were quite fascinating I was able to combine them with generative models and I think even now when when we look at uh uh some of the modern approaches they use a similar

14:53 ideas in machine learning now where you don't generate all the pixels right you have to parameterize the image with the with the with the the coordinates and the angles to to fit lots of data in so that this is a pretty cool approach yeah I I I don't know they they never really made the link but positional encoding in Transformers I feel like is very similar to where cvpn has started yeah so lot of these early ideas are pretty pretty importance so do do you think it's fair to say um that even though RL has Maybe started to work yet you are still uh still very much in The evolutionary strategies Camp um and given this I'm like curious what you see for the future of your work in evolutionary strategies and and kind of the field in general yeah think what what was interesting for me around 2017 was that there there were starting to be many papers uh about using Evolution strategies which is like you

16:01 know like a subset of all Evolution algorithms uh to to apply them to RL benchmarks and uh that really caught people's attention like they were able to a few of these papers I one of them was from Tim Solomon who who might be here uh was like you know you can you can show that very simple techniques uh that scale well like in evolution can achieve really good performance compared to RL and this is something that's uh stri me as uh I was not surprised uh because I I feel like uh the the simplest algorithms when scale to to you know like a to large large scale can can solve most of the problems and this is something that we learn in ml as well and this is something that maybe uh Rich Sutton also believes in in the biter lesson when we're trying to Simply find Solutions uh to arbitrary neural networks uh it's the best tool for the right tool for the right job so maybe if if it's a very like a a well-defined

17:05 problem like image classification uh it makes sense to use gradients but in in reinforcement learning it becomes a bit more trickier right because like the the credit assignment problem this is the entire field of RL and what what I'm I'm interested in is not not things like sample efficiency like because like the a big set of RL research is devoted to trying to make make an agent train in a 100,000 time steps rather than a million time steps I'm I'm more interested in in just getting a solution for an arbitrary system uh to do something that was not may not be possible earlier so so in that sense like I'm more interested in defining like novel problems and and getting some rudimentary solution to to First solve them first and so I think uh in that sense that that these Evolution strategies algorithms are easy for me to to try new ideas and and to iterate on the problem side whereas maybe the RL uh

18:14 approach is is more useful when we're trying to to make learning more efficient in terms of applications I think uh many people in RL uh particularly for example the even the brain robotics Locomotion team uh they have adapted many Evolution strategies even our own libraries for for their their research because like if you don't care about the learning efficiency you you just want something that works uh in in robot controllers it turns out that Evolution may as well like produce some good Solutions in evolution research it's like some of it's like the Opposites of RL research in a complimentary way or ml research like the the problem with Evolution research is maybe no one's trying to aim for state-ofthe-art uh but everyone's trying to do novel things so so rather than getting better Evolution algorithms you you end up with newer Concepts like cppns as you discussed earlier or quality diversity uh or a novelty search

19:21 these uh these kind of Concepts eventually make their way into RL where the the RL community will will make them perform better in a sense so I think they're somewhat complimentary yeah are they talking enough though because there's like the the neuro evolutionary um neuro Evolution communities are like the aive the gcko and then mL of course we have our own community and RL Community do you think the the communities talk to each other I I don't see that happening as I do see some more Synergy I I think for instance like a new rips two years ago right before the pandemic there uh there was like a workshop on quality diversity and evolutionary methods given at NPS and with with fairly good amount of attendance uh in the RL track in workshop at NPS I think many many people like people you we know like the the Jeff CL and the Ken your work has been presented as well yeah well exactly like Jeff and Ken and Jo had to do so much to

20:29 sort of like introduce to the ml Community what evolution is which is that they exist for a really long time so feel like it seems to still be hard to for these two communities to connect of course we've seen progress yeah yeah we I think inside Google or alphabet there has been a trend upwards like like at Deep mine uh they have started to publish papers at Gecko since around the time I joined uh there's a dppn paper from Chris antha Fernando the the people in neuroarchitecture search team like estabon Rio they have published papers uh related to evolving several architectures and th those have become well cited but and those have published in like mcmls and newps type conferences as well yeah for those who don't know you should publish at Gecko because geeko is always at Fantastic locations I think of course they've been virtual for the past two years but like every time ml conferences like Baltimore or like

21:31 Louisiana but then you look at geckos like VI and maau not that that's the only reason to publish but just keep in mind G Geto is probably my my favorite conference uh Geto is is tough as well I think the the review process is not easy I think the people are are like just as strict as I I clear I think but uh but you do get a lots out of attending and a lot of good feedback on the research and to to connect with with people who kind of know a bit of evolution and and machine learning and so think thinking a bit more about uh how perhaps RL ML and The evolutionary communities can uh work more together um you've been working on a library called Evo jaxs um and perhaps this this type of infrastructure uh could also help with Intergroup collaboration so could you talk a little bit more about this yeah well EV Jax is actually led by my team member Eugene Tang uh it's it's like a library for that has several state-of-the-art

22:39 Evolution algorithms implemented in Jax uh and this this Library I think it it was launched like just just three months ago uh he shown that we can accelerate uh traditional training pipelines that might require like like a week or maybe a few days of training into several hours like the same thing the theme that we have seen is the in success of ML and RL is you want to have tools work on accelerators uh then you can iterate the training time uh of uh of the research and that that will ultimately lead to better iteration and and progress so I think this these tools uh can be can be affect factor in helping researchers speak a Common Language I I also think uh the Jack ecosystem is pretty good uh for this like like in general uh the design of Jax is is kind of brilliant in the way that it it it doesn't pigeon hole itself into being a deep learning framework it's it's like they just we're

23:50 just numpy with gradients if you want them and we can paralyze the things to like tpq pods or multiple gpus and you can use them for whatever you want but we're not going to become like a full-fledged deep learning library and we we've seen cases like in the science communities use Jacks for this purpose and uh we're trying to use them in in the complex systems and evolution and I think uh having this common language of numpy would enable better like a collaboration and cross-pollination of ideas for example it's like some some people have have uh used like Evo jaxs or or extended uh in their own libraries to incorporate quality diversity algorithms uh with with reinforcement learning so I I think this uh this type of a the open source uh approach to ml using like these uh less uh specialized tools is ultimately a good thing yeah St to like how to get communities to talk to each other first you establish a

24:56 language and in our field in research language is just tools that we use to program so I think it's a very significant step in hopefully like connecting multiple different communities together yeah I hope so too like uh there is like the the concept of I guess as Sarah hooker mentioned the hardware Lottery and and whether we like it or not we're we're stuck with a whole bunch of tpus and gpus so they yeah try to unlock them yeah and thinking back why like um new Evolution never really cut on like before um Ken and Jeff and Jo's effort was like they don't work well on gpus right they only work on CPUs so I I hope like with this Jack's framework and tpus and you know more awareness around how cool these tools are this can really generate some effect yeah I think it's we're gonna see lots of exciting things ahead you know so many different directions so we talk about all these interesting directions you were interested in but

26:04 let's not forget that you came from Finance how did that happen how did you get into finance and out of finance and what did that Journey teach you when I graduated from University a long time ago in uh in uh engineering and control theory I think I I almost got into grad school uh to to do like uh to actually properly study control system and optimal control but back then it was a recession and then people were scared of you know like maybe they can't get a job so what what I did was I wanted to you know like make some money so I went into the finance industry uh particularly applied some of my skills to the derivatives markets like uh where I worked in in options uh derivatives modeling and ended up becoming like a trading uh these like these uh financial markets and eventually moved away from the quantitative side but I realized that all of this quantitative stuff in finance it's it's like maybe it's

27:01 important at the micro level you know when people want to you know trade stocks at you know at when you're next to the exchange or algorithmically but like it's like a chaotic system you can't really predict things maybe one month or six months from now so so I I eventually uh moved on to macro trading and like as I moved higher up like became like a uh running the the fixed income and macr trading groups and it's it became more more of a real job but then you know you're working with things like like uh the central banks various governments uh to make and and like when they're issuing government bonds and so on and uh eventually like uh it did take a toll so so I wanted to uh to to try something else uh it's like it's like at at Google when you have a perf and then people want to get EES and sees every term and finance it's like okay you know you have to make all this money every single year so or you're out so at the

28:09 same time you kind of get bored of that and just try try something else and I that's when I I wanted to like you know I had some Hobbies back then and I just decided to leave around 2 15 just to maybe try I try like maybe perhaps become a Adventure capitalist or something like make a make a fun but luckily I was a like there was some opening at Google brain for for research role it was called the brain residency back then and just for fun I applied to that program and and was accepted in the first patch so this is how may maybe maybe it would have been better if I didn't get accepted and you know became a became a VC and I got to ride all the crypto and all all of the way yeah could have been a never know I always find it so interesting how these moments of serendipity can change our lives in such dramatic ways and it often it's striking how also how often that story comes up for many different

29:14 people yeah yeah I I really enjoyed the brain residen program it's a shame that it's gone now I think the very first batch we got really interesting people like like for example Ryan daho like the the creator of node.js he's kind of a head legendary hacker joined our batch we had people like Laura deing who who was a Theo grad uh she's someone who who who quits her MIT degree uh and got like maybe 100K or something to do that and then she successfully successfully founded uh a venture capital Fund in genomics and she was in the first batch of the Google brain residency so there and we had many successful research researchers who who are still here at Google maybe here in this call as well that that worked their way up after the first batch and and became like a well known in their field so so I think uh it it was a it was a pretty good program that addressed the need at the time maybe maybe the the

30:20 foresight that maybe the lead management had was uh you wanted to bring in people who were not like traditional uh phds in statistics and Cs and and throw some new ideas at the problem and in in our batch like uh we like we're like we're able to to publish things like change the field like a neuro architecture search for example uh there there's um and also many generative models uh were were were kind of pioneered from from the early batches of the brain res prescy so that that was kind of cool to give example when when I joined I think it I was not alone but I had no idea what NPS or icml was yeah I was like a paper to me a machine paper was just some PDF you found on archive and then like like you you would just kind of like hack around other people's code so that it it very interesting and then like uh people had to to I guess like ultimately you need to to publish to to move things forward but I think it's

31:33 it's also a good thing though because like without without some sort of like a like publication uh medium uh like like maybe conferences is one thing or maybe a blog post can be can also be just as effective uh you you want to to organize the knowledge that you gain from your insights of running experiments and and present that insight to the world so I think conferences and Publications are still good but you know as as you you know uh maybe we can talk more later about like there could be alternative ways like in especially in this era of social media to to presents research as well yeah totally collab notebooks do twet did that few days ago like what you should publish that make that citable um talking about social media I feel like we cannot get away from this conversation without talking about Twitter your Twitter Fame I'm not that famous W Among Us I think when I started using Twitter

32:40 a very long time ago it was for like things like like what political activism and stuff like that like a very long time ago and suddenly I I started doing these ml blogs and I I posted some of my results and blog post on Twitter and I I I became became part of the ml Twitter uh and that that eventually like stuck and and like uh I don't know I I think Twitter is a a useful platform for me I get a lot of value from using it I I don't get any value from Facebook Instagram uh for for like not just professional life but even like catching up with with what's going on around the world and I'm I'm probably too old to use like Tik Tok or or like are there are researchers on Tik Tok you know like talking about research papers I've seen that don't use it I've heard that you can do that I I can't make a Tik Tok video of I can write like what Twitter is good at is like a it it forces people to to write in chunks of like you know

33:47 280 characters uh without the ability to edits uh for now anyways before Elon mask whatever we uh you know like yeah I think like that that really like is a is a useful skill to to compress or to to make you like you know say as many things as possible in with as little text as possible even with Twitter threads uh you have to organize threads in in such a way to to bundle the knowledge of the paper and then uh you can do things like you know the I think 80% of people will only look at your first tweet of the thread and no one will read the rest of it unless they're interested so it's also important how to communicate but I I think a scientific Twitter and ml Twitter is really uh fascinating like like most people these days uh they they want to not just in ml but in science they they want to write a tweet thread you know about their Publications and and like it seems Twitter has maybe Tik Tok in the future

35:00 you never know is is becoming like the the how to say the Town Square like as like many people say now of the facto Town Square of science like and Twitter has become the Town Square of many things like politics and news and in Twitter people can learn to to defend their ideas as well where some some people will question you and you have to learn to to strategically defend your ideas and these skills might be useful in in academic discourse people should use Twitter to explain their works I think that they get their work notice uh the the way I use it is at the beginning is to to tweet about some stuff I do but I I like to to tweet about random stuff like like what what other people are doing uh in particular people who are maybe like the the research that I think is cool but no one is really talking about it so I I try to tweet some of some of that stuff as well uh and I think maybe maybe some people like

35:59 to follow me to to get some weird perspectives on on on Research rather than follow like if you want to get the the state of the art research you follow the Google AI accounts and open AI accounts and then that AK guy who tweets all these papers yeah yeah well recently would be people would follow you for the Del pictures did this is so addictive yeah I think like oh my God I I don't know I got like someone I didn't ask for like someone put me on I guess it's from the pur of Twitter you're well following is someone put my name onto the the waiting list and I got a early approval I started to play around with it was this is pretty amazing I wish I think I think Google should try to do more of these Outreach as well like like U cab is a great way I think but having having demos you don't have to be like run on the web browser on the client side but but having having like a some backend effort to to serve

37:08 models and to get people to play around with it with sufficient disclaimers so does what does the future of research look like should the should the PDF uh paper should that die or is there a place for it in in research communication uh in 10 years time yeah I I've been thinking about this this you know like a lot as you know even the talk that I've given at the i e Vis an AI conference like I I do think the the presentations of papers have are shifting like uh in particular I I think like some of the work I did uh have set the trend as well like like for instance uh I did papers like the world models uh and the weight agnostic neural networks where we we took the distill Pub templates uh and it was fairly easy it to if you didn't have to work on any interactive demo it was fairly easy to to put uh the archive paper as as a webp paper uh and it it was now it's free to host as a github.io web page as well so

38:26 so in in addition people can register their github.io page and I've seen many papers do that now like not not just me I think PE some other people in Google AI uh other groups other other groups academic groups are doing the same thing of putting uh either a blog post or exactly verbatim their paper online uh and that will increase the the Outreach of its uh I I do think that there is a balance and like personally I I do continue to publish the paper on on the archive uh I I think that's important uh because it's it's like a it's become like the archive is kind of like the the Twitter for for academic Publications people want want their their time stamp on it even for published papers people end up reading the if they want the PDF of it they're not going to go to the ACM website to read some paper you're going to read it on archive uh even even maybe even if nature papers people want to

39:34 find it on archive so so I don't think the traditional papers are dying U I also appreciate a well-written uh paper on on like traditional formats like in the PDF uh because like with with online websites especially with like the lots of visualizations like like sometimes your phone loses battery is very fast so you you want to to have uh a place to just concentrate and read the paper sometimes like even print it out occasionally on double-sided paper uh to to properly study it so I I don't think it's going away I think these new tools like visualizations collab demos they they could be extensions a supplementary material uh or even the main paper I I would argue maybe the PDF is the supplementary and the the online version is is the main material and I think these are all complimentary and and people I think there's there's some push back as well like people want to okay this takes time we should all focus on

40:42 just publishing and moving on but this is kind of like entropy you can't really move back right like we're already moving in this world like people are not going to stop using the social media right we're not going back to the Bell Labs era right so this is is something that even if people don't embrace it they should be aware of of what's going on and and so that they they can possibly adapt their their strategy uh when they're trying to uh you know like aim for more impact or beyond the research Community for for their Publications so you've lived in many many different places David uh so Canada the UK Hong Kong us Japan um and we were curious what do you miss or not miss about these places uh and also how is the research culture like in each of these places or how is it different from place to place yeah I mean I I spent most of my life first half of my life in in Canada and to some extent the US and I think I

41:43 really like the the openness uh I think people are generally more open to to new ideas less of a hierarchy uh U UK is really great I think maybe that's one of the reason Deep Mind stay there it's like I love the museums there there there's there's so much culture uh that and there's people of of all ethnic cities live in the UK that's something I I miss as well like having having a a culture that's less less like a mono less of a mono culture I I think that in my experience the the West Coast in US suffers a little bit from that the monoculture uh but in a place like London or UK you see or Europe you see more of a of a of a mix of a variety uh the Liv living in Asia is good experience uh the the food is great here and and I I think um it's good and bad like sometimes you get disconnected with the rest of the world but at the same time you have this this bubble that where you can concentrate on on your own

42:52 idea like a gapos island where you you can you can de develop things in somewhat of an isolation sometimes if you wants without being too entrenched with being have to part be part of uh all this other work so so that that's uh from a research culture standpoint it's it's also like it's a mixed thing I think you in the past you see Japan develop many Niche technologies that work really well but but it didn't really integrate well with the rest of the world and they eventually lost outt so uh so it could be good for early stage Innovation sometimes but but ultimately it's it's also important to to be able to to mix and adapt to the rest of the world do you see yourself living uh in the long run in Asia or in Japan my family now I have two kids they're they're in Japan so I don't I don't think it's easy for me to move so I'm probably gonna be here for like did Asia as a base for for the long term but one

43:59 like I think the pandemic is winding down so I'm I'm gonna be traveling a lot yeah yeah speaking of family and kids a lot of AI researchers say that the best inspiration you can get for AI is by looking at your kids do you feel the same way yeah I think so I tweeted something like this uh yeah I love I love this maybe was you you see you see office this this is uh sorry sorry yeah I think I my my kids gave me a similar like uh I tried to do something similar with my kids like just just let them play play play and observe how they play and I I'm just like excited at at how fast they can they canar exploration amazing great insiration strategy here exactly they they learn to cheat yeah they they get what they want somehow they don't have to know how to do it but they just they just can explore and and get what they want so I I think yeah I really like having my two kids around especially at the toddler

45:04 age and it's a lot of fun uh it takes a lot of time but but it does help me think about some of the bigger picture things yeah I wonder if it's just for AI research or any researcher would get inspired from observing their children yeah I don't know I I'll wait to to find out from you um what about time management I mean men probably don't get that questions a lot but like do you still find you can concentrate on Research when having cols running around yeah it it's gets a bit difficult like sometimes uh like after they sleep at around 10 p.m then you have like like a like from 10 P.M to 1: amm you can you can read some papers and catch up on some books uh so time management it's very difficult uh one one strategy I learned recently is is to rely on Audible like using audio books uh so I I discovered there's there's so many textbooks on on like academic subjects that that are actually as available as

46:12 audiobooks so if I need to go to the gym uh then I can just like listen to the some audio book and listen to someone talk about like entropy or or or like complex systems and why why like do any part they deal with equations do they just read the equations out it depends on the book I think uh they have figures uh usually they comes a free uh they comes with a free companion PDF uh and then you can like look at all the figures before the chapter and then start running listening to that chapter it helps a lot I think I'll have to try that yeah I don't know I can I already cannot concentrate even listening to fictions I am I going to do ta I'll be bored so much just run slower I wonder if it's a great exercise in like mathematical visualization though that you sort of have to really really work hard to try and imagine the Fig maybe yeah if we're listening to textbook what's the next are we going to

47:14 listen to code at some point some read out the whole Jack's code for us and what we exercise oh my God so one of the topics we wanted to spend a little bit of time on um was about how you think about your research Vision um and in particular for perhaps anybody listening or myself included who are thinking about how to formulate uh their own research Vision uh and agenda do you have any advice like how is it how did you come up with yours how do you think about what to work on yeah I think over the years I learned that like it's important to come up with um like a long-term vision and also short-term strategies as well like uh formulating a thematic longterm vision is something that's that's seems to be important in an academic or even industry research and and in my case I'm I'm really interested in the area of complex systems like looking at how we can build networks that are like more

48:19 like a like ants colonies making a bridge rather than like engineering like like a very rigid Bridge so that that's for me that's a a long-term uh like it's a vision over the next five to 10 years but short term we I I have to look at many strategies of like what what topics specific topics to explore that could be stepping stones to that vision and I think I I I actually tweeted something about this I found we're talking about Twitter uh earlier right so for for how how I find like a research search directions is like because so many people are looking at benchmarks and chasing state-ofthe-art it's actually somewhat advantageous not to do that uh because it's like the efficient markets right you want to do the things that not everyone else is doing so things that I like to do is looking at task or formulating task where humans are okay at doing but the current methods cannot do at all and we're trying to come up

49:21 with new machine learning algorithms that can perform okay some of those tasks like for instance like uh like humans are able to adapt the the sensory modalities to to to different things like maybe the we we can we can have a have a brao language to to learn to read by touch right but EXT standed machine learning algorithms they're they're very rigid and you assume that input number two is is this gears torque or something like that in RL so that led to like uh uh my my collaborator uh teammates Eugene ten and I work work on things like permutation in varant neural networks and getting neural networks to adapt to to different things so so I I think I I like to uh my heris is to look at things where existing methods don't do well at all and trying to push progress from a low base uh this is something we I tried to do in our work on a weight agnostic neural networks as well in this work we we try to do things

50:23 like uh find evolv architectures that can do tasks without training the weights at all when we're looking at things like inductive biases so so these networks would just work uh and we can even get them to work on mnist uh and like just just being able to show that these new things are possible uh that's not possible in the literature can go a long way like for instance the performance we got on amness was 92% it was much less than the state of the art but this paper got accepted at the news at the spotlight so I think this is like the we the state of the art in the anti state-ofthe-art that that can still get into one of these conferences that that's kind of like short-term strategies of how I I find research topics to to complement with the rest of the community yeah it's state-ofthe-art in terms of novelty of solutions not not that one metric that we should use to evaluate or Solutions yeah um Frontier yeah like

51:27 some efficient Frontier of of performance versus diversity yeah diversity novelty why do you think more web page focused journals like distill have not become popular yeah that that's a tough one like this still was such an inspiration for me like uh I think ultimately uh when we want to change the the behavior of a large community something like the scientific establishment like change takes time right you can't just like abandon PDF papers right that is ingrained in a culture right so so I I feel like uh this still has carved a really good Niche uh in the publication of how to explain uh ml Concepts uh in in clear visual ways uh but I I haven't I do think the still is quite popular like if you look at the the say traditional impact factors metric per article they do get decent number of citations I would argue even more than top conferences but they haven't achieved like the the conference level

52:36 impact because I think uh it it was difficult for uh for Chris and his team to to scale themselves to to have like this uh open up the the bottom up model of publication uh it was still very top down they have editors uh they have to have a certain level of quality and by Construction so I think uh the for bottomup approach maybe like this organic method of self-publishing uh both the archive paper and and the the online version might might be a good middle ground and that seems to be the one the version that's becoming more popular in so a question from Bert uh what do you think about research in complex systems and its relationship with ml/ Evolution well yeah this from the brid h I think he he is the expert in in the complex he to to ask this question to maybe yeah exactly right but I think this is like in my view this is this is GNA be the next uh the you know the next the next breakthrough in 10 to 20 years

53:45 might as well come from uh a complex systems ideas adapted in in neural networks like the way we're building all of these like a large language models and D is like it's really well structured and engineered like we we put together this gigantic model and train it on as much data as we can so we've shown that the current Paradigm is really good at creating a neuro data sponge like a neuro data sets that can do fancy things with the data encoded as as latent weights like latent space or weights but but when we look at like complex systems like like how we we we have several hierarchies like we have the level of of sell and every cell communicate within themselves but not necessarily to dist cells and then there we have another hierarchy of organs which are consists of cells and then we have another hierarchy of of body bodies like that that comp like we have a brain and a Hearts so this several levels of of

54:44 these uh these hierarchies of of how how systems can can interact in complex ways have not really been rigorously studied in deep learning and and even Beyond us as as individuals like like we're talking now as groups like as as a social environments and I think some some people in like like Natasha jocks uh is the expert in the social RL that is kind of really exciting work so there there are several levels of complexity and I think being able to properly understand uh complexity and its role in in how this decisions are being made uh will have profound impacts in in ml and evolution cool with that um we will end this retrospectives thank you so much David for coming thank you thank you great to talk to you folks bye thanks bye thank everyone for joining we all learn from one another research retrospectives are a series of conversations the dive into a researcher's technical body of work to

55:56 understand how papers build on top of each other and create a legacy we learn about paths not taken serendipitous events collaborations and setbacks that shaped their research our interviews are informal friendly and fun welcome persectives

Questions about this episode

Who speaks in "Research Retrospectives: An interview with David Ha, Co-founder and CEO of Sakana AI"?

Rosanne (Host), David Ha (Guest).

What is "Research Retrospectives: An interview with David Ha, Co-founder and CEO of Sakana AI" about?

David Ha, co-founder and CEO of Sakana AI, discusses his journey from a managing director at Goldman Sachs to a research scientist at Google, focusing on his interests in complex systems and creative machine learning applications. He also shares insights into his personal projects, including interactive web visualizations and experiments with AI, highlighting the inspiration behind his work and…

What is mentioned in this Google Research episode?

9 people, works and ideas are mentioned, including David Ha, Artificial Intelligence, Google LLC, Evolution Strategies, Evo Jax, Twitter. Each one on this page links to its own page with every episode that mentions it.

How long is "Research Retrospectives: An interview with David Ha, Co-founder and CEO of Sakana AI"?

The episode runs 56 min; the transcript is about 9,121 words in 52 timestamped sections.

More from Google Research

• All analyzed episodes
• People mentioned on Google Research
• Topics mentioned on Google Research