Machine Learning in Drug Discovery Symposium: Marinka Zitnik

What is discussed, and when

1. 0:00 – 3:00: Swissro
2. 3:00 – 6:00: Proion
3. 6:00 – 9:00: Proion
4. 9:00 – 12:00: Proion, Imperial College London, King's College London
5. 12:00 – 15:00: atomica, MIT

Notable quotes from "Machine Learning in Drug Discovery Symposium: Marinka Zitnik"

• The human protein that remains inaccessible to models that would just read PDFs of our papers. So we developed an open source and model that we refer to Proion and what Proion…

  about Proion · at 3:16

• Localization essay for that protein in the human protein atlas. Here's another example of how our collaborators from Imperial College London and King's College London use…

  about MIT · at 10:17

• Approach is that because it's atom scale approach that operates at the level of auto atomistic graphs, we can train a single unified model across these different molecular…

  about atomica · at 13:23

Who and what is mentioned in this episode?

Companies & orgs

• MIT — Company · Affiliation · 1 mention · first at 14:28
  A leading institution where the models Proion and atomica were developed, showcasing its role in advanced AI and biological research.
• Imperial College London — Company · Collaborator · 1 mention · first at 10:24
  Collaborated in using Proion to study and characterize targets of multiple sclerosis, demonstrating the model's practical application in medical research.
  Validated through subscellular localization essay for that protein in the human protein atlas. Here's another example of how our collaborators from Imperial College London and…
• King's College London — Company · Collaborator · 1 mention · first at 10:24
  Collaborated in using Proion to study and characterize targets of multiple sclerosis, contributing to the validation of the model's predictions.
  Validated through subscellular localization essay for that protein in the human protein atlas. Here's another example of how our collaborators from Imperial College London and…
• Swissro — Company · Referenced database · 1 mention · first at 2:49
  A canonical database used to explore functional notations of proteins, mentioned to highlight the gaps in protein functional annotations.
  That we all use to explore functional notations of proteins we are studying, they're just empty pages or they're very poorly functionally annotated.

Products & tech

• Proion — Product · Business focus · 8 mentions · first at 3:25
  An open-source multi-model foundation model developed to predict, characterize, and generate phenotypic descriptions of proteins, addressing gaps in protein functional annotations.
• atomica — Product · Business focus · 2 mentions · first at 12:24
  An all-atom based model of molecular interactions, focusing on modeling interactions between molecules rather than individual molecules, developed to illuminate the dark proteome.

Full transcript with timestamps

0:00 is okay. The analogy for we need a lab means that we need tools and so tools for AI are different types of foundation models and and other types of tools or data sets and knowledge bases. And second, for in order for uh scientists to use tools, we need then large language models that can orchestrate and work with those tools. So the purpose of today's talk is for me to show you so how we are making progress towards realizing this vision by by a developing tools that serve as a way as as a lab that scientists can then use and second developing scientists that that are AI based scientists that now can use those tools flexibly. So first there are a number of models under the AI paradigm that in the group we are developing and those span four biological skills. So we think about molecules and molecular models that can help with reasoning and understanding specifically what are the functions and the role that different

0:55 molecules have in specific biological contexts. Second, we know that molecules don't operate independently of each other but rather instead the form molecular interactions and that those molecular interactions then are embedded into broader cellular environments which are defined by cell states and cell types. And finally the fourth scale is that of reasoning at the level of tissues or entire human patients and human populations to get a sense of how well effective and safe those candidate new treatments are. So I want to give with to share with you a few vignettes of the models that we're developing on across these four scales. So first is that of molecular scale and the motivation for this for our research in this area is a broad observation that as a field we now have quite a number of high highly powerful models that can model and predict and characterize protein structure from primary amino

1:48 acid sequence and for important classes of protein families they can achieve atom level accuracy or more match that of experimental efforts and that has been tremendous tremendously important for the field at large and it it it paves the way towards realizing this grand challenge of having models that can predict and characterize the structure for every protein. There is however another yet grand challenge that we think it's very important and remains largely open which is that of modeling understanding functions or phenotypes that those proteins are involved in and play a role in which is crucially important for characterizing identifying and assessing targets among others. So we embarked on this problem and challenge of developing foundation models that can help provide functional insights for a very large broad regions of human protein. Why would you want to do that? We would want to do that

2:42 because still 20% of the human proteins lack functional annotations. Very practically that means that if you go to Swissro, Uniro, other canonical databases that that you typically use that we all use to explore functional notations of proteins we are studying, they're just empty pages or they're very poorly poorly functionally annotated. 95% of all publications in life sciences focus on 5,000 most wellstudied proteins. That means that simply by reading literature alone, we are just not able to cover these vast regions of the human protein that remains inaccessible to models that would just read PDFs of our papers. So we developed an open source and model that we refer to Proion and what Proion is, it's a multimodel foundation model that can predict, characterize and generate phenotypic descriptions. Proion operates at five different levels or five different knowledge domains that are represented here by different rows and

3:41 relate to modeling and understanding predicting biological functions, processes and cellular components. Second uh in reasoning about the roles of proteins in disease mechanisms. Third about the roles of proteins in target in drug mechanisms and fourth and fifth understanding of functional impacts of molecular interactions and functional impacts of substructural domains in proteins. When the model is trained, human scientists can interact with the model in plain conversational mode. And there are three types of interactions or three or or mo modes of collaboration with human scientists that pro supports.

4:18 First, first what is shown in panel A is an example where human scientist describes in their input problem the biological process of interest. They describe the task that they want the model to solve and then the model will look at that biological process and then be able to scan, retrieve, score and rank proteins that likely would satisfy the task described in uh in the input prompt. Second model can engage with the user in an interactive back and forth yes and no in session to have a conversation or as or do question answering about a specific protein.

4:50 Importantly the user input prompt here can be natively multimodel meaning that user can interle textual description of biological function with structure and or sequence for the protein they would like the model to operate with. And third mode is a generative mode. Here the model is asked to generate new outputs and the outputs that we are generating here are not structural outputs but text phenotypic descriptions. We call them sometimes protein caption cards that describe in a conditional manner what a protein that is input that is in the input prompt uh actually does. So this is how you can interact with the model when the model is trained. So how do we train the model? To train this model there are two things we need. We need a data set and we need the model architecture that we will then train using the data set. So first the data set the to to train proion model we created uh and curated

5:46 proian training data set that is openly available and it's a data set of 33 million data points where each data point is a tpple and you can think of a tpple as having two elements where the first element is protein ID and the second element is some paragraph of text a passage of text describing a particular function in that spec for that specific protein. In addition to this multimodel protein text data set, we also have information about primary protein amino acid sequences, information about predicted and experimentally derived protein structures that is that are used to train uh language protein language and protein structure encoders and finally a large number of natural scientific natural language in order to equip the model with ability to understand um and converse with scientists in plain language. Given this data set, we um we use it to train uh Proion model and architecturally speaking, what this

6:45 slide shows is at the high level key modules or components of Proion which we can read from left to right and we can see here on the left are sample prompts which can be either retrieval, predictive prompts or generative prompts and those prompts have different have can can combine different types of modalities. They are decomposed by the model into text structure sequent based components and different these three different modalities are processed by modality specific encoders that have additional projectors and those projectors ensure that the latent embedding space that is unified across these three different modalities captures the biology of and the structure of the training of the training data set. Then the model can either decode and rank score proteins from the leunate latent embedding space if it operates in the retrieval mode or if it operates in generative mode. The model can use auto reggressive sampling

7:40 strategies to generate text conditioned on the input prompt that the user specified. So we train this model architecture on the data set. We use a composite objective function that ensure that the model can follow human instruction that the model can retrieve correct proteins and rank them very highly and also that the model can generate text that is that is coherent to read and all together trained a model that has 11 billion model parameters that we made available to others as well as u to analyze ourselves as we usually do when we train large scale foundation models before using the model for anything new. What we typically do is to use the model and benchmark it across established benchmarks to see how well it performs. There are probably 30 to four there 30 to 40 pages of these type of plots that we have in supplementary information for our paper. So I'm just showing here one snapshot of those

8:32 benchmarking um experiments where we have tested proion in a few shot setups for different functional categories. These are different bar plots and for each bar plot we measure performance in in this slide as um fmax uh score higher value indicates better retrieval and we compare proion shown here in purple to a variety of existing models including models that are uni models state-of-the-art models like state-of-the-art sequencebased protein language models or state-of-the-art structure only based models that are automistic models and then also we compare them to emerging uh models that are multimodel and combine text with uh with sequencing structure such as brought SD or bot translator and also include more standard bionformatics pipelines that use blast and k nearest neighbor uh on top of blast sequence similarity in order to predict protein functions and other functional categories and we are seeing broadly

9:28 across many of those categories proion performing really well which has encouraged us to also go and use the model in some interesting novel ways. So here are a few vignettes of that. We've been working with collaborators who are studying Parkinson's disease and became very interested in protein AK81 recently implicated in the onset and progression of Parkinson's. But if you look at AK81, it's a stop. So in canonical functional databases, it's poorly functionally annotated. So we just used Proon and asked Proon whether it can generate these type of conditional pro phenotypic descriptions and it did so. And here are some examples of what it generated for AK81. And so one of the phenotype that it generated was about the involvement of AK81 information of acting cytokeleton which we then independently validated through subscellular localization essay for that protein in the human protein atlas. Here's another

10:22 example of how our collaborators from Imperial College London and King's College London use proion in order to study and characterize targets of multiple sclerosis. And so collaborators are very interested in very precise deep characterization understanding of those targets that they could map to specific pathology in MS. So what they came up with the their neuropathologist write wrote down these five distinct types of descriptions of MS pathology that is clinically relevant for for for MS. And so initial parts of those prompts are shown on the far left of the slide. And each of those prompts was sent to Proion and Proion was asked to retrieve and score proteins that would be most characteristic could could explain collectively that specific type of pathology. And so what we have done the next or our collaborators have actually be uh be able to stain uh 280 samples of postmortem uh brain and perform bulk

11:21 arna seek analysis on those and also classify them to each of those five distinct pathologies and we looked at what is the overlap between those differentially expressed proteins in a pathology specific manner that were identified through RNA analysis and those proteins that were retrieved highly by proion and were assigned high percentile ranks. And what we can see here is that across different pathology types there is a there there is a substantial enrichment where those proteins retrieve protein were indeed characteristic of uh of um in in experiments. And so this is an example of kind of information leakage proofed type of analysis for the model where it was used in a forward direction to test novel hypothesis. It made proon can reason about one molecule at a time and specifically it can it can it can take as input one protein potentially multimodel data about that protein and it can reason and provide some

12:19 functional insight about that protein and now we know that those proteins don't act independently of each other. So we recently been developing a model called atomica that is focusing specifically on this question of modeling not one molecule at a time but modeling interactions between those molecules. And so what atomica is it's an all atom based model of molecular interactions which is specifically focused on taking interaction complexes and representing interaction complex with a graph. So we show which is shown in the center of the slide that in that graph we have nodes that represents atoms or nodes that represents chemical motifs residues higher level structures and edges between those nodes indicate bond lengths at the the hedral angles and other important properties. We then create and build these type of graphs for the entire data set of two million com uh molecular interactions that span

13:12 different molecular modalities and cover protein protein protein small molecule protein uh um RNA protein DNA lipid ion interactions. The beauty of this approach is that because it's atom scale approach that operates at the level of auto atomistic graphs, we can train a single unified model across these different molecular interactions. And we train a model through a pre-training objective that is a one-step discrete diffusion objective. You can effectively think of them as an objective that masks specific block in the input graph and trains the model in order to predict what is the identity and the structure of the block that was just masked. So it's type of language modeling but not when we have a sequence of words but it really operate in this three-dimensional space.

13:57 What can we do with this model? Again we did lots of benchmarking. I will skip all those analysis plus some interesting um compositional structures of the embedding space that we can show. the uh the the PG student here Ada who has developed this model. She was very interested in this idea of dark proteom and whether we can illuminate dark proteom identify and predict uh functions related to molecular interactions for some of proteins that are poorly characterized here. So she used atomica to nominate five potential hem binders. Um, and so then our collaborators from the chemistry department over at MIT have synthesized those proteins and then ran functional essays on them to confirm that those those predicted hem binders were indeed hembers. What I find most exciting here however is that four out of five of those predicted hem binders could not be identified using standard approach.

14:49 standard approach being taking a molecule and scan it for the presence or absence of certain structural motives that are indicative of him binding. So this is an example that illustrates of how these models when used thoughtfully and carefully they can also provide insights that go beyond what traditional models can allow us to do. So now that I'm two minutes over time so I'll just wrap up with three slides perhaps if I can squeeze them in. So I talked about models. So models you can think of them are analoges for AI of what tools are in wet lab a scientific experimental lab.

15:26 Well, but tools are not enough. There should be people who operate those tools and who generate interesting research plan and execute them. And so that practically means that there's another component that is missing and specifically we need now orchestrators that will use these different types of foundation models across different types of data and different types of modalities. So we have built exactly that. So if you go to a scientist.tools you will see tool universe. Tool universe is a universe of 600 plus scientific tools that cover a broad range of ML models um software packages bionformatics workflows tools for automations knowledge sources rack systems visualization etc etc. And what tool universe allows you to do is to connect your favorite large language model which can be closed or open-source model. It can be cloth, it can be gemini, QN, GPT codeex or something else with that tool universe. So that what

16:22 you then get is an interface like that if you connect it with cloth where if you do go down you can see tool universe 600 plus tools and you can you have your scientists that will now be able to use the tools that are in tool universe and this is made possible by by kind of novel advances that um that relate to um AI human uh AI AI interaction protocols where models can interact with each other and interact with other types of tools that human scientists So with that I'll wrap up here uh acknowledge fantastic group of PhD students and postocs that I uh and research associates in my group uh and happy to take any questions.

Questions about this episode

What is discussed in "Machine Learning in Drug Discovery Symposium: Marinka Zitnik"?

In order: 0:00 Swissro; 3:00 Proion; 6:00 Proion; 9:00 Proion, Imperial College London, King's College London; 12:00 atomica, MIT.

What is mentioned in this Broad Institute episode?

6 people, works and ideas are mentioned, including Proion, atomica, MIT, Imperial College London, King's College London, Swissro. Each one on this page links to its own page with every episode that mentions it.

How long is "Machine Learning in Drug Discovery Symposium: Marinka Zitnik"?

The episode runs 17 min; the transcript is about 2,870 words in 20 timestamped sections.

More from Broad Institute

• All analyzed episodes
• People mentioned on Broad Institute
• Topics mentioned on Broad Institute