Progress & Knowledge

In this episode, I discuss:

1. Progress. What makes progress possible? What makes it go faster? What makes it go further?
2. Four elements of knowledge-creation. Variation, selection, attention, & knowledge.
3. The nature of knowledge. What is its role? How does it work?

Follow me on Twitter! @dela3499

Listen on Apple Podcasts, Spotify, Youtube, and more: https://carlos.buzzsprout.com/share

Chapters

• 0:00 What makes progress possible? What makes it go faster? What makes it go further?
• 3:30 The four elements of knowledge-creation: variation, selection, attention, & knowledge
• 4:47 Key questions: basic purpose, universality, quality
• 8:49 This last week: Knowledge
• 20:34 Questions from Twitter

Transcript

0:17

Howdy, howdy. Carlos here. So I work on artificial general intelligence, and my main research question is, what makes human minds so powerful? You know, it seems like every other species and algorithm is maybe doing interesting things, but humans are doing even more interesting things, going to space and whatever else. I think you can actually ask questions about humans and what makes them so powerful by zooming out and thinking about something more more abstract, just the nature of progress itself. And I think there are three really big questions there that I think you can apply to daily life, to business, to other things. They're really just about progress by itself. And three questions are, firstly, what makes progress possible? Also, what makes it go faster? And lastly, what makes it go further? So if you think about what makes progress possible, you know, one thing that would make it impossible is if reality just didn't allow much. If we all lived in some kind of game world where only one or two things could happen, then you would do those one or two things and nothing else could ever happen. So that's a case where just reality wouldn't allow progress. Another case would be maybe some system, like what if people couldn't exist in some game world so nothing could make progress there, let's say, or biological evolution, what if it couldn't happen there? So there may be these requirements that are really fundamental just for having any progress at all. Another question, like I said, is what makes it go faster? That's a question you can ask at the level of businesses, institutions, governments, individual life. You can ask about evolution, you know, what makes evolution go faster or slower at different times. So that's a pretty fundamental and interesting question. And then lastly, what makes it go further? You can imagine a case where, you know, if you blow yourself up, you stop making progress. So things that prevent that are useful. You could also imagine, like I said earlier, you know, if you have a game world and it only allows a few things to happen, then not much progress can occur. But if it was a bigger game world, then progress could go further. And if it's like our world, then maybe progress can go on infinitely. Incidentally, I remember being in a Twitter conversation with Anders Sandberg, and he was saying, maybe in the very far future, we'll just find that the universe keeps on existing, we can keep on trying new things. But maybe just time goes on so long, so, so long that we just run out of new things to do that are not new things, but better things, let's say. So it just sort of put it to me like, you know, if we go to Graham's number, so some large number of years, you know, will it, there's still even then be new things we can do. And, you know, it's not obvious. The answer is yes, I guess it's a conjecture that it will be the case. Obviously it's hard to imagine what those possibilities would be. And that makes it hard to kind of say what the right answer is there. But in any case, These are all interesting questions. So like I said, my main research question is what makes human minds so powerful? And I think you can zoom out and ask these three very more abstract questions about progress. What makes progress possible? What makes it go faster? And what makes it go further? So with that said, I want to turn to some stuff that's more relevant to what I've been working on recently. And really, all of my research tends to focus on a handful of, let's say, four things, thereabouts. The first of which is the creation of new things. I like the word variation for that. You know, variation and selection, as Darwin said. So the first is variation, creating new things. Second is selection, distinguishing the good from the bad. And then lastly, or not lastly, but thirdly, attention, which I think of as bringing the right resources to bear on the situation. So, you know, you have many things in your head, most of them are irrelevant. You want to bring the relevant ones to bear on your situation. And that makes you much more efficient as you try to solve problems. So that's variation, selection, and attention. And lastly is knowledge. You know, all this variation selection is occurring or is happening to something and that something is knowledge. This is information. So knowledge is really going to be the focus or has been the focus of my last week and is what I want to talk about here. But I do want to set the stage there with those four things, variation, selection, attention, and knowledge. Those are like the the big four of what I think about. And for each of those four, I think that a few basic questions arise. The first one is, what is the basic job of that thing? You know, for variation, the basic job is to create new things. And why is that important? Because if you couldn't create new things, you couldn't create better things. So that's an example where I'm just trying to take those four things I mentioned, variation, selection, attention, knowledge, and ask this very basic question about it. What is the fundamental job of this thing? Setting aside its role in humans, setting aside a lot of the details about how those things work, just what is the basic job? And usually the answer is pretty simple. So that's the first of these three big questions I ask. Yet the two are much more specific to this larger research question of mine about AGI. The second question I want to get to here is about quality. For each of these four things, variation, selection, attention, and knowledge, there are better and worse forms of them. So if you think about creating new ideas, a good method of creating new ideas would create better ideas, would create good things, a large fraction of the time. A really bad way of generating ideas would generate newer and better things only a small fraction of the time. So, you know, a bad way of generating new ideas would give you a good idea once every 10,000 years. And a good way would do it once a day, you know, let's say. And the point of asking this is to say that it seems like humans have very good versions of all these things. You know, we must be distinguished from other animals in these ways. So, the question is to ask, if these four things, variation, selection, attention, and knowledge, are the important things, then those should be the areas where humans excel in some way. So the question is, what are the aspects of those things that matter for quality? What are the better versions and what are the worst versions? So yeah. And so that's question number two. Question number three that again, I just applied to all these important things I think about is this question about universality or I call it completeness, something like that, where the idea is that, I sort of covered this in my last episode, that one of the really big ideas from David Deutsch in his book, The Beginning of Infinity, is that what sets humans apart isn't just that we can do more things than other animals, but that we have some kind of unlimited ability, whereas everything else has limits on its ability. So even if, you know, let's say a dolphin can do many things more than some other animal, there's still an infinite difference between able to do n things and ultimately able to do anything, an infinite number of things. So that's sort of the question here is like, for, let's say, creation of new ideas, you can ask, what sorts of universality would matter here? What sorts of completeness. And the answer, I think, one answer is, for instance, the ability to create any idea. That's something which, if a system can only create some ideas, obviously, there'll be other ideas it can't create. And those may be very useful, and the system will never be able to produce them. Whereas if you're a human, and you can think of anything, then no matter how complex the machine, no matter how complex the theory, you'll be able to eventually produce that thing. And that may be essential to your survival let's say so these three questions sort of are the basic armory the basic tools that i want to apply to all those things that matter variation selection attention knowledge and for each of them i ask what's its basic job what are the better and worst versions of that thing because humans presumably have the better version and what sorts of universality are relevant there you know if it's about creating new ideas then what set of things matters there? And in that case, it's sort of the set of all possible ideas and the ability to create any of them. So with that, I want to talk about what I've been up to this last week. That sort of sets the stage. That's the four things I care about. That's the three big questions I ask about each of them. And this week, I've been kind of focusing on knowledge. So, I think the key idea to have about knowledge is that, you know, the first thing is it's sort of like the subject of variation and selection. And the thing that I tend to think about most, like for a concrete example, is just genes. So, if you just read The Selfish Gene from Richard Dawkins, or I'm also rereading right now The Blind Watchmaker from Richard Dawkins, I think genes are just a really great concrete way to think about the role of information and replicators and all these sorts of things in knowledge creation. So you can ask these three questions that I mentioned about knowledge. What is its basic job? What are the better and worse versions of knowledge? And what kinds of universality matter with regard to knowledge? So if we think about what are the basic jobs of knowledge, I'd say there are two, really. The first is to cause things to happen, sort of to control physical reality, to make things happen. The second job of knowledge is to capture patterns in reality. So one kind of knowledge is about what to do, another kind of knowledge is about what is true. And so those are the two jobs of knowledge. But you can ask, how are those things achieved? And I think it pays to step back and ask kind of what's going on when we think about knowledge. And I think genes offer a really good concrete example of this. And I think really there are like three pieces to the puzzle when thinking about knowledge. The first is information. In the case of genes, you would look at the DNA and see A, T, C, and G. And that's what you have to work with. That's the basic information. Information. Secondly, though, you have to recognize that though the information is there, information by itself is always meaningless. It's just a bunch of chemicals, let's say, in that case. So it's meaningless until it's decoded by something. And I shouldn't say it's meaningless until it's decoded, but let's say there was no decoding mechanism at all, and you just had these A's, T's, C's, and G's. It wouldn't necessarily be meaningful. It might be like an encrypted piece of information, let's say, which is meant to be meaningless unless you can properly decrypt it. So what actually matters isn't just that you have information, it's that it sort of specifies something else. And in the case of genes, that's proteins. So that's sort of the second piece of this puzzle, is on the one hand, you have the information. On the other hand, you have, in this case with biology, you have proteins, this set of things that is being specified by the genes. And the third thing you need is a connection between these two sorts of domains, the information domain and the protein domain, in this case. So that role is taken up by certain parts of the DNA, that one part of it will grab the DNA, another part will grab some amino acids and connect them all together in the appropriate way and this sort of thing. But if you didn't have that in the DNA, there would be no method of decoding all of its information to produce specific proteins. It would just be garbage, random noise. So the basic point here is that you have to zoom out, I think, to these three different things. This information, I almost draw like two circles in a line connecting them. On the first circle is the information domain the second circle is some other domain like i'd say proteins or whatever else maybe programs and then some kind of thing connecting the two. Something where it says okay if this information is over here set up in a certain way then that will make this other specific thing let's say a protein happen and if that information was was different, it would lead to a different protein. So those three things matter, the information, this other domain, proteins, whatever, and a mapping or a connection or correspondence, something between them. Those two areas have to be connected somehow. So that's the basic picture of what's happening with knowledge. In the case of genes, again, you set a certain sequence up that then goes through some process of mapping and there are various details to be be known there. And then it leads to the creation of a specific protein depending on a specific sequence. And then if you zoom out, you could run that whole process in reverse in a different way, where you let's say looked at, instead of looking at genes, now let's look at brains. And let's ask about how things out in the world can cause information to change in your head. So senses, in a way, are like the reverse process of what I just mentioned. Instead of having information which then goes some process of affecting proteins. In this case, you have photons, this physical thing, eventually affecting the information in your brain. So this picture of these three things, the information on the one hand, something physical like proteins on the other or whatever, and then some connection between them, you can imagine that sort of whole process running left to right or right to left in different ways. The other thing to be said about knowledge is that if you think about what can knowledge do, I said it can do two things. The first is make certain things happen. And then the other is capture patterns in reality. So some of the things that information does aren't really about just causing a specific thing to happen. Yes, you have knowledge in your head about a particular recipe and how to cook the chicken a certain way or whatever. But you also have ideas simply about like what is true that aren't really explicitly about you know doing anything so if you think that there is a coffee table uh in your in your house and that there is coffee in the mug on that table that's just a fact and then it may combine later on with other facts you know like a desire for coffee will then let you you know allow you to pick that coffee up and drink it but at first you just have this fact that the coffee is there so that's not really a recipe of any kind for what to do, but it can participate in those kinds of things. And deeper theories about physics are similar. They're just knowledge about what's out there. So I like to think that knowledge sort of has these, again, two jobs. One is making certain things happen. The other is capturing patterns in reality. One is about what is true. One is about what to do. One is about explanation. The other is about transformation. These two roles of knowledge. So like I said earlier, if we just zoom way out here, I kind of said that three kinds of questions were what I always kind of circle back to. And the first is, what is the basic job of this important thing, in this case, knowledge? The other two questions were, what does quality mean for this thing? What is a better or worse version of this thing? And then lastly, what is sort of a universal version of this thing? What kinds of universality matter here? So if I turn to the question of universality, One kind of thing you can ask is, can this information specify any kind of thing, any kind of protein, let's say, or any kind of idea, or any kind of physical process or transformation? That's the kind of thing that starts to bring up ideas about universal computation, let's say. So if you say, okay, I have some information here, but what does it mean? What can I do? do, then it may be possible that if that information is found on a Turing tape, that information could encode any kind of program whatsoever. If that information was found in some other kind of computational system which was non-universal, then it can only specify certain kinds of programs, let's say. Or in the case of biology, it's worth noting that amino acids are what make up a protein and part of their job is to sort of fold and kind of just do a different structural role. Every amino acid sort of bends a certain way or attracts or repels other molecules a certain way. And if you have a full toolkit of amino acids, then you can make basically any kind of protein shape. And if you don't have a full set of proteins or rather amino amino acids. If you don't have a full toolkit, then there are certain kinds of proteins, certain kinds of shapes that you can't make. So our DNA apparently seems to have a full set. It can do all the different curves and repulsions and attractions that basically matter. And evidently, like there are other, incidentally, there are other amino acids that exist that aren't included in our DNA really, because they do a job that's already done. they don't add any additional power to the system. So you can just think of like computers too, like you know, you can add new operations to them, but they're already universal, there's no making them more universal. So when it comes to the idea of knowledge and universality, one of the really big questions is, what does it take for a system to be able to encode any idea whatsoever? I think that basically breaks down to a point of thinking of ideas as programs. And in In that case, achieving universality is all about achieving Turing universality, Turing completeness. But anyway, this is, like I said, one of three questions that I ask. And the details are interesting there, where I haven't really gotten too deep into this yet. But the big question will be, how is Turing completeness achieved in brains? There are interesting papers on that that are worth getting into. But I think it's worth noting just at this stage that that question of achieving universality is One of the basic two or three questions I think matters for AGI So zooming out a bit further. I. You can ask about quality. So that's the third question that I brought up earlier. And I guess I won't say too much more about it other than, let's say maybe you can have a system for encoding knowledge, which is more or less efficient. I think a really good example of that is like number systems. If you had to use Roman numerals or just tally marks to represent numbers, it gets inefficient pretty fast because let's say representing the number 1 billion takes 1 billion marks, whereas representing it with numbers takes only like nine marks or like nine zeros and a one. So the difference between like 10 digits versus 1 billion is pretty noticeable. So both of these systems can represent all of the numbers, but some of them do it much, much more efficiently. So I think that's one example of how some kinds of knowledge systems, some kinds of ways of interacting with information can be much, much better. And there's other more engineering-type concerns like just overall speed of the system and how it works. But this notion of quality I think is worth thinking about in all the cases of, knowledge and variation selection and attention. Those are the big four that I mentioned earlier. So for each of those, I'll ask all these questions. So with that, I will turn to some questions from Twitter. Firstly, one from ConvexBets. He asks, what do you know about the current state of DeepMind and OpenAI's AGI research? So I'm no expert on them. I'm in general not an expert on LLMs or any of those things. But I do think about what are the basic principles that an AGI needs to have, needs to meet? What are the fundamental requirements? That's my main focus. So like I said earlier, my main focus is always on these big four things, variation, selection, attention, knowledge. So my question whenever some new company gets to work is basically about those four things, let's say. One thing I will say though is that, just as an aside, I think Marcus Hutter, as a researcher that works at DeepMind, has a very long history in the field of AI and AGI research, and he's one of the few people that takes universality seriously. Thinking about the limiting case? What about a system that can do anything? And I would say, however, that he goes down a Bayesian path and thinking about what is called AIXI. So I think the Popperian path of thinking about a more evolutionary perspective is the one that actually matters. So yeah, I'm sure that lots of these companies are doing very interesting engineering work. And in this case, someone's doing some interesting theoretical work and taking universality seriously. But I would say that that Bayesian approach to universality is, I guess, if you could go with Popper, just fundamentally flawed and just impossible to make work. But anyway, setting that aside, we have a second question from SM Rad. And he asks, do you think there's an equivalent of attention in biological evolution, even if it's a less a less powerful one. So in the last episode, I talked about attention, and that's all about bringing the relevant ideas to the situation. So if you have a billion ideas in your head, or if you have 20,000 genes. Making alterations to any given one of those ideas at random is very unlikely to be a useful thing to do. However, if you can bring the right ideas to bear and focus your attention, focus your resources on a few of those ideas, the ones that are relevant, then you'll have a much better shot at making progress. So there's this question, it seems to happen in human minds, but does it happen in biological evolution? That's the question. and i think that really in a sense you could say yes in that not all possible mutations are equally likely if you look just at the overall dna strand from one end to the other the rate of mutation is different across that strand for all kinds of reasons so that's a case where maybe certain areas are less prone to mutation others more so and you could call that a form of attention although it's not particularly you know related to the evolutionary situation of a of an organism so in that case maybe maybe really important genes are somewhat protected and that's useful although even that is somewhat suspect but i'm not sure that actually happens that much but uh but as far as rates of mutation that is kind of a variable across the genome, let's see here. Uh so i think that's about the main question that i all that i have here yeah so with that i shall end it for today um and give you a quick overview here like i said you know my main question is always what makes human minds so powerful and i think you can zoom out ask about progress what makes progress possible what makes it go faster what makes it go further and if you look at these big four things that I mentioned, variation, selection, attention, and knowledge, you can ask basic questions about all of them. What's its basic job? What are the better and worse versions of it? And what are the universal sort of universal versions of it? The thing that sort of you take it to the limit. So those are the really basic questions. And in this case, I think the most important thing, or rather what I covered this last week was all about knowledge, applying those questions to knowledge and seeing that knowledge is always about taking some information and mapping it or connecting it somehow to some other domain, let's say proteins or ideas, or maybe running that process in reverse. But seeing that information is something that's important, some other domain of programs or proteins or whatever else is important. And then lastly, but also importantly, the connection between those things is important. Going from information to proteins, let's say, or going from something in the world to information itself. Yeah, so there's a quick overview and that's what I've been up to this week. So adios for now.