Intelligence Explosion Microeconomics
Machine Intelligence Research Institute
I. J. Good’s thesis of the “intelligence explosion” states that a suﬃciently advanced machine intelligence could build a smarter version of itself , which could in turn build
an even smarter version, and that this process could continue to the point of vastly
exceeding human intelligence. As Sandberg (2010) correctly notes, there have been
several attempts to lay down return on investment formulas intended to represent sharp
speedups in economic or technological growth, but very little attempt has been made to
deal formally with Good’s intelligence explosion thesis as such.
I identify the key issue as returns on cognitive reinvestment—the ability to invest
more computing power, faster computers, or improved cognitive algorithms to yield
cognitive labor which produces larger brains, f aster brains, or better mind designs.
There are many phenomena in the world which have been argued to be evidentially
relevant to this question, from the observed course of hominid evolution, to Moore’s
Law, to the competence over time of machine chess-playing systems, and many more. I
go into some depth on some debates which then arise on how to interpret such
evidence. I propose that the next step in analyzing positions on the intelligence
explosion would be to for-malize return on investment curves, so that each stance can
formally state which possible microfoundations they hold to be f alsif ied by historical
observations. More generally, I pose multiple open questions of “returns on cognitive
reinvestment” or “intelligence explosion microeconomics.” Although such questions
have received little attention thus far, they seem highly relevant to policy choices
aﬀecting outcomes for Earth-originating intelligent life.
Communal Assembly Paper