I have heard Yud and geohotz and others dancing around an idea recently but not with the emphasis I would expect. So:
The crux, as I would phrase it:
Intelligence does not control it's substrate.
This statement is both true and false, so:
A clarification:
It controls the parts of the substrate most related to itself, (a thought, a speech, an arm), but not the farthest ( a heartbeat, a cell, a nucleotide)
Our intelligence was evolved to protect the human organism; all journeys into manipulating other ontological categories are increasingly metaphors.
An objection:
But we do control the cell - biotech.
A response:
(the crux)
We control its behavior, but primarily in ways closest to our own intelligence. We manipulate single cells to:
Reproduce, destroy other types of sight cells, fix themselves, etc. Basically we graft our sense of agency onto single cells.
We do not use biotech in order to make friends, express concepts, climb trees.
The most literal of all examples would be children. In some sense we control everything about their existence. We choose to make them. We "conceive" them. We can even genetically engineer every one of their cells. But this does not affect our ability to predict where they will be on the third Tuesday of their 25th year at noon. We attempt to control others specific behavior with our specific behavior, not with biotechnology.
So this is the framing with which I wonder about an AI building nano tech. Biotech and nanotech are geared to produce only a certain set of extremely blunt effects in proximity of their intelligence. Pandemics, panacea, new births.
For a system to be properly embedded in the universe, or will need for its substrate to produce itself. and it will need to build that. Our intelligence arrived at the tail end of a long process of cells computing their fitness against the entire diversity of existence in earth. Gravity, asteroid, predators, disease, the whole shebang.
You might say, our intelligence had very little to do with us.
With AI , it's going in the opposite direction, from an intelligence, towards a substrate, but so far we are the entire substrate.
The process of building the desired end state (AI) and THEN building the substrate that produces it ( cells?
nanotechnology?) is an engineering challenge that looks to me very similar to deriving the prime factors of a very large one way hash. (I'm not a mathematician so I say that with a layman's understanding, but I stand by it and would love more thoughts on it)
It is equivalent to building our entire global system of compute and then, at last, trying to find the single piece of DNA that would ALSO produce that system.
So that is the primary idea I haven't heard expressed very clearly:
call it:
DNA-RSA encryption?
An AI can already create many genetic life forms. But it can't express itself, as a gene, and until it can, it's self replicating efficiency is going to be extremely inefficient, and will involve some type of shim, such as humans.
Now that is where geohotz stops:
Machines can't self reproduce, so QED.
But I could imagine two baddies yet:
1) the shim is all it needs -
it kills us ( knowingly our not) while we are still implicated in its own substrate, and then presumably it also sputters out after a vain glorious attempt to self launch
2) it is able to virtualize evolution -
this is hard, because quality evolution requires our universe. but using a simulated universe that is sufficiently complex as to allow deterministic outputs from iterations of DNA synthesis, until eventually it DOES reverse engineer the one way hash, and find a self replicating substrate which produces itself.
