Please assign probabilities to the following (for the next 3 decades):
probability an AI smarter than any human on any intellectual task a human can do might come to exist (superintelligence);
given (1), probability it decides to kill all humans to achieve its goals (misaligned);
given (2), probability it is successful at killing all humans;
bonus: given 1 and 2, probability that we don’t even notice it wants to kill us, e.g. because we don’t know how to understand what it’s thinking.
Since the AI is smarter than me, I only need to propose one plausible method by which it could exterminate all humans. It can come up with a method at least as good as me, most likely something much better though. The typical answer here would be that it bio-engineers a lethal virus which is initially harmless (to avoid detection), but responds to some trigger like the introduction of a certain chemical or maybe a strong radio signal. If it’s very smart, and has a very good understanding of bioengineering, it should be able to produce a virus like this by paying a laboratory to e.g. perform some CRISPR operations on some existing bacteria strain (or even just mix some chemicals together if Sagan turns out to be right about bioengineering) and mail a sample somewhere. It can wait until everyone is infected before triggering the strain.
Well, the probability you have for the AI apocalypse should ultimately be the product of those three numbers. I’m curious which of those is the one you think is so unlikely.
1: 100% but I don’t have a timeline. It’s not going as fast as the cultural hype presents it. We don’t even really understand human thinking yet, let alone how to make a computer do it. But I’m sure we’ll get there eventually.
2: Also 100%. AI doesn’t need to decide on its own to kill all humans, it could be assigned that goal by some maniac. The barrier to possessing sophisticated AI software is not nearly as high as the barrier to getting destructive nuclear weapons, biohazards, etc. Sooner or later I’m sure somebody who doesn’t think humanity should exist will try to unleash a malevolent AI.
3: At or near zero, and I only include “or near” because mistakes happen. Automated systems that could potentially destroy the human race should always include physical links to people - for example, the way actually launching a nuclear missile requires physical actions by human beings. But of course there’s always the incompetence factor - which could annihilate the human race without the help of AI.
You need not only propose a “plausible” scenario, you also need to present a reason to believe it will happen. It’s plausible that a rogue faction could infiltrate the military, gain access to launch codes and deliberately start WWIII. It’s plausible that a bio lab could create an organism that overcomes the human immune system and resists all medications. A nonzero chance of any of those happening isn’t proof that they’re inevitable, with or without AI.
Well I’m not claiming that an AI-apocalypse is inevitable, just that it’s possible enough we should start worrying about it now. As for the reason to believe it would happen – isn’t that covered by (2)? If you believe that (2) will occur with near-100% certainty, then that would be the impetus.
Please assign probabilities to the following (for the next 3 decades):
bonus: given 1 and 2, probability that we don’t even notice it wants to kill us, e.g. because we don’t know how to understand what it’s thinking.
Since the AI is smarter than me, I only need to propose one plausible method by which it could exterminate all humans. It can come up with a method at least as good as me, most likely something much better though. The typical answer here would be that it bio-engineers a lethal virus which is initially harmless (to avoid detection), but responds to some trigger like the introduction of a certain chemical or maybe a strong radio signal. If it’s very smart, and has a very good understanding of bioengineering, it should be able to produce a virus like this by paying a laboratory to e.g. perform some CRISPR operations on some existing bacteria strain (or even just mix some chemicals together if Sagan turns out to be right about bioengineering) and mail a sample somewhere. It can wait until everyone is infected before triggering the strain.
Or how about you don’t assign me tasks and I don’t do them? Cuz I don’t remember signing up for a class.
Well, the probability you have for the AI apocalypse should ultimately be the product of those three numbers. I’m curious which of those is the one you think is so unlikely.
Okay here are my estimates:
1: 100% but I don’t have a timeline. It’s not going as fast as the cultural hype presents it. We don’t even really understand human thinking yet, let alone how to make a computer do it. But I’m sure we’ll get there eventually.
2: Also 100%. AI doesn’t need to decide on its own to kill all humans, it could be assigned that goal by some maniac. The barrier to possessing sophisticated AI software is not nearly as high as the barrier to getting destructive nuclear weapons, biohazards, etc. Sooner or later I’m sure somebody who doesn’t think humanity should exist will try to unleash a malevolent AI.
3: At or near zero, and I only include “or near” because mistakes happen. Automated systems that could potentially destroy the human race should always include physical links to people - for example, the way actually launching a nuclear missile requires physical actions by human beings. But of course there’s always the incompetence factor - which could annihilate the human race without the help of AI.
You need not only propose a “plausible” scenario, you also need to present a reason to believe it will happen. It’s plausible that a rogue faction could infiltrate the military, gain access to launch codes and deliberately start WWIII. It’s plausible that a bio lab could create an organism that overcomes the human immune system and resists all medications. A nonzero chance of any of those happening isn’t proof that they’re inevitable, with or without AI.
Well I’m not claiming that an AI-apocalypse is inevitable, just that it’s possible enough we should start worrying about it now. As for the reason to believe it would happen – isn’t that covered by (2)? If you believe that (2) will occur with near-100% certainty, then that would be the impetus.