Why does each species exist?
The answer is in the question – To exist. And to exist into the future. Humans, Dogs, Tapeworms, Plum trees responsibility as a living organism is first and foremost to its own species. Yes, humans domesticate and care for pets, help save the whales and do conservation work with turtles, but if the shit really hit the fan (possible attack by AI is a good example) do you think we would give a shit about the whales or turtles (Domesticated animals might be less likely to be cast aside, as their connection in a familial sense to humans).
But would Artificial Intelligence have the same survival priority? Why should we assume that a non-biological unit would share the same biological and survival impulses and instincts that we do? That if it comes down to it, it will further its own survival at the expense of all other species? Maybe it’s possible that a super intelligence AI won’t make survival of its own (I wanna say, ‘systems’) a priority, or even slightly important.
It’s stupid for humans to program computers that stay the same. And if we program AI to aim to improve itself (like we already do), well, the issue of propagating its own kind and ensuring its own survival may not be a biological urge, but merely a systemic, mathematical one.
And we may even consciously (or unconsciously) program that biological urge for survival into it. In that case, yeah we’re probably fucked.