Inhuman behaviour is a problem that scales with intelligence.
Evil cat? Lock it in a room whenever it does evil things.
Evil human? Call the police
Evil billionaire? Protest/push for law changes whenever his company does evil shit, hope it’s enough to blunt the worst of his behaviour.
Evil superhuman ai? Guess I’ll die.
Edit: to be clear, don’t think billionaires are smarter, but felt wrong to ignore them in the list, consider them the worst case of a single evil human.
A: That’s true until it isn’t. Preparing for/predicting things before they happen is our best hope for not sticking our collective heads into a guillotine any time soon.
B: corporations are only very weak analogues of superhuman intelligence, they’re different from us in “wisdom of crowds” sense (and ofc in the “too many cooks” sense).
But they’re basically just distilled from human intelligence and match our own style of intelligence somewhat closely as a consequence. Also, we’re pretty good at the alignment problem for corporations, they do largely what the combination of their investors, government, society, and workers want because they’re inner workings are fed through human brains at every stage and those humans even if incentivised with money will alter the behaviour of the corporation towards human preferences.
The fact even corporations that have thousands of intelligent human filters (most of whom are presumably in the middle of the human bell curve) monitoring every single mental process still manage to occasionally do terrible things is not a particularly compelling reason to think that a mind that has barely any human understanding or oversight into it’s internal function will be very safe to keep around.
Or a “star in a bottle”