Well the most pressing question is whether it will kill us all. There are good reasons to suspect that; Nick Bostrom's Superintelligence: Paths, Dangers, Strategies (2014) remains my favorite introduction to this thorny problem, especially the chapter called "Is the Default Outcome Doom?" Whether LLMs are sufficient for artificial superintelligence (ASI) is of course also an open question; I'm actually inclined to say no, but there probably isn't much left to get to yes.
A lot of smart people, including myself, find the argument convincing, and have tried all manner of approaches to avoid this outcome. My own small contribution to this literature is an essay I wrote in 2022, which uses privately paid bounties to induce a chilling effect around this technology. I sometimes describe this kind of market-first policy as "capitalism's judo throw". Unfortunately it hasn't gotten much attention even though we've seen this class of mechanisms work in fields as different as dog littering and catching international terrorists. I keep it up mostly as a curiosity these days. [1]
That future is boring; our current models basically stagnate at their current ability, we learn to use them as best we can, and life goes on. If we assume the answer to "Non-aligned ASI kills us all" to be "No", and the answer to "We keep developing AI, S or non-S" to be "Yes", then I guess you could assume it would all work out in the end for the better one way or another and stop worrying about it. But we'd do well to remember Keynes: In the long run, we're all dead. What about the short term?
Knowledge workers will likely specialize much harder, until they cross a threshold beyond which they are the only person in the world who can even properly vet whether a given LLM is spewing bullshit or not. But I'm not convinced that means knowledge work will actually go away, or even recede. There's an awful lot of profitable knowledge in the world, especially if we take the local knowledge problem seriously. You might well make a career out of being the best informed person on some niche topic that only affects your own neighborhood.
How about physical labor? Probably a long, slow decline as robotics supplants most trades, but even then you'll probably see a human in the loop for a long time. Old knob-and-tube wiring is very hard to find expertise around to distill into a model, for example, and the kinds of people who currently excel at that work probably won't be handing over the keys too quickly. Heck, half of them don't run their businesses on computers at all (much easier to get paid under the table that way).
Businesses which are already big have enormous economic advantages to scaling up AI, and we should probably expect them to continue to grow market share. So my current answer, which is a little boring, is simply: Work hard now, pile money into index funds, and wait for the day when we start to see the S&P500 start to double every week or so. Even if it never gets to that point this has been pretty solid advice for the last 50 years or so. You could call this the a16z approach - assume there is no crisis, things will just keep getting more profitable faster, and ride the wave. And the good news is if you have any disposable capital at all it's easy to get a first personal toehold on this by buying e.g. Vanguard ETFs. Your retirement accounts likely already hold a lot of this anyway. Congrats! You're already a very small part of the investor class.
I can no more prove my speculation than Dr. Bostrum can prove his, and have no more need to do so than he does. All any of us can do on this topic is speculate. We'll only know for certain what's going to happen when it happens.
If you can't prove it, then debate it! Dr. Bostrom's argument is not mere speculation - he's a trained philosopher, with a comprehensive argument and plenty of his own assumptions laid bare. If there is any thought to your logic beyond "It is what it is, what will be well be", surely you can find something concrete in there you can challenge.
Agreed with you that (a) AGI is a real, transformative, and possibly calamitous event, and (b) with the amount of money getting pumped into it we will almost certainly get there.
However, I have a personal hobby horse here, which is reminding people that if we shut the flow of investor money off using the right economic policies [1], we can almost certainly just ... stop the research outright, before it becomes an issue.
True but my view on that is that governments don't want anyone else to be the first with AI AGI... and therefore they will strive to be the first. Kinda like how a big motivation for the atomic bomb was the threat of Germany developing the bomb.
And the amount of capital means that these companies could pay a lot more for labour than they are paying now for the sorts of labour we would need to deprive them of. It's an interesting idea though.
Never said they don't exist, merely that they are "minimal", aka no other policy I could think of seems like it would obviously lead to lower costs while still achieving the desired outcome.
>I'm also talking about the cost that the system imposes involuntarily on others who are neither actually guilty nor bounty seekers
This is a case from negative externalities. First, consider the simple argument from scale. If you buy the idea that a Bostrom-like AI is both (1) very likely to be created on our current technological trajectory, and (2) will probably kill us all, then it's not hard to argue that the benefits reaped from avoiding that fate justifies a similarly high cost to society, maybe several percentage points of global GDP. After all, you're not just risking deflating present-day GDP, you're risking multiplying all future GDPs by 0. Every country in the First World already imposes tremendous involuntary costs on people for things of much less significance, like 'forcing' you to go through TSA even though you would never in your wildest dreams try to hijack a plane, so the mere existence of involuntary costs doesn't sway me.
Alright, but what would the magnitude of those involuntary costs be? If this policy costs everyone a hundred bucks a year in hassle, we've still got a vexation. There is strong reason to believe that, for almost everyone, it really would be very, very low in absolute terms. How much of the population is currently engaged in frontier-pushing AI research right now? 1%? 0.1%? Actually probably a few orders of magnitude lower. OpenAI still employs less than, what, a thousand people, etc. etc.
The vast majority of people will never do anything remotely like that in their lives. So one would expect very cheap private insurance policies to appear as an effective way to get out of being pursued and harassed by private bounty hunters. The firms which pop up to provide this service would probably get very, very good at negotiating with private bounty hunters very, very quickly, to leave anyone not directly in the know out ASAP. The cost for almost everyone would be on the order of cents per year of protection, that's how unlikely it is that some random clerical worker in Kansas has any serious involvement in creating the next self-improving AI. In exchange, of course, these insurance policies have a very strong reason not to insure people who actually are involved in such activities, and so they could form a critical source of information for helping the bounty hunters target their own search.
>A major source of the international threat is governmental, where private bounties aren’t going to work at all
Strongest argument I've heard so far, thank you for raising it.
First I'll point out it would already be a dream come true for extending our AI doom timelines if the only people who can actually do AI research without fear of being extradited to a bounty-friendly nation is to work in a government lab. The Department of Defense is very impressive, but they still don't move nearly as quickly as private industry and independent universities do when it comes to work like this. That could be generations more of humanity around to live and love and prosper before we get snuffed out. Don't let the perfect be the enemy of the good!
Let's get serious. AI researchers in your home country are the easiest case, because you have unilateral law on your side. AI reseachers in other countries are quite a bit more difficult, because now you're in the messy world of international diplomacy. If the other country adopts a bounty law as well, you both win. If neither of you do, you both lose. But what about the case where one of you does, and one of you doesn't? I posit that here, in the end, you probably have to make it so the bounty-friendly nation is the one that wins, with force - that is, allowing bounty hunters to turn in and extract money from even foreign employees if they get within your borders. And, yes, if the other governments decide to respond by closing their private AI businesses and opening up government labs only ... Well, you've slowed the wave quite a bit, but you're probably going to have to be more careful. No one ever said shifting the Nash equilibrium would be easy.
But you do have other options, even here. One hazy possibility in my mind, would be offering US or EU citizenship to any foreign national who is both (1) a credible AI researcher and (2) precommits to stopping their research as soon as they take the offer. Bounty hunters win because (duh) you now have a heavily pre-filtered list of marks you can watch like a hawk for the instant they slip up. And chances are good that someone on that list will, even after getting citizenship, and then you can extract a tidy sum from them for minimal effort. The foreign researchers who take the offer win because living and working in e.g. New York City as an employee of Jane Street is probably much nicer than working on recursviely self-generating AI in a secretive, underpaid, underfunded, underground lab in e.g. Chengdu. (It's important to remember that cutting edge AI researchers are, almost by definition, really really smart and really really good with computers and math. They have a uniquely great set of careers they can switch into easily.)
The world wins because the risk of self-caused extinction goes down another iota. China "loses", but it loses in the smallest way possible - it decided to undertake risky research instead of telling its citizens to choose something more straightforwardly good for humanity, and it suffered a bit of brain drain. That's aggravating, but it's hardly worth launching a China v. NATO war over. And hey, if China wants to stop the brain drain, they already have a good model for a very effective law they could implement to get people to stop doing dangerous research - that same bounty law we've been discussing.
I freely admit this is the weakest part of my theory, becuase it's the weakest part of anyone's theory. International stuff is always much harder to reason about. Still, whereas most policies I've seen put forward seem to fail instantly and obviously, mine seems only probably destined to fail. That's a big improvement in my eyes.
> First I’ll point out it would already be a dream come true for extending our AI doom timelines if the only people who can actually do AI research without fear of being extradited to a bounty-friendly nation is to work in a government lab.
Yeah, the problem with AI doomers is that they let fantastic baseless estimates of p(doom) drown out much more imminent risks, such as AI asymmetry facilitating tyranny, which is an immediate, near-term, high-probability risk.
If you lower the problem to "stop people from future developments on AI", then it seems pretty easy to get most people to stop fairly quickly by implementing a fine-based bounty system, similar to what many countries use for things like littering. [url-redacted]
I guess you could always move to a desert island and build your own semiconductor fab from scratch if you were really committed to the goal, but short of that you're going to leave a loooooong paper trail that someone who wants to make a quick buck off of you could use very profitably. It's hard to advance the state of the art on your own, and even harder to keep that work hidden.
That only works if all governments cooperate sincerely to this goal. Not gonna work. Everyone will develop in secret. Have we been able to stop North Korea and Iran from developing nuclear weapons? Or any motivated country for that matter.
The US could unilaterally impose this by allowing the bounties to be charged even on people who aren't US citizens. Evil people do exist in the world, who would be happy to get in on that action.
Or one could use honey instead of vinegar: Offer a fast track to US citizenship to any proven AI expert who agrees to move and renounce the trade for good. Personally I think this goal is much more likely to work.
It's all about changing what counts as "cooperate" in the game theory.
Maybe. I'm not very concerned from an x-risk point of view about the output of people who would put in the minimum amount of effort to get on the radar, get offered the deal, then take it immediately and never work in AI again. This would be a good argument to keep the bar for getting the deal offered (and getting fined once you're in the States) pretty low.
If you make the bar too low, then it will be widely exploited. Also harder to enforce, e.g. how closely are you going to monitor them? The more people, the more onerous. Also, can you un-Citizen someone if they break the deal?
Too high and you end up with more experts who then decide "actually it's more beneficial to use my new skills for AI research"
There's an asymmetry here: Setting the bar "too low" likely means the United States lets a few thousand more computer scientists emigrate than it would otherwise. Setting the bar too high raises the chances of a rogue paperclip maximizer emerging and killing us all.
> ... move [to the US] and renounce the trade for good ...
Publicly.
Then possibly work for the NSA/CIA instead.
> ... bounties ... on people who are not US citizens.
Because that's not going to cause an uproar if done unilaterally.
It works for people that most of the world agree are terrorists. Posting open dead-or-alive bounties on foreign citizens is usually considered an act of terrorism.
> Have we been able to stop North Korea and Iran from developing nuclear weapons?
Yes, obviously. They may be working on it to some extent, but they are yet to actually develop a nuclear weapon, and there is no reason to be certain they will one day build one.
Also, there is another research area that has been successfully banned across the world: human cloning. Some quack claims notwithstanding, it's not being researched anywhere in the world.
I wrote a short essay about using a little known economic system to combat black ball technologies in Nick Bostrom's sense.
I keep it online now, even though I have since developed some private criticisms of it, because of the wide variety of ideas it has sparked from others.
A market of human-oriented hardware keys, where the keys are only intended to be sold to actual human beings, with legal or otherwise cash bounties in place for people who can provide evidence of the keys being sold to or otherwise falling into the hands of non-human entities.
As mentioned, a bounty system. Someone who buys a thousand to use would have to be very clever to evade the eyes of all the people interested in profiting off of revealing his actions and getting the chips turned off.
I decided early on that wasting my one existence trying to solve a problem that seemed unsolvable wasn't what I wanted to do, especially given the stakes (it's very easy to reason yourself into working 14 hour days if you think it means saving the world is 0.001% more likely). However, I never found a satisfying argument against the default outcome being doom.
The closest I ever came to a "solution" is to target the economic forces which lead to people researching and innovating on AI. I eventually found fine-based bounties to be an unusually potent weapon, not only here, but against all kinds of possible grey- and black-urn technologies. I wrote up my thoughts very briefly about a year ago, and they still live at [url-redacted], but I suspect the argument is fundamentally flawed in a way I as a non-academic don't have time to suss out.
So I find myself in a strange place: Live my life mostly as normal, with a good chunk of my finances in low cost index funds, just in case the exponential starts shooting up - and just in case we don't all perish soon afterwards. C'est la vie.
Alignment is hard, maybe impossible. Implementing alignment is at least as hard, and might be much harder. Perhaps there is the option to just not build an AGI, safe or unsafe, in the first place.
For one person, this is easy: Pick a different career. For a small group of people, this is harder: Some people might want to build an AI despite the risks. The reasons why often touch on their stances regarding deep philosophical issues, like where qualia comes from. You won't convince these people to see it your way, although you may well convince them your caution is justified. There’s no getting around it: You need to employ some kind of structural violence to stop them.
In both cases the upshot is that no single person so far seems to have ever had the ability to build even an unsafe AI by themselves (proof by “we’re still here”). Few people are smart enough in the first place; of those, few possess the conscientiousness to build out such a project by themselves; of those, the world is already their oyster, and almost all of them have found better things to do with their time than labor in solitude.
The real danger consists of large, well-funded, groups of these people, working closely together on building out an AI - a danger which only becomes likely if you have a large enough population to work with that you can assemble such a team in the first place.
We unfortunately do live in such a world. OpenAI has over 100 employees as of 2022, and Google Brain probably has at least that many. As an unsafe AI seems most likely to emerge accidentally from the work of large groups within firms seeking to maximize profits, we should look towards the literature on the tragedy of the commons for guidance.
In Privately Enforced & Punished Crime, Robin Hanson advocates for a fine-insured bounty system.
Non-crime law deals mostly with accidents and mild sloppy selfishness among parties who are close to each other in a network of productive relations. In such cases, law can usually require losers to pay winners cash, and rely on those who were harmed to detect and prosecute violations. This approach, however, can fail when “criminals” make elaborate plans to grab gains from others in ways that make they, their assets, and evidence of their guilt hard to find.
Ancient societies dealt with crime via torture, slavery, and clan-based liability and reputation. Today, however, we have less stomach for such things, and also weaker clans and stronger governments. So a modern society instead assigns government employees to investigate and prosecute crimes, and gives them special legal powers. But as we don’t entirely trust these employees, we limit them in many ways, including via juries, rules of evidence, standards of proof, and anti-profiling rules. We also prefer to punish via prison, as we fear government agencies eager to collect fines. Yet we still suffer from a great deal of police corruption and mistreatment, because government employees can coordinate well to create a blue wall of silence.
I propose to instead privatize the detection, prosecution, and punishment of crime. […] The key idea is to use competition to break the blue wall of silence, via allowing many parties to participate as bounty hunter enforcers and offering them all large cash bounties to show us violations by anyone, including by other enforcers. With sufficient competition and rewards, few could feel confident of getting away with criminal violations; only court judges could retain substantial discretionary powers.
Hanson does not focus on any specific suite of crimes for the mechanism he proposes. So let’s try conspiracy. Suppose a conspiracy exists between n conspirators, each with independent percent chance 0% < p < 100% to remain silent over a given period of time. Then the chance at least 1 person speaks up on the conspiracy is 1 - p ^n^ . That’s already pretty good: A 100-person conspiracy where everyone has p = 99%, or a 1% chance of speaking up, has an overall discovery rate of about 1 - (0.99 ^100^) ≈ 100% - 37% = 63 percent. p = 97% has one of 95.
Now suppose bounties are offered to the bounty hunter at a rate of $1000 per person turned in. Do I think, in the 100-person company envisioned, my own chances of keeping quiet would drop from 99% to 97%? For a bounty of 99 grand? Absolutely. People grind Leetcode for months to get comp packages like that. Even if I had to implicate myself in the documents I released, I would just be paying a bounty to myself. And even if I fully believed in the mission, I might justify it to myself by saying that that kind of runway allows me to strike out on my own and attempt my own lone-genius AI production on a remote island for a decade. Now suppose I decided against turning in my coworkers - would I, myself, want to stay there? Absolutely not. The bigger the company gets, the more of a risk there is of me being turned in myself.
It is easier to shift the Nash equilibrium of working on AI in the first place than it is to create safe AI. Financial incentives have driven the vast majority of AI improvements in the last decade, and financial incentives can be used to stop them.
Indeed, B = $1000 is low considering the stakes at play - or considering the money would come directly out of the pockets of the guilty. A better metric may be to peg the bounties directly to 10 years of TC ( $2.25m, as of 2022, likely to be higher by the time you read this). Even if the accused shirked getting an insurance policy or saving funds to cover it, they, as highly skilled, remote friendly workers, could almost certainly work them off over the next 10 in non-AI fields from the comfort of a minimum security - traffic monitored - prison.
Well, you can police the entire world quite cheaply on this or any other scientific research program by using fine-insured bounties to do it. Whether it's a good idea is a different question. https://web.archive.org/web/20220709091749/https://virtual-i...
>Oh wait…every hedge fund bro is already doing this. And most of them aren’t billionaires.
The contra to this is some of them are billionaires, and therefore this strategy is working, but for just a few of them.
>why would any one system ever have a large majority of the compute? Compute will be distributed in a power law.
A power law probability distribution means one system absolutely can have a large majority of the compute. Player 1 gets 80% of the compute, player 2 gets 16% (80% of 20%), and so on. The scaling constant would have to be very weak indeed in order to avoid this fate. In fact, the wealth of those hedge fund billionaires probably fit a power law themselves. But to be fair, that power law does indeed to be (for now) weak enough that we don't have to worry about e.g. Jim Simons being richer than every other hedge fund manager put together. So there's a buried assumption in here that the power law scaling is weak, and that is something purely empirical.
>The smart regulation isn’t capping the FLOPS in training runs. That’s creating a powder keg. If the FLOPS are artificially restricted, and one person breaks the restriction, you could end up with a single dominant system.
The smart regulation is to use something like fine insured bounties [1] to give people a very strong incentive not to break the FLOPS cap, and to heavily financially reward people who turn in other people who are doing so. If such a mechanism didn't exist, then I would agree, the free market would be our next best bet to deharsh the power law.
[1]: [url-redacted] - I sketched out the mechanism a few years ago, but sadly it didn't generate much interest online. To be fair the atmosphere was a lot more anti-regulation back then.
> The contra to this is some of them are billionaires, and therefore this strategy is working, but for just a few of them
I have no clue how hedge fund managers make their money, but I was under the assumption that it involved charging their clients hefty fees for managing the funds.
I do, and that's actually incorrect in a strict sense, but it's correct-enough for the average person to follow to Vanguard et al and have a generous nest egg without a lot of risk attached.
Your intuition however is correct in the sense that there is a principal-agent problem at play with all kinds of hedge funds, where if the hedge fund manager isn't the one investing his own money he is by default incentivized to do things besides just maximizing hedge fund profits. But there are indeed managers who have such an ability to generate edge that they do in fact invest their own money solely, usually money they generated while working for other hedge fund managers before striking out on their own, and these people are terrifying forces to watch in action indeed.
When I read George's thought on this immediately Alexander Gerko and XTX Markets came to my mind. They operate one of the worlds largest GPU clusters (10k A100) [0].
They aren't really a hedge fund but a prop trading firm, but they seem to be winning the game [1].
A lot of smart people, including myself, find the argument convincing, and have tried all manner of approaches to avoid this outcome. My own small contribution to this literature is an essay I wrote in 2022, which uses privately paid bounties to induce a chilling effect around this technology. I sometimes describe this kind of market-first policy as "capitalism's judo throw". Unfortunately it hasn't gotten much attention even though we've seen this class of mechanisms work in fields as different as dog littering and catching international terrorists. I keep it up mostly as a curiosity these days. [1]
That future is boring; our current models basically stagnate at their current ability, we learn to use them as best we can, and life goes on. If we assume the answer to "Non-aligned ASI kills us all" to be "No", and the answer to "We keep developing AI, S or non-S" to be "Yes", then I guess you could assume it would all work out in the end for the better one way or another and stop worrying about it. But we'd do well to remember Keynes: In the long run, we're all dead. What about the short term?
Knowledge workers will likely specialize much harder, until they cross a threshold beyond which they are the only person in the world who can even properly vet whether a given LLM is spewing bullshit or not. But I'm not convinced that means knowledge work will actually go away, or even recede. There's an awful lot of profitable knowledge in the world, especially if we take the local knowledge problem seriously. You might well make a career out of being the best informed person on some niche topic that only affects your own neighborhood.
How about physical labor? Probably a long, slow decline as robotics supplants most trades, but even then you'll probably see a human in the loop for a long time. Old knob-and-tube wiring is very hard to find expertise around to distill into a model, for example, and the kinds of people who currently excel at that work probably won't be handing over the keys too quickly. Heck, half of them don't run their businesses on computers at all (much easier to get paid under the table that way).
Businesses which are already big have enormous economic advantages to scaling up AI, and we should probably expect them to continue to grow market share. So my current answer, which is a little boring, is simply: Work hard now, pile money into index funds, and wait for the day when we start to see the S&P500 start to double every week or so. Even if it never gets to that point this has been pretty solid advice for the last 50 years or so. You could call this the a16z approach - assume there is no crisis, things will just keep getting more profitable faster, and ride the wave. And the good news is if you have any disposable capital at all it's easy to get a first personal toehold on this by buying e.g. Vanguard ETFs. Your retirement accounts likely already hold a lot of this anyway. Congrats! You're already a very small part of the investor class.
[1]: [url-redacted]