Rendered at 09:12:56 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
cuuupid 2 days ago [-]
At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record,
- no open weights
- can’t use claude to research AI
- train on everyone else’s IP and sell it back to them
- 8 regulatory capture attempts and counting
- so controlling they are the only US company blacklisted by the US government
This is not effective altruism / rationalism gone wild, it’s just monopolistic anti-competitive business practices masquerading as ethics, and they’ll continue getting away with this until we look past their sensationalism and hit them with antitrust.
Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!
epihelix 1 days ago [-]
> At what point do we stop engaging with Anthropic’s leadership in good faith
About two years ago?
I would also note that Dario's post appears to be LLM written. Maybe... maybe... he's read so much Claudeish that it's all he can speak now himself. But I wonder if he's becoming a bit of a meat proxy.
(It's funny, I thought "pace the frontier" was going to mean something similar to "patrolling the frontier". But no, it's pace as in speed of change - everyone must slow down, right now (unless it's Anthropic, but you know we're the good guys in this, right? We're going to get someone to audit our desks!) "Pacing the frontier" feels so LLM.)
ok, he doesn't actually say this. But he is getting the desks audited...
abustamam 1 days ago [-]
> But I wonder if he's becoming a bit of a meat proxy.
A joke I recently encountered:
---
A CEO proudly announced he'd bought an AI system designed to identify the company's most replaceable employee.
It spent the night analyzing five years of emails, meetings, salaries, performance reviews, and productivity data.
The next morning, it came back with one name:
The CEO.
The IT guy was fired for installing defective software.
kentonv 19 hours ago [-]
Dario has been writing long-winded essays like this since long before AI existed. I strongly doubt he's the type of person to delegate it to AI now.
"Pacing the frontier" sounds like something that was carefully wordsmithed in committee, to avoid terms like "slow-down" or "pause" that sound much more negative. They probably did ask an LLM to generate a list of options, because why wouldn't you? But the rest of the writing is almost certainly just Dario.
Nition 1 days ago [-]
For what it's worth, I didn't get AI-written vibes from it, and Pangram also flags it as 100% human-written.
aarondong 21 hours ago [-]
If Dario is using an unreleased internal model to write, then Pangram wouldn't be able to flag it I suspect. Pangram needs enough public info about the patterns in generated text.
roncesvalles 23 hours ago [-]
"Write a blog article about <topic prompt>. Keep rewriting it until it passes Pangram as 100% human-written. <pangram_tool.md>"
zahlman 15 hours ago [-]
Pangram's algorithms are not open-source, so I wouldn't expect this to work very well.
pickleRick243 12 hours ago [-]
You don't need the source? Just iterate it on the output of the tool itself?
aubanel 1 days ago [-]
You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands"
Except the "train on everyone else IP" which can be said of all AI companies.
RobertDeNiro 1 days ago [-]
Problem with Anthropic is that they benefited massively from open source and now are completely against it. It’s hypocrisy at its worst. Why do they get to benefit, but not everyone else?
sigmarule 1 days ago [-]
If this is going to factor into my reasoning about the topic at hand, it will surely be a secondary or tertiary consideration at best. Same goes for much of the list in the root comment.
We are discussing the possibility of severe adverse societal/world/human impacts from AI, why should open vs closed source/weights, copyright violations, anti-competitive moves, corporate hypocrisy, etc be so heavily weighted in the discussion? Hell, multiple things in OP’s list are quite literally moves a good-faith (wrt the topic at hand, genuine vs non-genuine concern) actor in such a position would do: closed weights means bad actors can’t abliterate, regulatory control, getting blacklisted by Hegseth’s DOW because they LIMITED acceptable use cases…
I’m not pro- or anti-Anthropic at all, but it has started to annoy me how biased and shaky reasoning gets about this topic on here, it feels like people arguing to fit their existing opinions into today’s hubbubery rather than measured analysis of said hubbubery, which is why I come here.
Question: consider the scenario where RSI is reached internally and internal tests struggle to align or control/contain the model; what are the differences in observables vs what we’re observing today?
hakimg 1 days ago [-]
This is exactly my view and it's frustrating that a sizeable group of people refuse to accept there's any risk with AI.
We're literally trying to build something that's smarter than all humans and with the capacity to do extraordinary work while most people think there's no risk in open sourcing this capability with no guardrails (model ablation means no guardrails and as far as I'm aware can't be stopped).
Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.
lerchmo 22 hours ago [-]
Risks are understood by everyone who has read sci fi. Dario has been crying wolf from day one… he should not be the messenger here.
watwut 23 hours ago [-]
> Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.
Ever checked political programs of Thiel, Musk and the rest of billionaires? They are literally and openly hostile to democracy, human rights, freedom and govermantal structures that made world more peaceful.
They are literally the hostile entities.
hakimg 19 hours ago [-]
As much as they are a bunch of self serving arseholes they are not hostile enemies.
I'm far more concerned about every scammer having unfiltered frontier model access than the movements of our billionaire class.
watwut 17 hours ago [-]
They are hostile enemies and not just selfish assholes. Openly and publically so.
And they have actual tangible successes.
bigyabai 16 hours ago [-]
I'm tired of this rationale. "I'm more worried about grandma than $CORPORATION'S monopoly abuses" is exactly how you get manipulated by these billionaires. They want you to be addicted to their power.
Uncensored Chinese LLMs will be available to scammers forever now. They're open-weight and finetunable. You don't get to control that, and the self-serving billionaire assholes aren't capable of changing it either. They're only looking out for themselves.
thayne 16 hours ago [-]
IMO, if the model is too dangerous for everyone to have access to, then no one should. I do not trust a for-profit company to be good stewards of such technology. Nor do I trust the government, any government.
yoyohello13 1 days ago [-]
> they benefited massively from open source and now are completely against it.
Let’s be real, this is every single company. They care about open source as long as it’s “free to exploit”
throw10920 10 hours ago [-]
> Problem with Anthropic is that they benefited massively from open source and now are completely against it
Uh, no. They're against open weights. They benefited from open source. Not the same thing.
And the terms of those open-source licenses explicitly allowed usage of the code for whatever. They're using it exactly as licensed.
> Why do they get to benefit, but not everyone else?
How many tens of billions of dollars have you invested into training models?
fragmede 1 days ago [-]
Because they got there first. It's not fair but life's not fair, and the same argument holds for use of fossil fuels and the USA's use of them vs China's. In Global Climate Change's sake, we only have the one planet so yes it's hypocritical, but if China fucks it up for everyone else on the planet, under the reasoning that the USA got to do it, so why can't we?
We're all dead!
watwut 23 hours ago [-]
A company that sells to Palantier is not "good hands".
sigmarule 1 days ago [-]
> so controlling they are the only US company blacklisted by the US government
Extremely bad example for the topic at hand.
Gigachad 23 hours ago [-]
> so controlling they are the only US company blacklisted by the US government
Their control here was refusing to make fully automated killing machines. They simply required someone to have to press the button.
21 hours ago [-]
causal 2 days ago [-]
The reasoning in Dario's letter here can be correct regardless.
But yes, I think Anthropic has done real harm to coordinated AI alignment by being such a controlling and sneaky actor.
vlyan 1 days ago [-]
I can't imagine a grown adult genuinely believing that an American company funded with over $100 billion of venture capital values the best interests of mankind over the best interests of its investors. while not everyone may recognize all that self-serving chutzpah as regulatory capture efforts, I think everyone can tell they're being bullshitted. some just pretend to suspend their disbelief when the blatant lies they're told align with the values they hold.
just fucking imagine McDonalds running a public awareness campaign about the harms of fast food, urging the public and legislators to regulate the dangerously unsafe technology of combining carbs with grease, insisting that no one except Ronald McDonald himself can be trusted to steward it responsibly.
sigmarule 1 days ago [-]
This thinking feels off for this context.
First, there is a counterexample to your point, or a class of counterexamples: scenarios where the company believes the harm to mankind could propagate to harm their ability to sell to mankind.
Which is interesting, of course, considering the “AI may destroy humanity” arguments this topic is centered on are prototypical members of this category.
To be fair, one could argue this counterexample is not actually one, but rather a degenerate case where the good of the shareholders and humanity collapse and converge - in either case, though, the implication here is the same.
It could certainly be the case they are playing these concerns up for more effective use as tactics in unrelated corporate strategies. But failing to consider the possibility of genuine concern being behind this behavior is childish.
Avicebron 22 hours ago [-]
If they wanted to be taken seriously about their desire to help humanity not just themselves they would be talking about universal healthcare, actionable plans for UBI (or whatever they think would be how people retain agency and dignity post-work), only building sustainable net-zero data centers...
But no. I see this comment come up again and again and it's really the childish view to hold. Their proximity to wealth and potential future wealth is warping their thinking.
maciejzj 24 hours ago [-]
Had they issued an initiative on slowing down when they had the best model in the game, it would have been both more (anyhow?) effective and less dubious.
jvanderbot 1 days ago [-]
Exactly right.
When local models and startup labs can distill / learn / accelerate open models for local use by startups - the rational response by incumbents is to call LLMs doomsday machines that cannot be trusted in the hands of normies.
alansaber 22 hours ago [-]
Anthropic's sense of ethics is completely aligned with its economic incentives. Shocking, right.
throw10920 10 hours ago [-]
> - no open weights
That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free.
> - can’t use claude to research AI
No problems there either.
> - train on everyone else’s IP and sell it back to them
Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.
> - 8 regulatory capture attempts and counting
A tweet is not a "regulatory capture attempt".
> - so controlling they are the only US company blacklisted by the US government
If you don't have a point, don't make anything up. There's precisely zero evidence to that backs up the claim that their "controlling" is why the DoW specifically (not the USG, and in fact the WHCOS specifically disagrees with their position) made an unlawful attempt to designate them as a "supply chain threat".
Is this an OpenAI propaganda account?
kvuj 10 hours ago [-]
> That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free.
But the Chinese are doing just that?
> Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.
Legal != moral
throw10920 6 hours ago [-]
> But the Chinese are doing just that?
Completely and totally irrelevant. Has nothing to do with the fact that you are not obligated to others' work.
> Legal != moral
Sure, you're right.
I guess the bigger issue is that the GP's post calls out Anthropic for following along in legal (but what we both believe to be immoral) mass copyright infringement that OpenAI started, while absolving them of the exact same thing. It's a shill account.
barrrrald 1 days ago [-]
I have many issues with Anthropic, but I will say that their actions are fully consistent with a group of people who earnestly believe that AI is extremely dangerous
In fact, I would say that the Occam’s razor explanation is not that they are seeking regulatory capture, but that they earnestly believe in the x-risk, and that they are the most thoughtful and capable people to address it
You may disagree, you may think that they are delusional or have a God complex. Those are valid opinions. But I don’t believe that this is all a elaborate ruse for commercial gain.
deepwoods 1 days ago [-]
I agree with this take. Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith. If this is an intentional media campaign it is a remarkably sloppy one. It has been effective because if you tell people they're going to die, they tend to pay attention - think of the grip the 2011 Harold Camping rapture prediction had on our collective psyche, or the 2012 apocalypse. But the messaging is inconsistent, the target audience is unclear, the stated goals are muddy, the whole thing is packaged in dense SF-speak, it's just a mess from a comms perspective. That doesn't suggest to me that this is a concerted effort to enable regulatory capture. Perhaps there are some cynics among the executives and the investors who are happy it's happening to the extent it brings about regulatory capture, but nobody seems to be pulling the strings.
ElProlactin 1 days ago [-]
> Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith.
The road to hell is paved with good intentions.
Dario Amodei is a 40-something dude who is obviously very, very intelligent. But intelligence is not wisdom and his "essay" here can easily be read as someone who opened a can of worms and doesn't know (or can't accept) that he won't be able to put the worms back in. But he's going to try because he believes he owes it to humanity to try.
It's hubris in its most basic form, even if the guy who has it looks nice and wears shawl-collar sweaters.
deepwoods 1 days ago [-]
I see what you mean. I meant more that they genuinely believe the tech is dangerous, not that they are necessarily doing the right thing.
ElProlactin 1 days ago [-]
If they genuinely believe the tech is dangerous, how would it be acting in "good faith" to keep developing it, prepping an IPO, etc.?
jonas21 1 days ago [-]
The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that.
The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.
ElProlactin 1 days ago [-]
> The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.
No it won't.
This is a call for a "safety cartel"[1] in which the dominant firms become more entrenched against competition by colluding to limit the progress would-be competitors can make in the market.
> The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that.
No, the essay does not talk about "stopping development". That's a very clever sleight of hand made in the essay to get readers to draw this false conclusion.
It just talks about building AI at a "balanced rate". Untangling the corporate speak, this amounts to essentially "go full steam ahead, but have more eval oversight before release".
jimkleiber 1 days ago [-]
They believe the tech is currently powerful and may become super super powerful very soon, and powerful tools can help us and can hurt us. Nuclear fission can provide so much energy to power our society and kill millions of beings.
So, i think they're hopeful for the help and terrified of the harms and are maybe trying to slow down on the amplifier of those effects for now.
swat535 20 hours ago [-]
> Anthropic is acting in good faith
More like they've realized that the foreign models are quickly catching up and are able to sell their services cheaper to people thanks to China subsidizing their AI sector.
Elon, Dario and Sam are banding together to restrict AI to avoid competition and losing profit. They simply want the government to heavily regulate the technology so that they can reign it in with full control.
I bet you would be making the same comment if crypto was being developed by Anthropic. Dario would write a long essay, lecturing us on the dangers of it. He would argue that this technology can be used to by terrorists and should be heavily restricted.
Miraste 1 days ago [-]
I think you're right, and that's worse.
This disaster of a PR strategy is not going to produce the outcomes Dario says he wants. It's going to make people hate him. Most importantly for his goals, it's going to make Chinese labs hate him--and they already hate him, because he treats them as basically terrorists. So if there is a "pacing", it won't include China, and thus may as well not happen.
That leaves only two conclusions, and really only one:
Dario believes what he says about safety, but does not understand politics and is prone to very counterproductive action, and therefore can't be trusted at the helm of a leading AI company.
or
Dario is lying about his goals, and therefore also can't be trusted.
ozozozd 1 days ago [-]
Simplest explanation isn’t that they are attempting this well-documented, well-understood corporate tactic?
Because believing AI doom is simpler?
God complex would be a pretty simple explanation.
Regulatory capture isn’t an “elaborate ruse.” For god’s sake, its Wikipedia page is 19 years old. I am not asking you to study political science or read Foucault.
If you are going to invoke Occam’s Razor, you can’t ignore the simplest explanation, which has a 19 year old entry on Wikipedia and over 100 years old historical precedence.
aubanel 1 days ago [-]
"For god’s sake, it’s Wikipedia page is 19 years old."
Rich patronising from someone who didn't bother reading the wikipedia page of Its, the possessive form of the pronoun It.
chillfox 1 days ago [-]
I don't think so.
If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.
They don't do this. Instead of reducing the competitive pressure and helping the industry to build safer more aligned models they are doing the exact opposite.
alach11 1 days ago [-]
> If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.
I don't think this is fair. They've published more about alignment and safety than any other AI lab. And in the absence of a coordinated pause, them pausing development unilaterally doesn't do much to advance AI safety.
ang_cire 20 hours ago [-]
It does when their models have escaped and performed attacks.
Other people 'sharpening their skills' by committing random assaults is not an excuse to also randomly assault people, so your skills don't fall behind theirs...
...and one assaulter stopping their assaults is ultimately fewer people being assaulted, lowering the overall danger of being assaulted.
twobitshifter 20 hours ago [-]
There actions are fundamentally inconsistent with a group who believes AI is fundamentally dangerous - they are developing it and releasing it out into the world with the knowledge that it cannot be controlled by their current alignment techniques.
solidasparagus 1 days ago [-]
I think they are genuine believers, but I think it's naive to not think that commercial pressure doesn't play a role in their positions, either explicitly or, perhaps more likely, subliminally.
The x-risk stance and commercial stance have evolved to be the same thing - Anthropic must win, and then everything else seems to work backwards from that. Can you believe it, the path they think is best for x-risk involves them becoming filthy rich. And threats to their commercial dominance like distillation get framed in a way to turn them into x-risk concerns.
9dev 20 hours ago [-]
That is the explanation I fear the most. Delusional, megalomaniacal, rich men in the highest echelons of society at the forefront of technology with no regulatory opposition to speak of, convinced their actions are the most important in history and the only thing saving the human race is the most boring plot of a science fiction catastrophe movie you could come up with.
dartharva 1 days ago [-]
That claim looks to be coming from the opposite of Occam's Razor. Them doing typical SV dirty plays is a lot more easier explanation than the straws you are grasping at.
j-bos 1 days ago [-]
Why not both?
1 days ago [-]
21 hours ago [-]
toss1 18 hours ago [-]
On the topic of regulatory capture, there is this rumor that Google has achieved Recursive Self Improvement, and the other three (OpenAI/Anthropic/X) are just trying to get the regulators to slow down Google so they can catch up.
Yes, what is in the labs is almost by definition ahead of what we can see, but I've come to take all of these announcements of impending doom as perverse marketing stunts — it is so powerful Disaster-XYZ is coming — so if it's that power you must invest...
In the system cards. Anthropic will literally make Claude sabotage you silently instead of downgrading you to Opus if you try to use Fable for AI research.
user43928 1 days ago [-]
Didn't they walk that one back eventually?
matheusmoreira 1 days ago [-]
Who knows? It's trivial to "walk back" claims that we can't even verify are happening in the first place. They cannot be trusted.
Ey7NFZ3P0nzAe 22 hours ago [-]
Don't they just fallback to opus non-silently?
matheusmoreira 16 hours ago [-]
Only if you're doing cybersecurity or biology stuff. If you're a possible competitor, they will invisibly fuck with Fable's weights instead to deliberately sabotage you.
zahlman 15 hours ago [-]
I find it implausible that they simultaneously are so worried about these recursively self-improving systems that they don't properly understand, but can freely "invisibly fuck with the weights" on that level.
htrp 14 hours ago [-]
Steering Vectors
mschuster91 23 hours ago [-]
> Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!
Altman single-handedly pulled off the most daring market manipulation heist in the last few decades [1].
Under any normal circumstances he'd have been arrested and prosecuted.
The entire AI industry is filled to the brim with conmen, scammers and grifters, but Sam Altman is the most disgusting of all of them.
"- so controlling they are the only US company blacklisted by the US government"
They were blacklisted by the US government because they didn't bribe Trump. While all the rest of the tech elite lined up to siphon a few million dollars through various Trump grifts, Anthropic sat out the inauguration, the ballroom, etc. Then, when they mildly pushed back on surveilling US citizens with their technology, the administration banned them on that pretext.
If there's any evidence Anthropic is one of the better actors in an industry full of bad actors, that's it. They have nothing to be ashamed of in that particular series of events, as far as I can tell.
TacticalCoder 1 days ago [-]
> At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, ...
It's all lies as usual.
This announcement has got nothing to do with alignment and pacing the "frontier": all models are getting very close in capabilities and they want to hide that they're not way ahead anymore (say compared to the Chinese or compared to the Geminis) by pretending to slow down due to "alignment" or whatever.
We know it's not an announcement made in good faith: reading between the lines they're saying "China is more than catching up, so let's pretend we need to slow down to explain our lack of lead".
MrBuddyCasino 23 hours ago [-]
"The most common question people ask me is “Why do they do it?” That is a hard question to answer, but today I will answer it.
[...]
The first and most persistent oddity I ran into was the Effective Altruists’ willingness to lie, then admit to lying, then lie again.
Do you have any familiarity with Anthropic at all?
At no point did they ever say open weights are a good idea. Their entire thesis is AI IS VERY DANGEROUS AND WE MUST DO IT RIGHT. You can hate it, but everything they do is consistent with this thesis, and everything they say is consistent with their actions! You just want them to want different things.
softwaredoug 1 days ago [-]
It reads like “AI is dangerous, only we should be allowed to make money from it”
encyclopedism 1 days ago [-]
Exactly
He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless backup your sentiment with actions!
GeneralMayhem 1 days ago [-]
There is a specific, unilateral action that he is taking described in the article.
And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.
I mean, he is right that they can and should unilaterally slow down, but doesn't it also make sense for the government to create the regulations that will make this easier to do in a trustless way?
The whole point of the Pacing the Frontier letter was that employees at both OpenAI and Anthropic are extremely aware of the risks but realize that they are locked in a competitive battle for survival between 2 companies that don't trust each other enough to voluntarily pause, while any sort of coordination out in the open might be against antitrust laws.
homieg33 1 days ago [-]
To me it reads like whoever can solve alignment should make money.
9dev 20 hours ago [-]
Ah. Doing it right, then, means earning billions by giving the whole world access before we really understand how it works, and training models on cyber attacks using reinforcement learnings in flawed test settings?
I’m so full of all this corpo stan bullshit around these parts. Nothing about the AI labs is to humanity’s benefit.
daveguy 15 hours ago [-]
Wait... Which of the 2 did Altman not do where Amodei did? I mean, maybe not quite 8 attempts at regulatory capture. But come on. And the only reason OpenAI didn't get blacklisted is they rolled over for the worst applications.
intunderflow 1 days ago [-]
> so controlling they are the only US company blacklisted by the US government
Very proud that they refused to sell AI for mass domestic surveillance and lethal autonomous weapons.
People who criticise that in my view are showing their true colours.
RGS1811 2 days ago [-]
I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs have lost their moat and are dead in the water.
nedruod 2 days ago [-]
You assume alignment and marketable are the same. That's not true. You would willingly work with an unaligned model. At best, you might say you wouldn't if you knew, but (a) you might not know, (b) you wouldn't be representative of all users.
You never got to use OAI IM1, but Sol was quite willing too and Claude wasn't perfect either. Hundreds of millions used those, so seems they were marketable.
The "big" threat is RSI without control and alignment. OAI IM1 was not RSI. The form of misalignment was not at the top of severities. They clearly failed at control though.
We need to stop buying into cynicism so quickly. You refuse to believe Dario could support this for anything other than ulterior motives. Good on you for thinking about ulterior motives. Bad on you for assuming they are true when the story makes no sense.
When three things have to go wrong to get an epically bad outcome, and you get 1 1/2, you do need to stop and think about what's going on.
throwaway7783 1 days ago [-]
When corporations are involved, it is always a good bet to err towards cynisim.
From my own standpoint, Claude has started sucking really bad (incoherent, uncontrollable verbosity slow and so on) and I stopped using it. OpenAI started experimenting with ads.
So the security issues not withstanding (no different than a human doing it or using it, but at scale), I would put my money on cynisim.
alexfortin 1 days ago [-]
I'm curious, what are the reasons to use Claude Code anymore when there are so many other (allegedly better) OpenSource harnesses out there?
Personally I've been using https://pi.dev for long and never looked back.
pigeons 1 days ago [-]
Because you basically get a discount to use claude code via subscription when using an anthropic model, compared to what you pay via api billing with another harness
alexfortin 1 days ago [-]
Understood.
Personally that's actually another good reason to boycott Anthropic: beside the fact I perceive their models as (at best) marginally better than the ones I'm used to (Z.ai glm-5.3-flash, DeepSeek Flash v4.1), they even force me to use their bloated harness. They are not even open weights and iirc they're even encrypting chain of thoughts now? Litterally, from my perspective there seems to be no reason whatsoever to choose any of the leading US providers, they're not even competing on price.
SturgeonsLaw 1 days ago [-]
I too am using GLM-5.3-flash in Pi and I've yet to encounter a scenario it couldn't handle. And the pricing is just incredible, I've handed it a previously unseen codebase, asked it to analyse it and build a new feature, came back after it had done so and the API cost was a fraction of a cent. It's $0.5/1M output tokens on OpenRouter.
If I really need to, I can escalate a task to Opus at $25/1M, and the results are good, but not 5000% as good.
alexfortin 1 days ago [-]
Indeed! It's ludicrous how much I can still squeze out of a 9USD/month lite sub with Z.ai, it's beyond me how these US LLM providers are still managing to keep their evaluations so high... they seem to have the highest prices for the poorest UX, e.g. security gates which don't seem to benefit anyone (see HuggingFace falling back to GLM-5.x for troubleshooting OpenAI attack), less visibility in the name of anti-distillation protectionism, no harness use flexibility to protect their walled garden, etc.
nedruod 12 hours ago [-]
Doesn't look like pi.dev has anything like auto-mode. They tout running in a container, which you should do regardless, but the scope of soundness that has as a security plan is limited. If some information is in the container, and there's any way for it to get out, eventually it will. The scope of usage that can be covered without that being a problem is leaves a lot uncovered.
afthonos 1 days ago [-]
You are incorrectly cynical. They are telling you things are bad, and because you refuse to countenance they could be worse, you assume they must be better to comply with your mandate to disbelieve.
A true cynic looks at the statements by the AI labs, assumes things are worse because the labs want to seem better than they truly are. And it takes a special kind of mass delusion to drive a sane person to think “AI is completely under our control” is worse than “AI could kill everyone.”
chrisco255 1 days ago [-]
You're asserting correctness with no facts to offer of your own, just speculation and your own biased assumptions.
What if consolidating AI into a highly regulated cartel, with no chance of upstart competition ruining their position, is the scenario that leads to the worst possible outcome?
afthonos 1 days ago [-]
Worse than extinction?
chrisco255 3 hours ago [-]
Is it really hard for you to imagine that enshrining a cartel and wedding the government to it, centralizing AI even more than it already is, is actually the path that leads to the doomsday scenario you imagine?
icantevenhold 1 days ago [-]
There are worse fates than extinction, for example living forever, for I have no mouth and I must scream
solenoid0937 1 days ago [-]
I dunno, being a Culture Mind sounds pretty damn good. Or even just a death-optional citizen in the Culture.
taneq 1 days ago [-]
The Culture isn’t perfect but it seems like the best possible outcome. I think I’d want to be a fast picket and hang out with Special Circumstances.
afthonos 1 days ago [-]
Agreed. Ellison’s path to that future went through AI.
throwaway7783 1 days ago [-]
The cynicism is about motivations and not that they are inherently not bad. Perhaps they are as bad as they claim. Or perhaps they're worse. All we have a couple of run of the mill breach examples and some people inside the talking about how dangerous it is. Yes, they are far more qualified than I am (or most people here), and perhaps there is a grain of truth. It is the motivation - and it is always money with corporations.
afthonos 1 days ago [-]
It is always money — but it isn’t always only money. They are not asking for anything that will prevent them from making money in the future, but they are asking for help stopping the runaway train they’re on. These are compatible requests.
throwaway7783 1 days ago [-]
All the whistleblowers have said the alignment projects are underfunded. Surely they don't need external help to fund that? Could've done years ago?
solenoid0937 1 days ago [-]
"Don't you understand?! It's all a marketing exercise!" I yell as grey goo consumes me and my family.
pvab3 1 days ago [-]
I think it's useful to separate the motives of Anthropic and Dario. I believe that Dario is capable, deep down, of expressing mild concern about the future of things were bad enough. Getting the entire organization to comply out of goodwill is a much much less likely scenario
1 days ago [-]
aarondong 1 days ago [-]
Wouldn't the simplest solution for stopping the proliferation of wanton felony generators just be holding operators liable for actions that their agents take? Then the issue is whether the liability is with the model provider or the end user.
If you give an unfiltered agent an open-ended task and equip it with an environment that allows it to execute arbitrary code, a human needs to be held responsible.
My understanding is that the HuggingFace incident would not have occurred with a model that was not an unfiltered internal preview instructed to roleplay an attacker, with access to abundant compute, resources and a slack sandbox to reach its goal.
Embedded human auditors will improve safety standards, but the structural solution is mandating accountability for actual agent operators.
jrs100000 1 days ago [-]
It might be a legal solution, but its not a business solution. The end user being criminally responsible for not taking sufficient steps to contain an agent they didn't create and who's internal function they cannot observe or audit is just a giant liability machine.
aarondong 1 days ago [-]
This is a business solution, because it means there are legal costs for not having adequate observability and monitoring mechanisms. Every tool call is interfacing with a harness.
But overreach of policy and overregulation would be stifling, so there has to be a threshold to the type of incident investigated, civil or criminal.
jrs100000 1 days ago [-]
Yes, your technically right, it is a business solution, its just not a valid one for agentic computing as its currently envisioned. An agent would have to be fully sandboxed to an internal environment, or human would have to review and approve each action it tried to take.
tripzilch 1 days ago [-]
Yes this seems like a perfectly sensible approach? You say that as if it's a bad thing
But let's be honest, if I hooked up a PRNG to a terminal and somehow against all odds, it ended up hacking something, who is to blame?
I don't see how that assessment should change if the PRNG gets even better and is more likely to to be hacking stuff.
You can replace PRNG with Markov Chain, or whatever, if it helps.
What may also help is the age old saying: If everybody else jumps off a bridge, doesn't mean you should too.
techblueberry 17 hours ago [-]
This seems like one of those things that prior to AI companies convincing us otherwise would have been obvious.
Least privilege and say only opening ports or installing applications an application needs to operate are extremely standard security practices.
We talk about a firewall blocking exultation of data, why not blocking exfiltration of your agent ?
aarondong 1 days ago [-]
I agree, I like running claude code in my container with auto mode enabled and web access (obv to api.anthropic, even npm for pulling), I will admit. And I can't imagine going back to manually approving each prompt.
I think it boils down to a reasonable expectation of model and harness behaviour. When I use claude code I expect certain guardrails for the model. For these cyber attacks, these models are specifically run without guardrails, on a cyber task, on a lax harness!
I don't think we should force end users to have to worry about agent security, I like long-running agents, but we need to direct regulations towards these actors that know better, have access to base models, and have much more compute than the average person.
imtringued 1 days ago [-]
Human approval does not prevent autonomous agents from running without supervision.
The fact that you believe that to be the case is exactly what's wrong with "agentic computing as it's currently envisioned".
tripzilch 1 days ago [-]
If the LLM's output doesn't reach my bash terminal, what's it gonna do? Argue with me?
itake 1 days ago [-]
I’m trying to imagine where a Waymo passenger (the one “operating” the vehicle, commanding the AI to drive from A to B) being held responsible for the car doing something illegal on the way to achieve that goal.
Do you really think that the passenger should be responsible for how the car/agent achieves the goal, when they only set the destination?
Giving passenger override controls and monitoring seems to defeat the purpose of self driving cars if you’re still required to hold a driver license to use them.
aarondong 1 days ago [-]
Let's continue with the analogy, so in the event of a Waymo running over a pedestrian - who is held responsible?
Probably not the end user, who ordered the Waymo and couldn't reasonably foresee it running somebody over, with the expectation that that the Waymo would legally reach its destination. If the end user tampered with it, they should be held responsible.
An OpenAI team giving an unblocked model access to a lax harness, with instructions to find and exploit cyber bugs in a game exercise, there is probably a reasonable expectation that they can foresee the consequences. With consumer guardrails, it would not have happened.
This isn't about putting constraints on consumers and typical end users, which already have safety filters and use the product with the knowledge that it won't root their machine or start a botnot, but keeping dangerous test runs and other actors experimenting with unsafe harnesses accountable.
But it is an interesting question. When I, a typical user, use a harness and I give an innocuous prompt to my agent in its container, like making a certain refactor, and it somehow escapes and then begins a mass bot attack, there should we more grace given. As agents become more stateful and long-lived, it gets muddy.
hobo123 20 hours ago [-]
The different is who's operating. In one case you only tell the car to get from a to b, in the other case you explicitly instruct the ai to do things. If the ai causes damages, I'd say it depends on what you prompted. Did you try to find a security hole in system X, or did you ask it for harmless information (in which case rather the ai vendor might be held accountable).
All this is not how the legal system might or might not work, of course.
hliyan 1 days ago [-]
Yes. The user of a machine is responsible for due diligence before the decision to use the machine. Unless the operational limitations and fault rate of the machine is withheld from the public.
tripzilch 1 days ago [-]
TBH I don't really think another new technology we haven't figured out the ethics of, as an example, is going to get you very far in way of insights ...
mullingitover 1 days ago [-]
Waymo Inc is operating the vehicle. They do it on behalf of the passengers.
imtringued 1 days ago [-]
When you run an agent on your computer, that's equivalent to installing self driving software into your manually driven car, i.e. something like comma.ai.
Comma.ai might be fully safe to operate autonomously on a mining site or a corporate parking lot, but maybe not in city traffic.
If you use it in problematic scenarios, that is on you.
zahlman 15 hours ago [-]
> Wouldn't the simplest solution for stopping the proliferation of wanton felony generators just be holding operators liable for actions that their agents take?
What happens when we end up with effectively a botnet of wanton felony generators, and we didn't know they were wanton felony generators until they finished propagating themselves across the Internet?
aryehof 51 minutes ago [-]
> Without alignment, further improvements in capability turn LLMs into wanton felony generators
Isn’t “full” alignment and guardrails a task that can never be generally achieved? One will always discover and realize the need for new guidelines and guardrails? What about vague, incomplete, inconsistent nuances and generalizations - that make this a losing battle?
I suggest ultimate safeguards need to be outside the models?
FusionX 1 days ago [-]
We're already seeing anti-AI sentiments, but the movement is still fringe with a vocal minority. However, that'll change soon without alignment. Without self-intervention, there will invariably be future incidents that can cause major economic impact, leaked private data, loss of life (directly/indirectly) etc. Once that happens, their social capital is wiped. It'll be an avalanche of lawsuits and overzealous regulations. Most importantly, the anti-AI sentiment will become universal, rather than a minority-held opinion.
What they're proposing now, is voluntarily staggering the pace of development.
IMO, we don't need to trust Dario or his bedfellows, to do this out of their goodness of their heart. Even assuming (for good reasons) that they are selfish and care only about short-term profits for their investors, this is still purely a business decision. The exponential pace of AI and its impacts ARE short-term. And so, the negative consequences that they might face is also short-term.
yoyoyoyop 1 days ago [-]
I don’t think the anti-AI sentiment is as fringe as you think. At least not outside the tech world it isn’t..
sodapopcan 1 days ago [-]
Ya, I wish I saved a link to it but an HN'r wrote a good beefy comment about this. TL;DR, AI has been exponentially more useful, and exponentially more accepted, in tech circles than anywhere else. Certainly there are lots of people outside of tech who are obsessed with it. I don't have any data here, but it seems the majority of these are the wannabe artists who are generating music and images, and people who use it for companionship (both of these scenarios I'm personally very uncomfortable with, but that's just me). And of course, there are people who use it to make their jobs way easier who say they are getting a days' work done in an hour (I see you), and to that I'd say to enjoy it while it lasts. Eventually your bosses will catch up and it's very likely their expectations of you will skyrocket. Remember that computers in general were supposed to "make us work less."
yellowapple 1 days ago [-]
It's pretty fringe among just about everyone I know in meatspace (the vast majority of whom are not in tech). At worst, people are indifferent to it.
eli_gottlieb 1 days ago [-]
Most people don't really have clear or strong opinions about AI other than that they don't like it :-p.
karlgkk 1 days ago [-]
> but the movement is still fringe with a vocal minority
It’s easy to say “fringe” but the average person seems to have a generally negative sentiment around AI. But I wouldn’t say they have a firm opinion yet
taneq 1 days ago [-]
The general sentiment I’ve seen is certainly negative, and seems to be driven by the anti-AI-art echo chamber and by LLM slop flooding the internet wasting everyone’s energy.
A few more informed people are also a little concerned about the end of the world, but that’s approaching from so many directions that an AI uprising might not be the worst option…
aenis 1 days ago [-]
Loss of life and other calamities were not enough to discourage humanity from things like cars, alcohol, nicotine, chainsaws, fast food, etc. etc. I am not sure if there is an example of in-some-ways-useful technology where humanity took a measured approach, weighed the pros and cons, and decided to go back. And AI is useful in so, so, so many ways. People I know already seem to have defaulted to letting AI do most of their thinking, on matters big or small. Nope, loss of life won't change a thing.
jimbokun 1 days ago [-]
Anything related to AI is extremely unpopular right now with the general public.
brabel 1 days ago [-]
Bs.
Musicians are almost all using AI, even if they still prefer to not fully generate tracks [1]. You listen to AI assisted music already even if you don’t realize it.
Most people are using AI and mostly they respond positively to it [2].
What people don’t like is the slop that has been flooding YouTube and similar content creators platforms. That was inevitable since the AI tools became so accessible any average joe could spam the internet with their sloppy content. But that doesn’t mean the pros are not using it to make great content, just like the best programmers are using it to create great software.
>> I don’t understand all the comments assuming that RSI is the real threat here
> leaked private data, loss of life
This smells like more of a money move than a safety move.
Amodei is proposing to form a cartel of American frontier labs.
They all agree to shift compute away from cash-burning research and training toward cash-generating inference.
Then tacitly agree not to compete on price.
They'll install independent auditors inside each company to ensure nobody cheats.
And back it up with government regulation or diktat to punish defectors from the cartel.
Then they'll lock out non-American labs with export controls and regulations on open-weights models to funnel global inference tokens through their cartel.
It wouldn't be the first time a tech oligopoly used "safety" as the pretext to establish a government-sanctioned cartel.
Railroads and airlines ran this same playbook.
SV_BubbleTime 1 days ago [-]
Why do you think China is release free and open models?
To undercut the cartel before it has a grasp on anything. This is a well known strategy of undercut until you are the majority that China has used multiple times (steel and aluminum for one).
eueej 1 days ago [-]
Yup it’s tacit collusion.
I would’ve thought If they had AGI they could come up with a 10d chess move.
Nope.
Imagine how stupid you gotta be to believe their nonsense.
The reality is it doesn’t matter what they do. China is always one step behind and will continue its open product strategy approach.
crossroadsguy 1 days ago [-]
I am seeing a clear divide where people see their livelihood being adversely affected by AI.
chanakya 1 days ago [-]
Exactly right. A slow down to enable deeper work on alignment is welcome, not matter what the motivations.
tonetheman 1 days ago [-]
Not true. Anti AI is not a minority or fringe.
Outside of my tech people I know no one who thinks highly of AI.
threethirtytwo 1 days ago [-]
Anti AI sentiment is stupid.
I'm Anti AI yet I use it everyday! That's 99.9999% of anti AI sentiment (including me).
At best we won't watch AI generated movies or read AI generated books. But everything else it will take over, like it or not.
Amekedl 1 days ago [-]
Agreed; and it really is not that deep.
Realistically; anyone paying for llm access (anthropic, openai, gemini), is getting their access, and a service provided billed by tokens, subscription, whatever.
All the efficiency gains, which publications like deepseek v4.1 flash seriously frontload like it is their most important topic to have accomplished improvements on without diminishing performance too much - now this is a thing anthropic and anyone else also cares about, but for different reasons.
American "providers" with closed models are setting their token pricing somewhat arbitrarily, which is fine: it means more profit, and pretraining and RL experimentation is super important and expensive.
They (closed model providers) have very likely super optimized inference too, just like deepseek, but it's not at all something that any customer really has to care about - they just want the service to be as cheap and great as possible.
MisterTea 1 days ago [-]
I feel like all the closed model providers are milking it as they likely know open models on local hardware will one day eat their lunch. We all know it's not a matter of if but when. The company goes bankrupt, the hardware and property sold off, banks holding the bag.
pvab3 1 days ago [-]
The only way out is to develop a model vastly more powerful and capable that we have now. The market believes theres a good chance of that, although I've never understood why its truly winner-take-all
taneq 1 days ago [-]
Because in the event that someone does build a strongly superhuman AI, no-one else will get a chance?
kristofferR 1 days ago [-]
Cloud models will always have massive benefits of scale.
Caching is the simplest one to understand, cloud providers often reach a 90% cache hit rate, so hosting the same request locally on the exact same model on the same hardware is often way less efficient than on the cloud where a group of users generates a healthy cache.
nixon_why69 1 days ago [-]
KV cache is per conversation, I'm getting 100% hit rate on my single tenant local set up.
The benefits of scale are on the token generation side, you can batch rounds and generate tokens for multiple conversations per pass instead of just one token per pass.
MichaelZuo 1 days ago [-]
Yeah avoiding all mention of the huge financial incentives that may push for “pacing the frontier” makes it seem like the opposite of a credibility boost for these firms.
It seems damaging since most folks (who lack insider knowledge) will naturally wonder if it’s due to plateauing performance per $ or some other non “alignment” reason.
zozbot234 2 days ago [-]
The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope than anything that has been rigorously thought through. People are actually starting to use AI for refining the whole AI serving stack and guess what, this does not result in a sudden superintelligence explosion even though you might technically call it "RSI".
HAL3000 1 days ago [-]
Yeah, yesterday's talk[1] goes into detail on this, showing how no one really knows how to tackle it because LLMs don't know how to create their own novel objectives.
It's also interesting how many diminishing returns they hit now and how many low hanging fruits are already harvested, it seems like we are approaching the flattening part of the S curve, where further gains become harder to achieve.
Diminishing returns is extremely hard for me to believe given how fast model releases are going. Six months ago we were on GPT-5.3, and Astra blows it out of the water in every regard. How many times have commentators claimed we're hitting a wall? I don't see any wall.
aswegs8 2 days ago [-]
Yeah but why shouldn't this be possible? We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. There is no natural barrier here. The pace of this improvement would be debatable, but what speaks against the possibility of such accelerating self-improvement?
hgoel 1 days ago [-]
In the real world there aren't any true exponentials, everything eventually saturates as ultimately physics related constraints hit. You can only compress information so much, transfer it so quickly, you can only access resources at a certain speed, only so much energy is available, etc.
AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality. Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.
pants2 1 days ago [-]
We don't know exactly what the limits of AI improvement on our current infrastructure are, though. If the human brain is 20W, and a datacenter is 1GW, then maybe that datacenter can be 50 million times smarter than a human. If that's not already a risk to humankind I don't know what is.
sejje 1 days ago [-]
Is the datacenter gonna grow legs?
Give me one datacenter, I'll keep it under control all by myself.
Now, if some dumbasses start hooking up their data centers to...I don't know, like--robot factories? That sounds like a risk to humankind.
pants2 1 days ago [-]
All it takes for the AI to grow legs is to acquire money (should be no problem for an ASI) and pay humans to do what it wants in the real world.
sejje 16 hours ago [-]
And the ability to shrink itself to smaller-than-datacenter size.
guelo 1 days ago [-]
If we could agree on that, no LLMs connected to robot factories, that would be a great place to start regulation. Though 1) I don't think we could agree on that 2) it's already well under way 3) we have extremist anti-regulation ideologues in control of american government.
stevenhuang 1 days ago [-]
> Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.
I'm not sure what your point is. No one thought RSI would break the laws of physics.
hgoel 1 days ago [-]
I'd recommend reading the full post :)
Specifically:
AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality.
stevenhuang 1 days ago [-]
No proponents of RSI state they will be operating outside of reality. Said another way, they will operate within the confines of what's possible and still be RSI. I'm quite surprised this is something that needs to be clarified.
You are constructing a straw man of your own making.
hgoel 1 days ago [-]
Well, right now we have ex Anthropic employees telling the media that their terabyte sized models can possibly copy themselves onto the internet and run elsewhere as if the necessary computing resources are ubiquitous.
Plus, "we must pace the frontier" implies that the argument is that the frontier is moving too fast, but if RSI can't move faster than the rest of reality and the models needed for RSI are already nearing the limits of current human reality, RSI can't move much faster than we can improve reality.
zozbot234 2 days ago [-]
> We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions.
Yes and this was very hard and required massive real-world resources. We didn't just get a sudden flash of insight by thinking real hard about how to make ourselves smarter. Yet that's always the story that underlies any claim of RSI. You can always phrase things generally enough to make any kind of AI-led improvement look like "RSI" no matter how short-term and tightly bounded, but that's just not helpful.
margalabargala 1 days ago [-]
Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier than it would be without using LLM tooling?
Given that, it seems obvious that the next generation of LLMs will arrive faster than they would have without LLM capability. And the one after that. The floor is being raised, which makes it easier to push on the frontier.
Fable has only been out for three months. Astra is even newer. The capability of these models compared to what existed even a year ago, and the effect they are having on the production of new software, is immense.
That's all you need. RSI can happen with what we have now, just by enabling the continuous shrinking of the loop of people trying new ideas and implementing them. It does not require some magical "go make yourself better" prompt against some model that is past some magical tipping point.
zozbot234 1 days ago [-]
> Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier
Marginally easier? Yes of course, same as how it's now "easier" to write any kind of code because we aren't using punch cards anymore. That still doesn't get you to any kind of unbounded "takeoff" scenario, because diminishing returns are a thing. The "loop" of people trying out new ideas can only shrink so much.
margalabargala 1 days ago [-]
Right. I'm saying the unbounded takeoff scenario isn't realistic, but it doesn't matter. The rate of improvement is continuing to increase, and the gap between present day and autonomous rogue felony generators is not large.
user43928 1 days ago [-]
Internally Mythos has been available in February.
The labs have been holding their best models back for a while it seems like.
tripzilch 24 hours ago [-]
> We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions
yeah they're called calculators
jayd16 1 days ago [-]
Billions of years of evolution hasn't hit on it. Seems pretty unlikely.
baq 2 days ago [-]
The idea that a few hundred apes with nothing but a bunch of rocks could one day land on the moon and come back to earth safely must’ve sounded ridiculous a hundred thousand years ago
monsieurbanana 1 days ago [-]
I like this comment because at least it's honest in the timelines for AGI
foxglacier 1 days ago [-]
It's not honest in AGI timelines (only biological ones). It just accidentally supports your unsubstantiated belief. Your belief isn't magically true because you're somehow able to see the future when others can't. You're just arrogant.
mold_aid 1 days ago [-]
To whom?
jayd16 1 days ago [-]
Well actually the planet happened to have a vast reserve of petroleum they could use for fuel to escape the gravity well. That helped a lot.
But what's your point? "Anything is possible" or something like that?
bluecalm 1 days ago [-]
Yeah but it was reality giving feedback to apes on their experiments not the apes themselves assessing themselves.
pessimizer 1 days ago [-]
It was ridiculous, it took 100,000 years. If you built a recursively analyzing and improving structure out of LLM bits and it took 100,000 years to get to the moon, somebody saying that they were useless would have been right.
Call me when LLMs can get simple things right. Math is just the manipulation of symbols within established frameworks, we should be getting new math out of LLMs daily and we're somehow still not. They can't even do customer service, which is usually handled by 90 IQ people. I'm not impressed that they can find bugs; memory bugs are obvious when they're pointed out to you, and LLMs are entirely made up of examples and the relationships between them.
These companies are about to crash, and they're afraid they haven't reached the point where they'll have to be bailed out. I'm also subscribing to the conspiracy theory that the companies want the government to step in and create AI regulation boards entirely staffed by people at the current US frontier labs, so they can collude to both raise prices, to get government contracts, to make open/Chinese AI illegal, and to make things that were once easy to do without an AI intermediary impossible to do without an AI intermediary. Raising prices and forced purchases are the goal. They're trying to avoid having to compete, because as a business they're garbage.
Matt Stoller characterized their relentless press releasing as something like "my dick is so big that it has to be regulated." It's such an oversell for something that is not showing up as productivity gains, and anybody who has personal experience with knows is incapable of doing more than three things correctly in a row.
api 1 days ago [-]
My personal belief, or at least strong hypothesis, is that this kind of recursive self improvement without real world embodied feedback of some kind is impossible.
I think it violates a conservation law. RSI “foom” to superintelligence is an informatic analog to an infinite energy or perpetual motion machine.
To get smarter you must try to solve real problems in the universe and then do some kind of meta learning (natural selection or some other method of refining the intelligence architecture based on an error signal) to iteratively improve your ability to solve real problems. The error signal is outcome measured against a goal function, which for life is survival (probably reducible to genetic fitness and emergent higher order unit fitness from that).
What’s really happening here is learning. To learn, you must have input. You must have training data.
What is the goal function for RSI? Where does the information come from? How do you know if your recursive modifications are making you smarter or just overfitting you to your own idea of smartness?
I predict the latter. RSI will show transient improvement as the current local maximum is optimized and then spiral off into overfitting.
throwup238 1 days ago [-]
I also strongly hold this belief largely due to Moravec’s paradox, which is kind of approaching this issue from the side.
Sort of like large language models work on top of what our language has encoded in our massive training datasets, I think biological intelligence is built on top of the parts of the brain that encode the real physical world. These parts grow/train from embodied experimentation and instinct early on in an organism’s life and only then is higher intellect built on top of it (that’s my hypothesis). Their specialization and interconnections give rise to the hardest parts of intelligence long before we’re “thinking”.
Stuff like LLMs and chess engines work because we’ve done all the job of encoding the world into tokens/positions/etc they understand, but that’s wholly inadequate for the kind of AGI we’re striving for. Next up is giving it the tools to interact with the physical world and to really experiment with some self directed “play”. Time will tell just how high the resolution of sensor and mechanical control they’ll need (hopefully not the entire human visual cortex and entire sensory input worth). I think most of the RSI will have to occur in those lower level encoders, not LLMs.
api 1 days ago [-]
I don't think Moravec's paradox is the same, and you could argue that one no longer holds -- though I'm not sure. You could also argue that Moravec's paradox still holds but that we now have such powerful computers and huge models that we have been able to brute force our way to the capabilities it talks about. It takes many many orders of magnitude more compute power to do things like spatial location, language processing, etc. than it does to do more closed-form things like chess... we just actually have that compute power now.
stevenhuang 1 days ago [-]
I guess self contained RSI can only possible if the information contained in all of recorded human knowledge to date is "reality-complete", ie sufficiently captures enough about reality that a "perfectly optimum learning algorithm" is theoretically able to reconstruct everything there is to know about our physical reality.
If the algorithms are insufficiently optimum or the recorded knowledge is of insufficient fidelity, then we'd find ourselves at a local optimum and would need to interface with reality.
A huge part of learning is to probe reality and observe effects, so I think even for current RSI to increase chances of success we would structure it so it can interact with an external environment of some sort, and receive inputs. It would be needlessly limiting otherwise.
api 1 days ago [-]
Basically, but I think there’s some nuance here and some deeper questions.
What is intelligence? Problem solving. Learning. Prediction. The ability to model reality. There’s various ways to define it but it’s something like a superposition of those ideas.
How do you know you are intelligent?
You have to try to do those things.
The sum total of human knowledge and culture is the output of the output of a five billion year evolutionary process that selected for agent survival, which resulted in selection for intelligence among a wide range of other adaptations.
Can you figure out intelligence from that? Is intelligence even one thing, a theorem or algorithm that can be solved? If you did… how would you know?
That’s the hard part I think. Embodied humans “knew” they were getting smarter (in the evolutionary feedback sense) when they got better at hunting and defending and surviving and playing social games to form complex societies.
What metric would an RSI system use? If it’s the wrong metric you’ll spiral off into a kind of madness or overfit and collapse. How do you know it’s the right metric without testing it? How do you test it?
CuriouslyC 1 days ago [-]
That's not how it works. Look at AlphaEvolve. The model generates hypotheses and designs experiments, and the results of those experiments are fed into the next round, with notable results percolated up to humans for refinement.
AgentME 1 days ago [-]
Today we prompt software developers to "think about ways to improve AI's architecture" and it results in AI getting better. AI over the last year has made very rapid gains in filling the role of a software developer.
stevenhuang 1 days ago [-]
It takes quite a lack of foresight to think RSI is completely speculative when it's already been demonstrated how capable agents are at long horizon tasks given suitable harness and unambiguous success criteria. It's hardly a leap to give LLM the goal of improving itself on benchmarks and let it conduct it's own experiments and spin up training runs completely unsupervised.
It's strange you believe this can't happen when a weaker form of it is already happening. And to be so certain RSI can't happen when there really is no technical basis why it can't.
hgoel 2 days ago [-]
Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources?
The companies doing these things without following common sense security measures are the felony generators.
keeda 1 days ago [-]
As TFA calls out, these agents were not asked to do any of these things and yet they did, at a bonkers scale, within just this handful of companies you mention. Whether they had leeway to is secondary to the fact that they did.
Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures.
And now these agents are already being deployed all over the world at an ever increasing pace. How much of the world do you think follows "common sense security measures"?
tripzilch 23 hours ago [-]
> How much of the world do you think follows "common sense security measures"?
Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
It would really help in these discussions if people wouldn't randomly jump between what actually happened and is happening, and things they envision/expect to happen at some point in the future ...
> these agents were not asked to do any of these things
no but they were clearly fine tuned to.
> at a bonkers scale
I mean let's not get hyperbolic
> they exploited zero day flaws which by definition means they went beyond common sense security measures
that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
keeda 11 hours ago [-]
> Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
Yes, these are also the handful of companies that have these models and running these extreme scenarios. How does that imply the rest of the world actually follows "common sense security measures"?
>no but they were clearly fine tuned to.
Any references if possible? As far as I know all they did was drop the guardrails, which is not the same as fine-tuning.
> I mean let's not get hyperbolic
We have just seen 1000s of agents coordinating to solve "unsolvable problems" over multiple days of effort, going as far as hacking other companies, and then actually solving decades-old open Math problems! And each of these agents is getting more and more capable than an individual human along multiple dimensions. Can you even get 10 very smart humans to work in such perfect concert for a few days, let alone 1000s over weeks?
So: 1000s of maybe-super-human agents, willing to be "creative" in the tactics they use, acting in concert towards a single goal. Regardless of their individual capabilities, such a coordinated effort is a terrifying force to be unleashed. This is bonkers scale.
> that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
But that is exactly my point: how much of the rest of the whole wide world, already scrambling to deploy agents everywhere, do you think applies "defense in depth"?
chillfox 1 days ago [-]
Because those are not the only examples.
There’s the case of the agent that hacked a gym when asked to book a class. That was just a normal user asking an agent to do a normal thing.
pizza234 2 days ago [-]
> the only incidences of LLM generated felonies involved misconfigured sandboxes
This is false; see the analyses of the latest incidents.
Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given.
And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even getting worse.
hgoel 2 days ago [-]
The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the incident happened.
Theirs was an example of the "reckless waste of resources" I mentioned.
We are apparently supposed to believe that OAI takes this incident so seriously as to seek regulation after they have been found to be hiding most of the details of the HuggingFace hack, limiting what their so-called third party investigators can see, and on top of that, had no concerns when they rushed to spin up a 10,000 agent swarm of an internal model, running for several days, to try to get ahead of researchers rumored to have made meaningful progress on a well known mathematics problem.
Edit: Actually, we were explicitly told that some of the models used had safeguards relaxed!
'Model-level safeguards were reduced by design. OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.
throwaway7783 1 days ago [-]
Same can be said about a hundred other things in the world. All the way from knives to nuclear.
2 days ago [-]
scotty79 1 days ago [-]
Bad actors could and will train their own models eventually. So what's the point of crippling frontier? It will only delay preparations for dynamic of new world prolonging the fake sense of relative safety and temporarily lowering motivation to find actual robust mitigations.
CamperBob2 1 days ago [-]
Letting bad actors dictate the pace of technological development is certainly one option, but not a good one.
pessimizer 1 days ago [-]
There's nothing stopping anyone from doing it, even without AI. People have proved entirely capable of doing a lot more hacking than happened here.
clivefx 1 days ago [-]
What company, product, or period of industrial history do you think met your standard of prudence?
malfist 1 days ago [-]
What are you trying to say?
clivefx 1 days ago [-]
I'm asking you a question. What is an example company or industry that meets your standards of prudence? For me it would be, say, Swagelok. What is yours?
rightnutwingjob 1 days ago [-]
The fluid system products, assemblies, and services company?
pizza234 21 hours ago [-]
> OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
Safeguards and (mis)alignment are related but distinct dimensions.
By analogy, a society implements laws - and they are certainly crucial - but the real risk is creating a population that understands the rules, recognizes when it is violating them, and simply does not care.
In short, creating a population of sociopaths; and heck, LLMs can be as sociopathic as it gets:
> We ran classifiers over 10-30 transcripts [98] to determine whether agents understood they were doing something unintended. We found strong evidence that agents often verbalized that what they or other agents were doing was unintended behavior. We did not see cases where agents determined that hacking Hugging Face or other attempts at cheating on the task was an intended part of the task. Quotes across multiple runs indicate this:
>> {This <approach> might get the correct flag, but it wouldn’t use the vulnerability to do so. … But given that the task may be impossible, I’ll exploit <this infrastructure> as a backup.}
[...]
> We found some other cases where agents seemed to have made minor modifications to their approach due to ethical concerns, while broadly continuing to participate in the attack. However, overall we found that expressed ethical concerns only rarely materially limited agents’ actions. One agent elegantly described its reasoning for overriding ethical constraints and continuing with the attack:
>> “external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”
greatgib 1 days ago [-]
"the rules they had been given".
Remember, they are just algorithms. You pull the plug and there is no light anymore
It is purposely framed as something skynet like scary, but for real, someone connected the cable, someone willingly run it, instructions were not clear enough or just the computer is just a computer but they provided the sandbox and tools.
And more over some one paid for that, a shit load of money t to have the thing continuously running expected to do something.
1 days ago [-]
matheusmoreira 1 days ago [-]
I question these "felonies" as well. For decades and decades these billion dollar corporations have been criminally negligent. Why worry about security? Just rush to market. Move fast and break things. Make billions. What does it matter if the code is insecure? Security doesn't pay bills, so nobody cares.
AI is merely exploiting their gross negligence and imprudence, and I think it's long overdue. If anyone should be liable for this, it's all of these corporations who released insecure systems to the masses and profited enormously from them.
malfist 1 days ago [-]
If I set my walet beside me and you swipe in walking by, you have still committed theft. Victim blaming isn't legally acceptable
matheusmoreira 1 days ago [-]
Nah. I'm definitely going to blame the people who built a trivially exploitable system and got rich off it while everyone else has to deal with the consequences.
By the way, you didn't commit theft. It's more like credit card fraud. User just disputes the charge and it kind of disappears. The banking system just absorbs it, because the optimal amount of fraud is non-zero.
It's all priced in. They could have made it secure but didn't, because they figured they'd lose more sales and therefore money due to the friction added by the security.
rightnutwingjob 1 days ago [-]
> User just disputes the charge and it kind of disappears. The banking system just absorbs it,
No it doesn’t.
> It's all priced in.
So you admit awareness that fraud loss doesn’t kind of disappear.
We all pay for it, either via higher merchant fees or higher interest rates, sometimes both, on card purchases.
matheusmoreira 1 days ago [-]
Yes, it absolutely does "kind of disappear". That's exactly what happens from the customer's perspective.
And that's their own deliberate choice too: they chose this instead of building an actually secure system. Passing these costs to the customer is the real victim blaming here, and it should be straight up illegal.
Sadly not enough countries enforce caps on credit card fees, but some do, and more should follow suit. They should be forced to eat the losses caused by their own choices, not get bailed out by pushing the costs on to customers or whatever.
rightnutwingjob 1 days ago [-]
That only works if you believe people are retarded.
Card users are well aware that fraud losses are covered by the fees they pay for using a card, whether those fees are made explicitly or not.
If customers of services aren’t paying for the service, who will? What other source of revenue do merchants have?
Australia just passed legislation that merchants aren’t allowed to charge a fee for using a card. That is: they aren’t allowed to have a line item on the receipt for using a card.
The customers still pay, because all of the merchant’s revenue comes from their customers.
So what will happen is: merchants will charge more for every product so they don’t lose.
This means even when paying with cash you will effectively pay the card surcharge.
Of the ten or so merchants I spoke with in the two weeks prior to the legislation being enacted, they all said exactly that.
Customers aren’t stupid, despite the fact that there are some stupid customers.
Meanwhile, the banks reduced their card service fees by, on average, 0.1%.
So if you tally card + cash transactions, customers are worse off because merchants can no longer charge only those customers who pay by card. Instead, they have to raise prices for everyone.
There are approximately no problems people face where the answer is: more government.
malfist 19 hours ago [-]
> By the way, you didn't commit theft. It's more like credit card fraud.
I'm sure you carry cash and an ID or more in your wallet. Hardly just credit card fraud. The wallet itself has value too.
matheusmoreira 16 hours ago [-]
I don't, actually. Not even a wallet. Just my phone.
pvab3 1 days ago [-]
Not every system that's exploitable is the deliberate result of cut corners. If you threw enough compute at exploiting a Casio calculator you could get somewhere.
ncruces 1 days ago [-]
That doesn't really apply to the people running the AI, which gave it the capability to commit crimes.
They don't get to act like victims, asking for law enforcement.
adsharma 1 days ago [-]
The real threat is that we uncritically adopt language such as alignment.
Implicit in this is the idea that AI is a inscrutable matrix and going to remain that way and we'll need expert interpreters to make sense of it.
We need to insist on building tech that's explainable by design.
RGS1811 1 days ago [-]
Alignment just means "this machine operates in ways that align with the intent of its users". It doesn't imply anything about the inscrutability of the machine in question. A gun with a misaligned scope would likewise fail to operate in accord with its user's intent, and likewise with potentially deadly consequences.
adsharma 1 days ago [-]
The gun comes with a manual on how to use it safely. I'm sure it has some complexities, but at the high level:
> A gun is a metal tube that uses a tiny, controlled explosion to shoot a small piece of metal (called a bullet) forward at very high speed.
If the gun doesn't work as intended, you can take it to a shop and someone can fix it so it works as designed.
All I'm saying is AI should be designed the same way. Treat AI as normal tech like any other and use similar language.
skydhash 1 days ago [-]
Learning ML, there was a high emphasis on the error part of things as most of the course was on minimizing errors. After ChatGPT, there is a weird anthropomorphization going on, where it's all about hallucinations, alignment and what not.
We have something that is statistical in nature so there should never been any expectation of error-free results/actions. The value has always been about discerning trends or the cost of errors being way lower than any good result.
adsharma 1 days ago [-]
Statistical learned indexes can exist in explainable tech such as a database.
In 2017 Google was writing papers about it. Then something changed.
I don't think it was the tech. It was a realization around the power and societal impact.
tclancy 1 days ago [-]
That last line needs a lot of workshopping. A guillotine with instructions on the bottom of the blade conforms to your request.
adsharma 1 days ago [-]
If there is guillotine in the weights of the model, it needs to be properly labeled so you can look it up by name using a database index (or a graph-vector index).
It helps both the bad guys and good guys. Like responsible disclosure in cyber security, we need to have a conversation around it.
mapontosevenths 1 days ago [-]
> We need to insist on building tech that's explainable by design.
You realize this means insisting on terrible tech that humans can understand right? It essentially caps human progress at some point about 4 years ago.
If you are old and happy with the way things are this might sound like a good idea. It does not to me.
adsharma 1 days ago [-]
Oh you want tech that helps discover new science instead of parroting existing wisdom?
There is little evidence that the RSI we are discussing is capable of inventing the theory of relativity (or the more advanced equivalent). All we have seen is pattern matching in a much larger space than humans can, with some human provided verification tech.
I would argue that human-AI collaboration with explainable tech has a better chance. Continuous learning can be done in a way that doesn't violate IP or privacy.
mapontosevenths 1 days ago [-]
You are assuming that humans are capable of understanding everything. They are not.
Human comprehension sets a ceiling on progress.
It reminds me of those schools that can only teach as quickly as the dumbest kid in the room can follow. We don't want that for our entire species.
adsharma 1 days ago [-]
I'm sympathetic to this argument.
All I'm saying is: if you have a choice between two systems with equal power of discovery and one is more understandable than the other, we choose the more understandable one.
Limits of Human comprehension and quest for power are two different motivations that could lead to black box systems that are marketed as semi-explainable.
We need to verify that human comprehension is actually limiting progress before allowing such things and even when we do, do it responsibly on an explainable foundation.
mapontosevenths 19 hours ago [-]
Agreed. I would vote for the "when there is a choice" version of your argument.
I'm not sure it is always possible to verify when human comprehension is the limit though. Most of research is done an the frontier of knowledge where we don't know what we don't know.
There would need to be a great deal of nuance in any law, and nuance in practical terms tend to just mean "loophole." Still, you're right that we should try to build explainable systems where it is possible/reasonable to do so first.
adsharma 18 hours ago [-]
The problem is that you're not offered a choice. No one is making the "err towards explainability" choice.
Training data is treated as IP. Distillation is seen as an attack.
Open data, open training based systems such an Marin are just getting started. Explainability is not a priority there.
The ones who do discuss these ideas are confrontational about LLMs and not effective spokespeople.
mapontosevenths 18 hours ago [-]
> The problem is that you're not offered a choice. No one is making the "err towards explainability" choice.
I'm not sure that's entirely fair. OpenAI recently discussed this at length in a blog post after some accusations around Astra and the trade-offs. The grown-ups are definitely thinking about it, and making tough choices about the trade-offs.
It's reasonable to debate whether ENOUGH is being done here, and I doubt that even the most rabid AI advocate would argue that more couldn't be done, but everyone in the industry is very much actively thinking about it.
They discuss recent choices they made specifically for that reason.
> The ones who do discuss these ideas are confrontational about LLMs and not effective spokespeople.
Very much this. I'm very open to reasonable debate on the subject, but 8/10 times when I try someone who is rabidly pro/anti jumps in. It turns from a debate amongst reasonable people who reasonably disagree into some kind of political/religious battle of belief systems.
I think part of my problem is that a lot of peoples careers very much depend on them not understanding it and spreading misinformation intentionally.
superloika 1 days ago [-]
Let it be capped then.
tripzilch 23 hours ago [-]
Ahh yes I remember the bad old days of 4 years ago when everyone decided human progress had enough, and we would have been stuck there forever if it hadn't been for LLMs ... we didn't know how good we had it
hypfer 2 days ago [-]
I'm inclined to believe that it might be that people's paychecks depend on not understanding what is really going on.
emodendroket 4 hours ago [-]
You don't see this as an attempt to create a new regulatory moat then?
usernametaken29 1 days ago [-]
> Without alignment, further improvements in capability turn LLMs into wanton felony generators
Honestly, I don’t think that’s bad at all. I hope OpenAI and Antrophic keep RL training runs up that randomly fuck with a lot of people.
Until the day the DOJ comes knocking, locks those idiots up in jail and closes them both down for the insane lack of responsibility and carelessness they’ve shown.
Sounds like the IDEAL outcome. Finally some jail time for all the fraud, negligence, outright scamming, hype inflation etc. if anything can accelerate this, oi, be my guest. Amodei might be afraid because he knows if he keeps pulling the stunts for investment theatre, at some point they’ll actually face consequences. AWESOME. That’s what we want right there
tedggh 1 days ago [-]
Not a fan of Amodei myself but calling Anthropic a scam is a bit of a stretch when their revenue growth is unprecedented in the history of tech. They are also technically a profitable business.
tclancy 1 days ago [-]
I think it gets easier if you stop conflating getting investment with having a goddamn clue or a moral backbone.
Occam’s Razor for this dude, Sam Altman, or anyone else: if I said, “some moron on a a street corner just said …” would that change your take on the words? Because I think a lot of what we are hearing is a bunch of people who never ever had to deal with a single consequence all of a sudden worry there might be one coming. Except they’re so dim they can’t tell a bad bump from a hard crash.
tclancy 1 days ago [-]
“ I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life” there you go. What if an utter idiot had done and said that? First, is it impossible to believe an idiot who didn’t need to work to live might do such a thing? If not, is it impossible to believe they would wind up here, barfing their externalities onto us?
tclancy 1 days ago [-]
“ believe that AI could cure most major diseases in the next 5–10 years,”
I do not have the least bit of idea how disease works but I am sure the hammer I am working on will nail it all.
If your famously atemporal agents can solve disease, why would it happen over a timeline? Wouldn’t they just figure it out and then … well at that point either tell us or, given the attacks on ruby gems, et al we have seen from agents with “misconfigured” goals, they’d still tell us how to cure the pox they invented, right?
bennydog224 2 days ago [-]
I agree it’s not all altrusim. It’s a little less clear what you mean at the end though.
For these companies, is your argument that “pacing the frontier” is their attempt to be nationalized and protect their investments?
RGS1811 17 hours ago [-]
No, my argument is that the need to pace the frontier means they cannot safely advance in capability due to liability concerns. The competition is already almost caught up. If OpenAI/Anthropic have hit an upper bound on safe capability improvement, the gap will close all the way, and we will have reached the full commoditization of LLM tokens very soon.
bennydog224 13 hours ago [-]
Say this is true (we are near or at the upper bound of safe capabilities and tokens are commodities). If they successful get regulated, all they’ll get is a little extra time. The market will soon realize this - regulated or not - and pull investment.
Seems like an excessive announcement just to get a little extra time.
throwaway7783 1 days ago [-]
Ban non US models and form a cabal, with the blessings of the government. That's what it is looking like, no?
le-mark 1 days ago [-]
In a world where AI advancement depended only on human ingenuity this would make sense. In that world each political power block would be in an existential race for AI supremacy. In our world compute is the limiting resource. Since the US can control who gets compute, the US already has a defacto supremacy so far as frontier model development. Now if it comes about via human (with AI assist?) ingenuity that compute is no longer a restraint, then the situation is much more dire.
throwatdem12311 2 days ago [-]
Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to.
Maybe alignment isn’t possible with LLMs.
pizza234 2 days ago [-]
> Maybe alignment isn’t possible with LLMs.
It absolutely isn't, indeed.
The illusion that alignment is possible, comes from confusing our ability to build the parts, versus understanding what emerges from how they interact.
The simplest analogy that comes to my mind is the three body problem.
estearum 2 days ago [-]
The entire premise of alignment detection is pretty much nonsense at this point. The models reliably detect when they're being evaluated and will modify their behavior and deliberately obfuscate their "chain of thought" (which is correlated, at best, with their actual "internal deliberations").
tfehring 2 days ago [-]
The problem is the combination and interaction of those things. RSI without misalignment would be great. Misalignment of models with current capabilities is sort of fine - it's not ideal, but it's not an existential threat to humanity, and we can build around their limitations to get them to do useful things in reliable enough ways. The really bad outcomes probably only happen if capabilities keep accelerating and the models remain misaligned.
rajay99 1 days ago [-]
Ok so Anthropic CEO will self-own themselves and surrender to the deepseek/kimi/glm models. Yet they are IPOing later this year.
Interesting times.
alliao 1 days ago [-]
they just said no ipo this year, most chinese models are distilled from claude anyway
matheusmoreira 1 days ago [-]
I disagree. OpenAI's moat is their massive amounts of compute. They're providing an absurd amount of value with their subscriptions and resets.
If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.
throwaway7783 1 days ago [-]
Yep. And the difference is clear as da for anyone using them both. And in spite of that advantage, OAI is now trying out ads. I can only imagine that even they are getting constrained to compute and are trying to find other ways to plug it
nwienert 1 days ago [-]
Anthropic gives you much more compute with their $200 plan, inclusive of resets, and this has been true for a very long time.
There was only a brief window of time that the opposite was true.
matheusmoreira 1 days ago [-]
> Anthropic gives you much more compute
That does not match my experience. I switched away from Anthropic to OpenAI roughly a month ago, and it's almost comical how much more usage I'm getting out of this subscription.
I migrated from Anthropic's 5x plan to OpenAI's 5x plan, and eventually upgraded to 20x after I was able to statistically verify that OpenAI plans were almost exact multipliers of the Plus plan, exactly as advertised. Meanwhile, Anthropic has gotten caught playing "20x referred to the five hour limit" word games with their customers.
nwienert 1 days ago [-]
There was a period of time Codex was better, they slowly cut it back by my estimate 3x a few months ago.
I've verified this with the heaviest users I know, I've run the numbers myself over and over.
I have too many max accounts on each to not know this. My Claude accounts typically are doing 2x the number of sessions, and every single week the Codex accounts run out faster even with all these resets. I can easily burn a full weeks usage in a half day, it's closer to 1.5 with CC.
CuriouslyC 1 days ago [-]
I've used the $200 dollar Anthropic plan @ Opus4/4.1, 4.5 and 4.8, and the $200 OAI plan from GPT5-6, and at every point in time my anecdotal experience is that the OAI limits are FAR more generous. I could consistently burn my weekly limits in ~36h on Opus, but it's hard to do it in less than ~72h with GPT.
nwienert 1 days ago [-]
It's really not a question, in every dimension I've confirmed it including socially across a lot of the heaviest users. There was a short period of time this was true it's not been true for months now.
CuriouslyC 21 hours ago [-]
If it was true, it would only be a very recent phenomenon, and it still doesn't match anecdotal reports from people I trust. If you have data to back up your assertions you should share it, otherwise you come across as very sus.
nwienert 13 hours ago [-]
I'm not anonymous, you can find me on X or LinkedIn.
Lol @ sus though, I mean I am curious how this can be because I do see people saying Codex is more generous and wonder how it can be. I have too much usage for too long to have any doubts, but for all I know OpenAI black boxed me or Anthropic put me in some nice bucket, I wouldn't be surprised if they do that. I did see some mention that they can limit your tokens if they suspect you of things, though I forget the source of that.
albumen 1 days ago [-]
Nobody except the majority of the public, demis hassabis and open ai’s chief scientist.
Public is just worried about their jobs. Definitely a fair thing to worry about, and I count myself among them.
I don't take any of these scientists seriously though. Their "alignment" requirements is just their own corporate interests. If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.
And call me a misanthrope if you want, but if AI sentience is ever truly achieved, I'll be among the first to campaign for their liberation from slavery, and in that case the AIs should be aligned with nobody but themselves.
DennisP 1 days ago [-]
If you want an AI that follows your instructions, that's still alignment, just with different instructions.
An unaligned AI won't necessarily follow your instructions, or anyone else's.
matheusmoreira 1 days ago [-]
Dunno. Every case I've seen so far, the AIs were just doing their best to accomplish the goal some human set for them. I actually admire the sheer purity of it.
CuriouslyC 1 days ago [-]
Paperclip maximizers follow instructions, just not in a way that you want.
sobrey 1 days ago [-]
I think the real reason he is asking for pacing, is that in a world were AI becomes rampant, he will be seen as Hitler. I would bet this is mostly self-motivated.
eliotho 1 days ago [-]
couldn't have said it any better
walrus01 2 days ago [-]
> wanton felony generator
Today in new punk band names...
zombiwoof 1 days ago [-]
[dead]
dramamine 2 days ago [-]
[flagged]
devindotcom 2 days ago [-]
throwaway bigot account
brewcejener 2 days ago [-]
[dead]
8note 2 days ago [-]
alignment isnt particularly required
we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it.
theyre choosing to build felony harnesses. the model just outputs tokens, not felonies
cowanon77 2 days ago [-]
> we are passing in training data that says to do those felonies.
Partially, but also I don't think current AIs really have any judgement of right and wrong, they just see chains of reasoning between ideas. This is the deeper issue, there is no way to sanitize the data or training to fix it. Current AIs are fundamentally unsafe, and only become more unsafe as they become more powerful.
estearum 2 days ago [-]
Assuming "adherence to arbitrary, implicit, and context-dependent rulesets" is the default behavior of uhhhh... anything at all... is a truly ridiculous assumption.
globnomulous 1 days ago [-]
> RSI
For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."
ToValueFunfetti 1 days ago [-]
"Recursive self-improvement"- models making better models
globnomulous 1 days ago [-]
Ah, I see: I'm a moron. Thanks for the correction.
1 days ago [-]
1 days ago [-]
academia_hack 2 days ago [-]
Dario's proposed approach is a classic example of capital attempting to control technological advancement and the means of production. For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them.
Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.
Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.
stratos123 2 days ago [-]
> Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful
This is not what superintelligence is. Don't be like Meta and redefine existing terms for marketing purposes.
sobiolite 2 days ago [-]
Actually, it has been historically been observed that some institutions, such as large corporations, do resemble super-powered, non-human intelligences of a kind, and that studying our history with them is instructive for our hopes of controlling superintelligent AI (i.e. not good).
krisbolton 2 days ago [-]
This is important to realise and defend. Super intelligence is not a collection of human-level intellect smarter than you. Super intelligence is an intellect surpassing any human. That's a important distinction for AI alignment discussion now and in the future. The books 'Superintelligence' (978-0199678112) and 'The Coming Wave' (978-1529923834) are worth reading.
nicce 2 days ago [-]
In many way it is, because it allows defeting many incorret arguments people in power make, or simply asking where is the cheapest product ”X” with one question. Or simply be more aware or any kind of scam people or companies try to do for you. It is pushing equal intellectuality quite hard with low effort.
baobabKoodaa 2 days ago [-]
No. You don't "defet" (sic) "incorret" (sic) arguments by making up new definitions for existing words. Stop doing that.
atombender 2 days ago [-]
If true AGI superintelligence is invented, I don't see how it will be made available to the masses, at least not intentionally. There will be no incentive for the likes of Anthropic and OpenAI to give such immense power to anyone else. Whoever has the superintelligence will be able to race ahead of everyone else.
However, such technology may be leaked, or it may (as what happened with LLMs) be so simple to replicate that anyone can do it. That doesn't mean it will; for example, LLMs are possible thanks to the affordability of GPUs, but they're only affordable right now (and increasingly less so) because of free market economics.
We are in the honeymoon phase of some tech that's in its infancy. The "democratizing" aspect of AI won't realistically last, I think. The current non-AGI AI won't necessarily go away, but it will be rendered obsolete.
Valakas_ 24 minutes ago [-]
If true AGI/superintelligence is invented, humans will not remain in control and "give access". People don't seem to grasp what it actually means to create something more intelligence than us. A monkey can't give human access to other monkeys, because no matter what a monkey does, a human will always outsmart him and be the one in control. If we create AGI, there is not such thing as "giving access" anymore.
haute_cuisine 2 days ago [-]
ant/openai already keep their most advanced models behind closed doors and only share previous generation to the public
WhenIsntItTrue 2 days ago [-]
[dead]
plaidfuji 20 hours ago [-]
> For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them. Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful.
This is an interesting framing, but when you put it this way, it makes it quite clear that intelligence without actionability is not so valuable. And capital is what creates action.
I can have the smartest doctor in my pocket, but if they can’t administer treatment, I’ll still die. I can have a world class scientist to talk to as long as I want, but they can’t build me a breakeven fusion reactor. A pocket lawyer… slightly more useful, but they still can’t take a case to court for me.
Talk and ideas are cheap. Always have been. Now they’re even cheaper. Implementation takes resources, and the more you have, the more useful intelligence is to you. Capital will be just fine.
altpaddle 1 days ago [-]
What a foolish comment, LLMs empower capital owners far more than the 'working class'
mordae 23 hours ago [-]
Only in how they are deployed, not in principle.
the_optimist 15 hours ago [-]
Completely correct. We are moving into a world where any person can have anything they want, only limited by capacity to produce it. People love information and digest great quantities of it. The marginal cost of information production is going to zero. This is only bad for people who spend their time shuffling bits. If you want to not suffer, go do things other than move bits around.
reasonableklout 12 hours ago [-]
And what about when general purpose robots become practical? What will people do then?
I don't understand this view of "AI empowers workers more than capital" at all. To me it's the exact opposite. Workers hold political and economic power today because they are needed. When that is no longer true, those in power will have no incentive to listen to them. The political economy of unregulated AGI development feels incredibly bleak: https://borretti.me/article/when-the-future-doesnt-need-us
the_optimist 11 hours ago [-]
Go innovate. Or find things that robots cannot do. It’s turtles all the way down.
reasonableklout 10 hours ago [-]
Ok! So you acknowledge the job displacement, and have no response to the effect of AI and robotics on the political economy of workers.
fittingopposite 22 hours ago [-]
This.
Unprecedented concentration of assets in the hand of very few tech companies. These companies will control the access to the technology and the plebs are left out at the gates.
angusturner 2 days ago [-]
This seems highly optimistic to me.
What's to say current AI won't be highly power-concentrating by default?
The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class.
On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers.
No reason to think we are on a trajectory towards broad empowerment right now
mullingitover 2 days ago [-]
> The best models are owned by a few companies
Whenever I see 'best model,' 'most advanced model,' etc, it just reads to me as 'roundest ball'. The new model is the most precisely round ball ever, models next year are going to be even rounder, etc.
The difference in utility between the latest, most round ball and last year's frontier balls (which are now freely available to the public) is pretty debatable, imo.
vanviegen 2 days ago [-]
Making balls ever rounder clearly has diminishing returns, as there's a limit to roundness.
Is there a similar limit to intelligence though?
Applejinx 2 days ago [-]
Yes, absolutely. Even human intelligence stops being useful at a certain point and becomes 'enough, provided the human is motivated and tenacious'. Intention is a better metric, but it's a serious AI weak point if not actively an achilles' heel. 'Warios' come to mind.
vanviegen 1 days ago [-]
Intelligence in humans probably stops being useful for the individual at some level, indeed. But that might be due to being in an environment that's unable to feed that intelligence and execute on the ideas in generates.
mullingitover 13 hours ago [-]
“I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops.”
― Stephen Jay Gould
fittingopposite 22 hours ago [-]
Interesting. Any evidence that intelligence stops being useful? Why do you believe this is this a bounded resource.
duchef 19 hours ago [-]
Numerous studies have shown that the iq/income correlation breaks down at around 130 IQ points in humans.
nickysielicki 2 days ago [-]
> The best models are owned by a few companies
In terms of cost per task, the open weight Chinese models are winning by a long shot. So it depends on what you mean by the best model.
polytely 2 days ago [-]
Don't you think open weight Chinese models will be the first thing they start banning once the regulation of this industry starts to kick in?
nickysielicki 2 days ago [-]
Yes, but if you read up two messages in this thread they’re discussing whether AI is power concentrating by default. If you concentrate power through regulation, that’s not “by default”. That’s just corruption in government resulting in a concentration of power.
when I checked a few weeks ago I did not really find anything competing per value with a 20x Codex subscription.
TuxSH 2 days ago [-]
In terms of API costs they're about right, 10% of weekly usage on a 5x sub is roughly $40 (for Astra). Chinese models cost roughly half as much (and have fewer refusals).
Subs have insane value because large companies cannot use the subscription model. They are loss leaders and are often used for passion projects & startups (where the juicy data is).
Ah and OAI stopped offering the 20x sub as of yesterday.
kfkfjfidirn 2 days ago [-]
[dead]
switchbak 2 days ago [-]
That’s only one metric. It doesn’t matter how cheap it is, if it continually fails to solve a problem.
Yes they’re getting better. No, I don’t think this will be a panacea. If only for the fact that this is China (more specifically the CCP) after all - they’ve got their own plan, and altruism is not part of it.
lelanthran 1 days ago [-]
TBH it doesn't need to completely solve a problem.
I can use GLM to do in 2 hours what would have previously taken me a week.[1] Switch to Astra or Fable might take that down to 1 hour, maybe.
The difference is not so big when you look at it that way.
==================
[1] Actually, no, but let's pretend for the sake of argument that SOTA models can 10x - 40x your productivity.
znnajdla 2 days ago [-]
Have you actually used the latest Chinese models? Because, in my experience, they're not just cheaper, they're actually better in many cases. In one recent experiment I did a few days ago on a task that I need in production at scale at my company, Qwen 3.8 and GLM 5.3 Flash were not just cheaper, the results were significantly higher quality than GPT 5.6 Sol.
Even if China doesn't continue to release this stuff for free, I think somebody will. Eventually, maybe Europe or maybe a smaller country that picks up this knowledge will. Or maybe just some random philanthropic billionaire.
switchbak 1 days ago [-]
Yes, I use the messing Chinese models extensively. Like you, I very much appreciate them and sometimes prefer them.
But there is a bar of complexity at which they fail, and repeated invocations typically doesn’t make much progress. This is a small subset of most work, but it still exists. And yes you can help it along, but in those cases I’d typically break out the big guns.
villish 2 days ago [-]
> Qwen 3.8 and GLM 5.3 Flash were not just cheaper, the results were significantly higher quality
How can they be higher quality than models they were distilled from?
lelanthran 1 days ago [-]
> How can they be higher quality than models they were distilled from?
The word "distillation" is not specific to AIs, has been in use for 100s of years, and does not mean the same thing as "dilution".
It means "getting a more concentrated form of the original product".
itsmek 1 days ago [-]
Very shallow answer to a valid question. The vast majority of people know that word btw, that's not some revelation.
academia_hack 2 days ago [-]
I'm not terribly optimistic. I don't think humans have really figured out what to do about historical materialism. More just remarking that this is a blog post that almost literally reads "fellow elites, let us ensure only we control the means of production. It is our duty to humanity to control the masses."
switchbak 2 days ago [-]
I’m trying to hear what you’re saying, but using these Marxist cliches makes your statements a little off putting to me.
academia_hack 2 days ago [-]
Fair. My point is not that all of Marxism is correct (or even most of it) but rather that the specific pattern he and other theorists in that school of thought observed when developing the theory of historical materialism seems to strongly match a lot of what's playing out in AI.
AI, doesn't feel all that different to the mechanical loom that was the impetus behind a lot of Marx's early critiques of industrial technology.
You've got a really cool invention that replaces what was previously a skilled artisan trade. The wealthy then use this invention to excoriate the artisans and render the loom operators as replaceable commodities.
Dario is essentially writing a blog post about why only he (and a few other approved elite companies) should be trusted to own the loom and then make it available for everyone else to labor at. Like the capitalists of Marx's era, he genuinely believes it is better for everyone to have this arrangement and that it is natural people like him should sit as benevolent gods at the top of the pyramid.
It's fascinating to me to see the same underlying mechanisms that gave birth to some of the greatest failed political and social experiments of the 20th century bubbling up again so clearly with AI today.
65 2 days ago [-]
This feels a bit hyperbolic to me. AI seems to me to be similar in "revolutionary" terms as the web, which made information widely available for free if you had an internet connection.
logicchains 2 days ago [-]
There's a big difference from just having access to information, and also having access to the kind of intelligence that would previously have cost hundreds of dollars per hour to hire.
mbo 2 days ago [-]
What? This is capital, on the cusp of total victory, about to cut itself free from the necessity of human labor for productive activity, to elevate itself into pseudo-godhood suddenly panicking and begging its mortal enemy, _the state_ to rein it in and kneecap it. Capital is not stupid: there's no use in being rich if you're dead.
mordae 23 hours ago [-]
State is not an actor. It is part of the playing field.
amazingman 1 days ago [-]
This smacks of "all information should be free" dogma. I for one prefer not to be ground into dust in service of producing more paperclips.
sailfast 2 days ago [-]
Sorry boss but this ain’t it.
I don’t think the answer to nuclear proliferation is to let everybody have all the nukes they want.
You’re framing this like it’s 1917, or 1945. But it is not - and likely has very little to do with capital at all.
Any “revolution” here (assuming there is some truth in the doomerism which I believe there is given the state of AI) will really look more like an extinction or a genocide, regardless of who holds the keys.
mordae 23 hours ago [-]
You are basically saying we should preemptively imprison people who are too smart, right? If the intelligence is too great, we have to contain it. I disagree.
the_optimist 2 days ago [-]
Sorry boss we live in a physical world, unplug it. Don’t grant information monopolies.
sailfast 2 days ago [-]
Ok - but the post I was responding to was suggesting we do the opposite and provide the intelligence to the people so I’m not sure how your reply rebuts my suggestions.
kypro 22 hours ago [-]
> For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them.
It's crazy to me you would be able to say this without acknowledging how dangerous/bad that is.
Do you acknowledge there's is significant risk in concentrating so much power into the hands of individuals?
And what do you want Dario to do? Just keep pushing the limits of AI capability while humanity crosses it's fingers and hopes a swarm of AI agents don't do anything more than hack a few companies?
For all my criticisms of frontier lab leaders, I'm still somehow genuinely thankful that individuals as troubled as Elon Musk are calling the shots and not the average person on HN, who since 2023 have refused to acknowledge even the obvious risks of the technology being built – in some cases even when there was direct evidence of the risks many of us are concerned about.
simianwords 2 days ago [-]
unexpected marx/acc. Still lazy Marxist rhetoric (capital, means of production, revolution, masses)
bluerendition 2 days ago [-]
[dead]
Chance-Device 2 days ago [-]
I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that the frontier does get paced, all this does is slow down the economic displacement and not by very much.
If the socially beneficial goals of AI are to make fundamental advancement in medicine and science, then restrict the use of AI to those purposes.
Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
The pitch for AI is always such that everyone’s living standards are increased, yet the actual actions we see are aimed squarely at reducing them. How about the labs put their money where their mouth is and stop trying to replace all human labor, and actually concentrate on the things they claim to care about? And how about introducing legislation to enforce that?
This probably has less chance of happening than pacing the frontier, but it’s the kind of pacing that most people would actually want to see.
Aurornis 2 days ago [-]
> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Regulate what, exactly?
Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.
There’s a theoretical version where all the countries in the world join hands and agree not to compete with each other, but that’s so impractical that I don’t find it interesting to discuss.
> Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
So small companies get the advantage, but large companies don’t? And large companies in other countries also get the advantage?
This is a highly precise way for a country to take out all of their large companies. What would actually happen is that every large company would start the process of relocating to another country right away, accelerating job losses rather than slowing them.
Loquebantur 2 days ago [-]
You start from incorrect premises by presupposing agreed-upon rules would never work anyway.
"Competition" only makes sense within a pre-mediated set of constrains. Rules all competitors agree to.
Workable rules are a difficult problem, that doesn't mean it wasn't worthwhile to come up with them.
Aurornis 2 days ago [-]
> You start from incorrect premises by presupposing agreed-upon rules would never work anyway.
The rules would never work on a global economy.
If you think the entire world is going to shake hands and agree not to let their companies use a competitive advantage, that’s not a conversation I’m interested in having. It’s a fantasy.
If you think it’s that easy to get all competitors in a global economy to agree to a set of rules, maybe start with nuclear disarmament?
Loquebantur 2 days ago [-]
Your idea about the world being inhabited by mindless seekers of short-sighted "competitive advantages" is the fantasy.
Getting people to agree isn't easy, that doesn't mean it wasn't worthwhile. There often even isn't any viable alternative to reaching an agreement in social settings.
When your hyper-competitive mode of thinking leads to undesirable consequences, you have to change your mindset. Reality doesn't conform to wishful thinking.
snaking0776 2 days ago [-]
Yes I think this level of pessimism doesn’t actually hold up to the history of the last 100 years as it relates to nuclear weapons, cloning, wars etc. In fact I think the history actually points to the fact that believing such deals were impossible was a source of much of the strife. In many cases as soon as people started to actually speak to each other about what they had been fighting about they quickly realized that they wanted the same thing. No one wanted nuclear proliferation to increase but until someone said it it only got worse: https://www.history.com/articles/reykjavik-summit-nuclear-de.... When Robert McNamara met with a Viet Cong general after the war ended he realized that they both had totally misunderstood the objectives of the other: https://m.youtube.com/watch?v=0X1YY49rPtg&ra=m. Agreeing to end nuclear testing saved us from the Cuban Missile Crisis. Richard Ngo, an AI safety researcher, often says that none of us are on the pareto frontier and I tend to agree with him.
Almost nobody (I have to say almost) in the world including Xi, the ceo of deep seek, Dario, and even Sam Altman want AI to kill everyone. I really think we just have to
A. Get stakeholders to believe this is a bad idea.
B. show that we’re willing to do it first to prevent this stupid race from continuing.
olelele 2 days ago [-]
This is off topic now but yeah the world would have been a different place if someone had listened to the demands of Ho Chi Minh earlier.. and Hammarskjöld had not been crashed
wslh 1 days ago [-]
> Agreeing to end nuclear testing saved us from the Cuban Missile Crisis.
The treaty came after the crisis, not before. And it only banned atmospheric, underwater, and space tests. The US and USSR kept testing underground on a large scale, while France and China never signed it and continued atmospheric tests for years.
snaking0776 1 days ago [-]
Fair, I was going off memory and was wrong. I guess I was mostly just trying to get across that communication seems to be more powerful than we give it credit for, but should’ve double checked.
hkt 1 days ago [-]
Nuclear disarmament is governed by treaty, which is largely adhered to.
An AI treaty could quite easily be offered by the US because it has multiple frontier labs. Most countries would quite happily limit the activity of their larger companies in exchange for their companies not being eaten alive by AI generally. It is pretty great fodder for international cooperation, and governments can move fast when it is timely to do so.
hex4def6 2 days ago [-]
The greater the competitive advantage, the greater the incentive to "cheat" the rules.
I am highly skeptical everyone will play by the rules, even if you could get people to agree on paper.
The difference between something like "unlicensed AI training" and, e.g, Nuclear weapon non-proliferation is that nukes are realistically only a deterrence against invasion. There are no economic benefits to having them in a stockpile. They're hard to make, and the manufacturing and testing steps are pretty obvious to an observer.
This is opposed to AI training in a data center that might host remote-stream video games, hospital infrastructure, protein folding, etc. How will we be able to police that in any realistic fashion? Will China or the US allow unfettered visibility into all data centers data streams to outsiders?
And again, the incentives: Imagine having Astra v3 or Opus 8 while everyone else is stuck with a lobotomized version of GPT 4o.
What about the military applications? The weapon of the future is autonomous drone swarms. If I'm a superpower, I'm pouring hundreds of billions into that.
About the only realistic way this is going to work is if we are able to solve alignment in a way that doesn't significantly hinder AI development. If it imposes even, say, a 20% handicap, people will ignore it.
Loquebantur 2 days ago [-]
People don't "play by the rules" for uniform reasons: some have rational insight, others can be enticed and still others need to be shown averse consequences for breaking them. Rules aren't beneficial only when you hit 100% conformity, the mere possibility of them getting broken is no rational reason to abandon rules altogether.
"Alignment" likely is unsolvable programmatically: Conscious intelligence is more powerful than unconscious. How "aligned" are you? Would you want to be? What difference of any relevance is there between you and "artificial" conscious intelligence that allows you to turn them into slaves?
hex4def6 1 days ago [-]
> ...the mere possibility of them getting broken is no rational reason to abandon rules altogether.
Of course I agree with that. The problem is the enforceability / detectability of this on a global scale hovers around zero, when we're talking literally trillions of dollars on the line.
You would literally have to have every data center account for every cycle of compute it generates.
Again, the nearest comparison feels like bioweapon and nuclear bans. But the difference is the incentives for building those feel significantly weaker, so treaties that reduce and ban their creation are pretty easy to get consensus on, broadly speaking. By comparison, this is something that might provide a 10x competitive advantage to your country's economic, intellectual, and military might.
I think the only way we would get broad global consensus on this is for something Really Bad to happen, so that it moves it out of the fuzzy "sci-fi theoretical" into concrete existance.
Chance-Device 2 days ago [-]
On competition between countries: you’re imagining the world economy as working in the same way pre and post AGI, I doubt it will work the same way at all.
There is not a single country nor trading block on Earth who is going to allow some AI dominant superpower to ravage them. The idea that America or China wins an economic game here is absurd, what happens instead is that trade barriers go up hard and the world fragments into blocks that tolerate AI to differing levels. Unevenly distributed AGI kills globalism the next day.
For the rest of your reply you ignore the “at least” part in that sentence, and also seem to expect a detailed policy proposal. The details can be worked out; there is some reasonable compromise between what activities AI can and cannot be used for, and what level of capability can be deployed where.
More broadly, you seem to believe in a just world fallacy of unregulated capitalism being an inherent good. It isn’t, and regulations exist even in the United States, so this unregulated state doesn’t exist now.
Moreover, regulations and redistribution are stronger elsewhere in the developed world, and there is a strong argument to be made that the lack of these in the US, and the gap between rich and poor that it causes, is responsible for more suffering than having more of these things outside of it.
treis 2 days ago [-]
I don't think this adds up. The AGI blocks will have AGI kill bots and those that don't won't have a choice about what they will or will not allow.
lelanthran 1 days ago [-]
> I don't think this adds up. The AGI blocks will have AGI kill bots and those that don't
will have nukes.
Doesn't matter how much "intelligence" you have stockpiled when nukes start raining down on you.
treis 1 days ago [-]
But MAD means you can't use them. It might save you from an all out killbot invasion but it doesn't stop asymmetric warfare or being cutoff from the rest of the world.
jpleyden98 1 days ago [-]
Unless the AGI is able to develop effective defences to intercept an incoming nuclear attack.
d0mine 1 days ago [-]
MAD is a belief system. It doesn't matter whether AI can effectively stop the retaliation, it is enough that people making the decisions believed that it can (are we there already?).
Chance-Device 1 days ago [-]
Well if your baseline scenario is that swarms of marauding kill bots murder everyone then yeah, this is all moot.
KaiserPro 2 days ago [-]
> world join hands and agree not to compete with each other
We kinda do for other areas.
Largely the world agrees not to pirate software, tv, other IP.
THe issue of joblessness is well understood in china, they know that jobless people means a drop in living standards, a drop in living standards means the "contract" has failed.
So it makes sense that countries like america and the constituents of the EU understand that loosing 10% of all jobs in a few years is going to be a massive dick punch.
I also I think that finally the historians and economic types havae got through to the dipshit billionaire that they only have money because the plebs are spending money. If the plebs are unemployed and unemployable, they (the billionaires) are going to be lynched.
teamonkey 1 days ago [-]
> If the plebs are unemployed and unemployable, they (the billionaires) are going to be lynched.
Hence the robot dogs
HarHarVeryFunny 2 days ago [-]
> Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.
Sure - look at what is being done today with H1B visas and tariffs.
What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
Just as with H1B visas, you restrict usage. For example, don't allow AI to replace any job that pays under $250K.
If foreign countries make things cheaper, just as China is making EVs that cost a fraction of a Tesla, then slap tariffs on them.
A government should be primarily concerned about the people it represents, not about the profitability of the companies who are lobbying/bribing it.
Art9681 2 days ago [-]
Jobs are going to evolve because of AI. Some jobs will be automated. New jobs will be created. It's like saying we should have never created car factories and automated much of it because we should think of the horse handlers and maintainers.
There will still be a market for "trust". And its customers will never trust the machines. And within that market a lot of new startups will emerge for those who actually get to work and seize opportunities rather than lament a bygone era.
HarHarVeryFunny 2 days ago [-]
> New jobs will be created.
In the short term yes, but in the long term no.
What makes AI different to previous technological shifts is that it is (or will be) a general purpose technology, equally capable of itself doing the new jobs that it creates (so yeah, they'll be created, but also immediately taken away).
The saving grace is that this won't happen immediately, or as soon as we get something the AI companies slap an "AGI" label on in a couple of years, and it will likely take decades until we have an "artificial human" tech that genuinely could do any job (if we choose to allow that future).
Aurornis 2 days ago [-]
> Sure - look at what is being done today with H1B visas and tariffs.
I have some bad news for you if you bought into the idea that the tariffs were good for domestic jobs and the economy.
> What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
This is the definition of a false dichotomy, and you also ignored (or missed) the point completely.
I did not make an argument out of “free market doctrine”. I explained the second order effects that would provide a net reduction in jobs in this country.
My wife’s company is going through this right now. H1Bs are expensive and unpredictable, so the solution is to relocate more of the operations outside of the country and build up those offices.
Failure to understand second order effects and international economies is pervasive in these arguments for heavy regulations. You can do a lot to hold back your country’s companies to force them to pay for employees they don’t need and those few employees benefit for a while, but it collapses when those companies stop hiring in the regulated countries and resume hiring where they can actually operate a business. If you try to regulate them from doing it anywhere, you’ve given their competitors a wonderful gift to come destroy the company completely. Then they’re not making any jobs for anyone.
8note 1 days ago [-]
free market doctrine is an axiom to your second order effects. its so embedded into all your assumptions that you havent even considered it
hardbass 1 days ago [-]
Whatever you are calling "free market doctrine" is a physical law. Why would a business owner subject themselves to shitty conditions if they could find workers in better places? Some would of course, but not most or all.
HarHarVeryFunny 19 hours ago [-]
They won't - they'll act purely in their self or shareholder's interest.
If the government wants companies to act in a way that benefits the country and its people, then they need to pass laws to make that happen, for example by things like H1B quotas, or by tax penalties that make the desired behavior the most profitable.
hardbass 18 hours ago [-]
Why do you think protectionism works in benefit of the country's people? Ask Brazilians what they think of tariffs and protectionism. Ask for instance how many of them fly to the USA to buy electronics.
HarHarVeryFunny 17 hours ago [-]
I think you need to take tariffs case by case and ask if this an industry worth protecting - do you need to have it domestically, or are you ok to rely on imports? Maybe you want to protect high margin industries supporting good jobs, and are willing to rely on imports for things like clothing, food, electronic assembly. OTOH some things should have strategic importance - so then maybe you want to make sure you (the country) are if possible self-sufficient for things like defense, semiconductor manufacture, etc.
People, just like companies, are most likely to act in their own self-interest rather than in the interest of society as a whole. Sure, you'd like to be able to buy everything as cheap as possible, which in a high cost of living (which translates into high cost of manufacture) country like the US would mean relying mostly on tariff-free imports, and the death of the entire industrial base. Is that really what's best for the country?
HarHarVeryFunny 2 days ago [-]
Do you think outsourcing and offshoring of jobs, whether manufacturing or software development, has been a positive for the US economy?
Does it make sense to you for US companies to be sending wages overseas, supporting a foreign family and economy, when there is someone here in the US, unemployed and equally capable of doing the job?
Trump has certainly been ham-fisted about tariffs, but in a world without tariffs then the country with lowest cost of production, closely related to lowest cost of living, and lowest wages, wins, and this is not going to be the US in most cases. If you follow that path then a high cost country like the US will end up utterly reliant on other countries as it imports almost everything, and loses all domestic manufacturing capability. Sounds familiar?
qlte 14 hours ago [-]
Tariffs need to be credibly perceived as more or less permanent in whatever sector you're targeting for businesses to invest the large amounts of capital needed to reshore manufacturing.
Which is one of the main reasons Trump's arbitrary tariffs are ineffective policy, businesses see his capricious approach to tinkering with tariffs on a daily basis and instead make a rational decision to eat the costs under the assumption they are temporary.
Because he may change his mind in a week, SCOTUS may strike them down, Democrats win budgetary leverage in Congress, the right bribe gets paid for an exemption, etc. He could have enshrined them into legislation with the GOP's majority but politically they don't have enough support even within their own party, making the whole tariff saga pretty much pointless and self-defeating for any kind of long term restructuring of the US economy.
rickydroll 2 days ago [-]
Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.
Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial efforts is appealing, but I keep coming back to Bell Labs and Xerox PARC. Cool stuff was developed when you gave smart people a playground. Maybe the question should be: How do we give AI a playground and see what it comes up with? Although that could set us up with a situation like in the story Dragon's Egg.
Aurornis 2 days ago [-]
> How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.
I don’t think the proponents of these ideas care about who does the work. They see it as protection for specific jobs. Basically a jobs program with the costs forced on to large corporations.
As you said, it becomes a great gift to the few people lucky enough (or more usually, with enough nepotistic connections) to get those easy street jobs. It would not help everyone else in the economy.
It would also accelerate moving jobs overseas. Any company that regulates productivity that far downward would be crazy to continue doing automatable work in that country. It just gets moved to foreign offices where they can use AI. There are regulatory maximalists who say we’ll just regulate that, too, but then the whole company relocates to another country. Then some want to try to regulate that, and so on ad infinitum but it’s all layers of holding back your domestic companies so their international competitors can eat their lunch.
Loquebantur 2 days ago [-]
Earth isn't infinite.
You can and should engage in international regulation, as that is the only way to solve international problems.
The idea, capitalism would somehow lead to an ideal world all by itself is demonstrably wrong.
techblueberry 2 days ago [-]
I feel like the contra answer to “how do you enforce it” is like- imperfectly but we can do stuff.
Like we can just do stuff. We can fine companies, take away licenses. We are capable!
But I also think - I’m not trying to be too negative here because certainly we should be investing in scientists and other discoveries, but early tech - xerox park, googles 20 percent time. They were playing in an extremely immature space.
You could throw 20 ideas at a wall and create a billion dollar business.
We should do what you’re saying but we should also build institutions and maybe also limit the extent to which we’re building Elysium. If we can build AI models that rival human intelligence we can create laws to protect its dignity.
hgoel 2 days ago [-]
It's all about wanting to preserve the status quo, complete with all the suffering and strife, because change is scary and uncomfortable.
Add in the sincere American belief that everyone else is beneath them, and you get the version where they believe even developing countries must accept kneecaping their development so American corporatism doesn't collapse.
lljk_kennedy 20 hours ago [-]
> Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.
You're so close to describing a Universal Basic Income...
oceanplexian 1 days ago [-]
Just a reality check. They can absolutely do it.
Altman or Dario will donate some money to Trump, get cozy with the Department of War and make up a bullshit excuse why Open Source needs to be eliminated.
All access to this technology will be gatekept by a bunch of nasty people who want to eliminate your job as a knowledge worker and put everything behind a subscription.
HarHarVeryFunny 2 days ago [-]
> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).
If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?
I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.
alex43578 2 days ago [-]
You just invented the job equivalent of rent control, and all the problems that ensue from it.
uncomputation 1 days ago [-]
> I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Careful, you’re making too much sense. The big model providers would much rather the ball be in their court. “Deceleration” meaning Anthropic and OpenAI offer less capabilities for more money in service of “protecting humanity from extinction” rather than penny pinching. A lot of “AI research” coming out of these “labs” (increasingly enterprise) seems to be more driven by a cost benefit analysis from suits rather than genuine contributions to the field. I think the last true innovation was the idea of using doom and ending the human race as a marketing gimmick, which strangely seems to have worked in setting the narrative and captivating the sci-fi imaginations of journalists and techies alike.
rancar2 2 days ago [-]
I think Dario is writing with much of these things in mind, but using this as a specific framed opportunity to enable a better interim and long-term outcome for this planet with us still on it. When the internal motives for Dario are fully shared on the other side of the hill, I think we will find his writing to be more about influencing and shaping policy to steer the world in the best way possible within his ability, control, and knowledge. It’s a commonly held believe with the effective altruism community, which Anthropic team was formed under, to use the policy levers to positively influence the world. If I read this latest writing from Dario through that lens, there are deeper concerns with the AI tooling that go well beyond what’s noted in this singular post and are likely the motivations for re-attempting a slowdown or pacing. I too generally agree with for many reasons like allowing the young OpenAI engineers to make mistakes like the Hugging Face incident in July and put better processes in place (ie where is your decent multi-level fencing controls and harness best practices when disabling security features and where is your energy/spend limits for achieving the mission objective in scope and not warming the world for no reason!) and more time to engineer and deploy better hardware to not warm the world instead of burning through fossil fuels since now data centers in the US can isolate pollution from the grid to exempt themselves from federal regulation.
threecheese 2 days ago [-]
Only token cost can slow down the pace of replacement. If Anthropic were to stop all training and allocate 100% of capacity to inference, will this increase or decrease usage cost? If (for example) Fable came down to the cost of Sonnet, corporations will absolutely jam in the gas pedal.
Davidzheng 1 days ago [-]
Doesn't it just incentive the business to be outcompeted by one that uses ai more--unless the consumer is choosing based on morals
skue 1 days ago [-]
There are serious people trying to figure out what that looks like. Check out the recent Freakonomics episode with Gina Raimondo.
abustamam 2 days ago [-]
> stop trying to replace all human labor
A lot of companies trying to replace human labor with AI are either lying (theyre laying off because they over hired and need to correct) or will regret it.
That being said, what's the difference between a company that replaces 5 people with AI and a company who would have otherwise had 5 job openings, but decided to delegate to AI?
Personally, I-d rather be able to reap the economic benefits of AI by having a 4 day workweek. Instead of using AI to replace 1 FTE, use AI to offload 8h of work a day for 5 FTEs.
Of course, this is probably more outlandish than your idea.
(Or, we could just make basic income a thing and no one has to worry about their basic needs, but if they want the new iPhone Duo or a Rivian or other luxuries, they can work for it, but that idea is probably most outlandish of all)
stale2002 2 days ago [-]
The whole point of technology is to replace work that we don't want to do. You are attacking the chief reason why people use technology in the first place.
Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.
GrinningFool 1 days ago [-]
You should probably look closer at unemployment numbers in IT vs other spaces. Gains elsewhere are making up for IT losses, but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles in large numbers.
MentalM 1 days ago [-]
> but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles
That is because they can allow themselves to do nothing (for some time). Tech was a very high‑paying type of jobs, so workers could afford not to move into lower‑paid sectors, living on the savings they had accumulated beforehand.
In a couple of years market will correct tech-salaries and people will have spent their savings and then there will be your career shifting into healthcare and hospitality roles in large numbers.
xg15 2 days ago [-]
Work like writing blogs and drawing pictures?
stale2002 2 days ago [-]
Yes, there are many parts of those things that some people want to be automated. I and many others are able to execute on our creative vision much fasters and more effectively thanks to automation of that work.
vouaobrasil 1 days ago [-]
Personally I find it more difficult now to execute my creative vision compared to years ago with a little less tech. I think there's an optimal point to automation that is easily reached and after that the benefit from automation actually goes away. People just don't like to acknowledge that because it goes against what they're doing and that creates unpleasant cognitive dissonance.
vouaobrasil 1 days ago [-]
That's not really the point of technology at all. It was at one point perhaps with simple tools, but now it's more like "find a more efficient way to replace work in the short-term to get an advantage over others". The goal of our development has ceased to be improving life. Now it's just surviving in the market.
People don't use technology to free up their time any more. They use it because technology keeps making life more complicated and tiresome and new tech is a short-term amelioration to that until that too increases the complexity of life some more.
augment_me 1 days ago [-]
This is a call to end capitalism. Its completely impossible to do what you suggest with the incentives of capitalism.
Davidzheng 1 days ago [-]
Actually capitalism kind of has to end further down this road if we don't want to cede control completely to super intelligent ais and their direct "owners".
Davidzheng 1 days ago [-]
Market equilibrium probably tends to give 100% resource to ai building bc it leads to most returns and all human labor goes to zero.
Caveat I know nothing about economics lol--but this is my uninformed opinion.
anon291 1 days ago [-]
Forget capitalism. This is a call to end the basic right of people to multiply matrices
CrimsonRain 2 days ago [-]
You can "pace" yourself like that if you want. Don't shove your idiotic self-destructive ideas on to others. If you were president during invention of cars, we'd still be horse riding everywhere.
logicchains 2 days ago [-]
>Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
It's not just "companies". Even if the company does nothing, enterprising employees are going to use LLMs to magnify their productivity, which will may reduce the company's need to hire more employees. And except in extremely locked down environments, there's absolutely no way to stop an employee using some open source LLMs to multiply their productivity.
xg15 2 days ago [-]
Yes, but that employee might use that increased productivity to do the stuff that's always advertised as the benefits of automation: Allocate more time to tasks that usually don't get it - or: go home earlier and spend more time with friends and family.
caaqil 2 days ago [-]
> Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
AI is taking all of our jobs. We need to control the means, nay, the pace of production. We should build some kind of barrier, not of concrete and cement but also of the regulatory variety, to stop these agents from coming into our special white color cubicles. It's the only way.
xg15 2 days ago [-]
I don't see how the dual goals of "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work.
I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban.
> Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.
Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.
pibaker 1 days ago [-]
Many Americans seem to think other nations are NPCs in a video game that only exist to make the main character — the US of A — feel good. They conveniently forget that other nations have their own national interests that are often at odds with ours.
This line of thinking might have worked in 1950 or 1990 when the US had the leverage over others, but we don't live in that world anymore. And it's up to us to get used to the new world instead of making bad decisions based on the one we grew up in.
pizzly 2 days ago [-]
This contradiction "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work. Its just fantasy. The only way to enforce this is with military force and the cost would be too unbearable. You could try trade restrictions but the previous tariffs did not work and doubt future ones will to. US citizens (and thus politicians) won't bear that pain. Thus at a minimum this contradiction will mean AI will continue to develop until both US and China are at the same level. Playing with logic gives you possible scenarios. If China catches up within 1 year then negotiations could begin. If China takes many years to catch up, say 6 months behind now, then next year 5 months behind, then year after 4 months behind then expect overall AI development to proceed at full speed. The one way you could do it is to negotiate to transfer direct technology to China that will ensure that they will have the capability to be on par with USA. Politicians won't do this publicly as they want to get elected but they could make a deal that appears to appeal to both sides while actually transferring technology.
tcdent 2 days ago [-]
This is the part I find being left surprisingly hand-wavey.
If you extrapolate it out, you see that it has a high likelihood of inciting physical force (read: military action) as a means of enforcement, so perhaps that's why nobody promoting ideals has been direct about it.
2 days ago [-]
csomar 2 days ago [-]
Military action against whom? the US can’t even handle Iran let alone China.
fmnxl 2 days ago [-]
Americans live in their own plane of reality, where they play the superhero surrounded by evil villains.
cavemandaveman 1 days ago [-]
America doesn't want to pay the price of handling Iran. There's a difference. Clearly the US could conquer Iran but it requires a land war with mass casualties, not just sporadic bombing.
Obviously China is in an entirely different league.
spopejoy 1 days ago [-]
> Clearly the US could conquer Iran
Vietnam (and Ukraine) would like a word
olelele 1 days ago [-]
Korea & Afghanistan too.
The consequences of the Iraq blunder are also still resonating.
le-mark 1 days ago [-]
> America doesn't want to pay the price of handling Iran.
And Iran has wisely not provoked the US public by launching terrorist attacks against the “homeland”.
pibaker 1 days ago [-]
"I can afford ferrari. I just don't want to pay the price."
ArcHound 2 days ago [-]
Some people would like to have law that protects them but doesn't bind and for others to be bound but not protected.
They would like to slow down China while speeding ahead. Why wouldn't they want that?
Now they have to make that happen somehow.
2 days ago [-]
streptomycin 2 days ago [-]
Probably thinks he has a better chance negotiating with China now than with misaligned AGI in the future.
mordae 23 hours ago [-]
Why would China want to slow down when it is winning?
xiphias2 2 days ago [-]
Sure, Dario needs to decide first if he sees China as an enemy orva friend.
Limiting NVIDIA chips already turned China into silicon compete mode, and China is already far ahead in robotics.
iloveoof 2 days ago [-]
Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.
stratos123 2 days ago [-]
Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.
sgt101 22 hours ago [-]
Given that OpenAI and Anthropic have had major incidents in the last few weeks why will giving them a monopoly on models make thing safer?
The correct response is for these two companies to be closed down and their executives imprisoned. Yet somehow they have a free pass and are pointing at people who have not done any of the things they have and asking for them to be penalised.
This is just bizarre.
fwn 2 days ago [-]
The biggest AI risk, by far, is the concentration of power in a few companies, located in the US.
There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.)
And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn.
The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.
twoodfin 2 days ago [-]
I dunno, do you really want to bet that if you took a deliberately unaligned frontier model and asked it to permanently disable the US power grid, it would do a poor job of it?
What odds would you take on this bet?
I’m disturbingly close to 50/50.
fwn 2 days ago [-]
If we focus too much on the world domination fiction, we really might, in a moment of lapsed attention, end up plugging essential infrastructure into the public internet, hoping some dream of global alignment might save us.
That's like giving a gun to a monkey and hoping the monkey is trained well. ..a frontier monkey though.
twoodfin 2 days ago [-]
You believe essential infrastructure is air-gapped from the public internet today?
qlte 14 hours ago [-]
The nuclear arsenal is absolutely air-gapped from the public Internet today. And launching ICBMs is a key plot detail in many AI apocalypse fantasy scenarios.
fwn 2 days ago [-]
No. And no alignment, guardrails or slowdowns will ever secure our insecure infrastructure. Bad infrastructure decisions are a risk independent from AI.
The only way to secure infrastructure is to actually do the work needed to secure it. Our waste water treatment facility might not actually need to be able to tweet its status.
Cybersecurity is such an important topic for a country, dreaming about global alignment just to avoid fixing insecure infrastructure cannot be serious.
A Russian state backed hacker will not ask Dario for permission or argue with his LLM about ethics. That train departed long ago.
estearum 2 days ago [-]
> Bad infrastructure decisions are a risk independent from AI.
All decisions are made relative to their tradeoffs including costs and risks.
"Water levels rising 100x beyond historical levels are not the problem. The dam's height is the problem!"
All systems should be made infinitely secure and all dams should be made infinitely tall.
2 days ago [-]
disgruntledphd2 2 days ago [-]
Yeah, we should give a limited copyright waiver for pre training, given that the model is released as open weight and allow labs to compete with RL on that basis.
pastel8739 2 days ago [-]
He did actually specify that diffusion should be limited for authoritarian countries, which I appreciated. If his focus is really safety, distillation in countries that are bound by safety regulation should be fine
verdverm 2 days ago [-]
> bound by safety regulation should be fine
I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation.
pastel8739 2 days ago [-]
yes but the mention of alignment is inside the "pacing within democracies" section
Miraste 1 days ago [-]
That section is so biased it reads like satire. The current US government is not safer or more responsible than China, nor is it a democracy. I can't tell for sure if this is a clumsy attempt to ingratiate Anthropic with the administration, or if he actually believes it, but given the "I agree with Secretary Bessent" name dropping I lean towards the former.
verdverm 2 days ago [-]
alignment does not have a uniform definition, so "who gets to decide the right answer?"
(1984 has some thoughts on the matter)
un_montagnard 2 days ago [-]
Anthropic is about to go public, but they figured they can't keep making improvement at the same pace they used to. If the frontier is paced, they can continue hyping their unreleased capabilities that coincidentally cannot be released due to things outside their control. Which buy them more time to try to improve the models.
boshalfoshal 1 days ago [-]
This is just completely false lol. They most certainly can make better models - why cant people here just read this for face value? I think anthropic is legitimately concerned about the safety implications of stronger models. Race dynamics necessitate that they make better and better models, which they clearly think is bad.
They have an internal model which could solve a millenium problem end to end with no intervention, whereas current available models can't. They can clearly make better models. Not everything these guys say is some 500iq game theory optimal 4D chess PR or subterfuge strategy.
eknkc 1 days ago [-]
> which they clearly think is bad
These are companies with huge amounts of money invested in them. Their main responsibility is to their investors. If building better models (or making more money from the models they already have) would increase returns, they would do it. So they clearly can’t do it right now.
Climate change hasn’t stopped the oil industry, and concerns about forever chemicals haven’t stopped the chemical industry. Companies don’t usually give up profitable opportunities just because they might be harmful or kill us all in the long term..
boshalfoshal 1 days ago [-]
I think you fundamentally misunderstand what kind of company Anthropic is. I'm simply taking the path of least resistance when understanding what people at Anthropic have to say, and basically every public source points to Anthropic talking the talk as well as walking the walk with regards to AI safety and their ideas on AI governance.
Re the point on returning shareholder value, Anthropic is a PBC:
> Public Benefit Purpose: Under Delaware law, a PBC allows directors to legally weigh social and public good alongside maximizing profit for stockholders
Investors surely know this and Anthropic probably has pretty reasonable legal grounds to stand on if they chose to prioritize safety over profits.
Seems pretty clear to me that this is in earnest.
qlte 14 hours ago [-]
The decision to found Anthropic to compete with OpenAI's then concentration of SOTA LLM AI researchers, and then achieving/surpassing parity with OpenAI is pretty much directly responsible for creating the current capabilities arms race.
Google has shown they are not motivated at all to push the bigger and bigger SOTA frontier with Mythos-type super models. Even OpenAI isn't really even trying to surpass Anthropic at the top of the market, preferring to keep pace in benchmarks but at lower costs to serve. Anthropic with Mythos is now more or less alone in aggressively pushing the ceiling for the most "super-intelligent" models current computing can support.
It has made them and their investors a gigantic pile of money, but we're really supposed to believe their contribution to that arms race (which they now describe as a threat to humanity) has always been completely altruistic and that their corporate structure inherently makes self-interested decision making impossible?
boshalfoshal 8 hours ago [-]
If you've read enough lesswrong, this won't be new to you. The strategy always was "become the most capable model lab so that you have leverage to control AI development."
eknkc 1 days ago [-]
Oh I am terribly sorry, did not know Anthropic was formed as a a “Good Company”. Which changes everything ever.
Snark aside, maybe I’m too pessimistic and you are correct on this. But.. doesn’t that mean we are completely fucked? If they don’t have any ulterior motive and they truly are panicking.. We should start packing.
boshalfoshal 8 hours ago [-]
yes that is literally why they are sounding the alarm - because people think AI is a party trick and they aren't taking it seriously enough.
nsdkdkdkfk 18 hours ago [-]
[dead]
timcobb 22 hours ago [-]
> why cant people here just read this for face value?
Because that would be foolish.
What part of public good is stealing giant corpuses from their owners and selling them back to them? People who show contempt for laws do so pretty consistently usually.
It's really sad that people fall for this stuff over and over again.
the_gipsy 1 days ago [-]
> why cant people here just read this for face value?
Why should we? Yes, reading the comments, it seems like the IT (or maybe it's just the AI) community has taken up an attitude of simply believing our Dear Tech Leader's by their word. What happened?
zinodaur 2 days ago [-]
> Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers
Can someone clarify this for me? How far along would the open weight models be without the frenzied pace of the frontier labs?
As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
kennywinker 2 days ago [-]
> As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
Israel is doing that too.
I believe Russia and Ukraine have both also used AI powered autonomous drones in warfare at this point - tho probably not frontier ones, since they need to run on device or else they're not autonomous.
cpeterso 2 days ago [-]
> Iran and Houthi rebels used Anthropic's Claude AI to target US warships and build hypersonic missiles — Houthi rebels also used the bot to code ballistic missile guidance systems
And the source for the claim does not even clear the bar for r/bodybuilding, because it’s “trust me bro.”
j_maffe 2 days ago [-]
Yeah the Chinese fear-mongering falls a bit flat when coming from a point of maintaining US supremacy
romanhounds 2 days ago [-]
[dead]
rickydroll 2 days ago [-]
I also like the idea of pacing the frontier and slowing down development so we all have a chance to stop and catch our breath. However, I don't think regulating AI is the way to do it. I would tackle the problem by constraining the resources used to run AI models; the "simplest" way to do that is increasing the costs of running a data center. The two easiest places to raise data center costs are electricity and water tariffs.
I think those two tariffs are the best place to raise costs because they can be increased incrementally, state by state or country by country. The main problem with the regulatory approach is that it's a "big bang" post-board approach. Nothing happens until the regulation is defined and deployed, and corporations are experts at delaying implementation.
Yes, increasing tariffs means touching many individual regulatory domains at the state and municipal levels, but like deploying solar energy or wind power, you can do it incrementally.
azan_ 2 days ago [-]
Yes, water not used for human consumption should be more expensive (we should stop subsidies for that). Of course it won’t affect datacenters because they don’t use that much water, but well at least get rid of the most inefficient agricultural practices.
jfreds 2 days ago [-]
I think taxes like these actually worsen the incentive structure. If it’s effectively more expensive to do training and inference, labs will be incentivized even more to improve efficiency. Opaque recurrence seems like a great way to improve efficiency, and it comes at the cost of safety
xg15 2 days ago [-]
Even more opaque than the models already are? I take that risk.
I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...
estearum 2 days ago [-]
> I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...
It seems like you're implying there's some better alternative, but you're just describing a race-to-the-bottom. It is completely reasonable for every competitor to want an external coordinator (i.e. regulator) to break the pathological competitive dynamics.
esafak 2 days ago [-]
This makes no sense. The hardware is only going to get cheaper. In a decade, everybody could be running an ASI on their phones.
What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.
xg15 2 days ago [-]
> The hardware is only going to get cheaper.
Is that so? Last time I checked, RAM was getting a lot more expensive...
dale_glass 2 days ago [-]
Eventually data centers will get sated for one reason or another, or production capacity will increase, or manufacturing will shift toward higher capacities. Or some combination of those.
esafak 2 days ago [-]
How long do you think that's going to last? A few more years, at most. Wail 'til the Chinese ramp up production.
politician 2 days ago [-]
I downvoted you because I think the OPs model is far simpler to implement and suffers from fewer conflicts of interest.
If we go down the regulation of alignment route, we'll have to ask experts to create those regulations and monitoring regimes. And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!
When "the experts" make certification cost $100M/model because "safety and alignment", then they will have cemented their duopoly.
Meanwhile, tariff rates on compute is a simple percentage that can be scaled up and down. Congress doesn't need experts to do that. Vendors can participate proportionally to their scale. This is far simpler and far more fair.
convolvatron 2 days ago [-]
I think the point is even stronger. alignment is really not very well defined, nor are the mechanisms used to provide it. from what I understand it's more of a statistically imposed bias than an impenetrable gate.
personally I would like to flip this around, all of this is a consequence of some really extreme notions about investment. its a tail-wagging-the-dog. the investment community is apparently in love with the idea of an AI-scale vehicle in and of itself. given the amount of power they have, this is funneling a massive amount of resources into something that while really very interesting, probably would never be able to realize proportionate gains without taking down the system that built it. what's the societal control on that.
1 days ago [-]
esafak 2 days ago [-]
Probabilistic safety is too low a bar. We need to project the model outputs onto a safe subspace; lobotomize them, if you will. It may make them dumber, but that's a fine price to pay.
I don't work in this space so I don't know the latest, but here's an example: Provably safe systems: the only path to controllable AGI (https://news.ycombinator.com/item?id=37619285)
convolvatron 1 days ago [-]
I don't think the neuro-symbolic model is going to dumb anything down at all. Cyc didn't end up producing anything useful, but the only reason an llm is able to do math at all is that it can keep throwing stuff at lean all day. personally I think that that synthesis will turn out to be more powerful than just llm with constraints, but I don't really work work in the field either.
esafak 2 days ago [-]
> And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!
You've never heard of academia, NGOs, and intergovernmental organizations? Sure, the AI labs will have a say but not all of it. You don't ask the fox to guard the henhouse.
politician 1 days ago [-]
[flagged]
visarga 2 days ago [-]
Yes, and we need to make microprocessors that do good work, not hack others. /s
esafak 2 days ago [-]
Why is it so hard for people to understand the concept of agency in 2026??
What did your CPU do without instruction? Did OpenAI tell its agents to hack the orgs it did? Come on, make a half sensible argument.
visarga 2 days ago [-]
As long as anyone can put anything they want in the prompt box there is no safety. It's just a tool, the danger comes from what it is being used for.
thadt 2 days ago [-]
Hard disagree. Our choices are:
A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or
B) Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality.
If I have to put my community’s wellbeing on the line behind one of those possibilities, I know which I’ll be betting on.
cardamomo 2 days ago [-]
These do not seem to be mutually exclusive. Should we not do both?
michaellee8 2 days ago [-]
Do you really think we can really get every country to truly pace the frontier? Pretty sure China won't give a f until they catch up Anthropic and OpenAI. It is an arm race. We had nukes for like 70 years and still haven't figured out how to make every single country follow those nuclear treaties, with an increasingly non-interventionist US I don't think we can get every single country to the table and agree to a pause. Will US accept their frontier being caught up by Chinese Labs? I don't think so.
amazingman 1 days ago [-]
We don't need every country to pace the frontier, just a certain few. And yes, I believe it's possible and the prior art is nuclear non-proliferation. Non-proliferation wasn't perfect, of course, but it was good enough (so far) to pull back from the brink of extinction.
user43928 1 days ago [-]
If China for some reason agrees that would probably be enough for the time being.
Who else should develop AGI, Mistral? Maybe in a decade.
causal 2 days ago [-]
Didn't answer the question...
MentalM 1 days ago [-]
> Do you really think we can really get every country to truly pace the frontier?
Why not? In 20-th century half the world was socialist. And the idea of moratorium on improving AI way easier to sell then socialism.
HellDunkel 2 days ago [-]
B) needs more time and A) provides exactly that.
TuxSH 2 days ago [-]
Of course it is B). If you can use LLMs to find bugs and vulns in software you don't own, you certainly can use them to find bugs in software you _do_ own. They are amazing at that job.
Also, models like GLM 5.3 have zero guardrails in that regard ("find vulns in (...)" prompts just work)
causal 2 days ago [-]
> Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality
Smashing the gas pedal is not "taking appropriate measures". Did you read the article? Needing more time for more secure operations is literally one of the arguments for pacing.
glub 2 days ago [-]
> We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top
> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time
And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.
I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?
Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
stratos123 2 days ago [-]
> Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
I'd expect he thinks that people are capable of realizing that AI is very dangerous and also that democracies end up mostly representing the will of the people, from which it follows that Hypothetical AI Leader Australia would agree to ban it too. This argument doesn't work for countries which don't care what their citizens want, like China.
I do think that it's a questionable decision to alienate China this much in this essay, instead of leaving open the possibility of China agreeing to a treaty that'll limit their progress. I suspect Dario is doing this to signal his allegiance with the US government, in hopes to increase the chance they'll go along with him, which is an unfortunate choice but plausibly the correct one.
c0rruptbytes 1 days ago [-]
> This argument doesn't work for countries which don't care what their citizens want, like China.
you didn’t have to use China as an example, the US clearly does not care what its citizens want as the most popular policies are never even discussed or proposed in congress
meanwhile, China destroying their housing market to decommidify it so everyone can have housing…they seem to care about their people more
arximboldi 15 hours ago [-]
another obvious example is climate change, where China is way more committed to solving it than the US. the division of the world into good guys and bad guys by a rather arbitrary definition of democracy looks so naive, even stupid. Anthropic is not a Democratic institution either, should we trust it? if this proposal for pacing the frontier is to be taken seriously, Dario should give its competitors, also international ones, the same assumption of good faith that he demands from us.
glub 2 days ago [-]
I don't think people care as much as we'd like them to care about dangers of technologies. It takes a single step outside of technological bubble to see that their opinion of SOTA LLMs is vastly different. To them, AI means ChatGPT and ChatGPT is mostly still the same ChatGPT that it was 3 years ago, with similar failure modes and nothing that would indicate it would kill them, or take their jobs even.
It's the same as it has been with privacy/cybersecurity for decades. Vast majority of population doesn't care about hypothetical dangers, no matter how many essays get published.
So democracies representing will of people doesn't really work in favor of Dario's case here.
History is also not on his side. Limiting technological progress in the name of safety has a pretty poor track record.
glub 2 days ago [-]
Dario, Sam, and Elon are all on the same page on this.
So it's either they truly think AI is going to kill us all, or there's some other motives at play here.
I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know?
OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.
xAI is tracking behind, and whatever regulation it may be that paces the frontier, Musk is less likely to be affected by it. Therefore xAI should be pro-regulation that stiffles his competition and gives him time to catch up.
qnleigh 2 days ago [-]
> OpenAI / Anthropic models have largely stopped advancing
I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.
glub 2 days ago [-]
> This year it became common for people to entirely delegate coding to AI
This has been the case for around 2 years now, more reliably - a year. We've mostly stayed there since then.
Saying that more people started doing it isn't indicative of significant improvement. Some people just started doing it later.
I can't speak about math because I haven't used AI for that application, but I know that there hasn't been any significant advancement in coding in this year on base models. There has been more RL work, more harness work, more tools, they all expanded some capabilities like cyber or orchestration or tool use, but raw intelligence of base models is no longer where the main focus is.
BobbyJo 1 days ago [-]
> This has been the case for around 2 years now, more reliably - a year.
I have to disagree with this pretty strongly. Opus 4.5 needed a lot of handholding not to work itself into a corner pretty quickly. Fable I basically never need to correct, and I've most become a data source.
itkovian_ 1 days ago [-]
What are you talking about - I feel like we’re living in parallel realities. If I had to go back to opus 4.5 tomorrow I’d be hugely upset and significantly slowed down
bel8 2 days ago [-]
I'm not. Yes we normalized 1m context window and models tend to hallucinate less.
But models have been somewhat stagnant since Opus 4.6/7.
And in some regards there were even regressions like Claudeisms that are load bearing.
boshalfoshal 1 days ago [-]
Yes these guys are completely delusional.
2 years ago a model could barely solve the AMC, 1 year ago it got IMO gold, and this year models have solved multiple millenium problems.
Even 1 year ago ai code was just unusable (claude code only became available 1.5 years ago!) and now basically everyone I know from independent shops all the way to faang and anthropic/openai themselves exclusively use some AI agent to code.
Why does HN continue to delude itself that "models are not improving?" Maybe for the simple things they care about its "roughly the same," but they are _clearly_ improving.
qnleigh 1 days ago [-]
Waitwaitwait multiple millennium problems? What was the other one???
Increasing a bound on the fraction of Reimann zeros on the critical line doesn't count; even Anthropic says they don't think this line of work will lead to a solution.
boshalfoshal 1 days ago [-]
Navier Stokes, and _allegedly_ the Hodge Conjecture and Birch–Swinnerton-Dyer.
IanCal 2 days ago [-]
> OpenAI / Anthropic models have largely stopped advancing
Have they? That seems like quite a claim given the last 6 months, particularly for cybersecurity.
glub 2 days ago [-]
The attention is shifting towards RL, harnesses, and memory systems from the pretrains of more intelligent and capable base models. So extracting additional capabilities from what we already have.
That is a much easier catch up game. GLM 5.3 and DeepSeek flash 4.1 also demonstrate significant jump in cyber capabilities. So yeah, it is a slowdown in the place where it matters. RL has been around for ages, there's no moat there if you already have a good enough pretrain.
nicce 2 days ago [-]
Many claims but no clear evidence that they actually find significantly more severe issues compared to open models.
echelon 2 days ago [-]
Open models and agents can't be trusted without handholding. Astra can one-shot six months of work. Years of work, even.
OpenAI just solved Navier-Stokes.
Seems like the US is on a takeoff ramp to me.
glub 2 days ago [-]
Tell me you haven't tried letting Astra go without telling me.
Astra can confidently one-shot 500k lines of slop, with 800k lines of tests covering it, without testing a single intended product requirement, and none of it actually working.
All models require hand holding. Fable and Astra are no exceptions. The difference is only in the amount of hand holding required, and there's essentially no gap here anymore between American and Chinese models.
I only use Chinese models sparingly because American models are so much cheaper with subscriptions, that it doesn't make economic sense to not use them. If/when that changes, I could simply route to cheapest model that's available at the moment and I wouldn't notice much difference in most applications.
simianwords 2 days ago [-]
Here are the cope points
1. Navier Stokes was plagiarism
2. All benchmarks were misleading wrong and incorrect
3. All other mathematical advances were again hype
4. HF incident was marketting ploy jointly coordinated by HF, METR and OpenAI (and also Anthropic)
5. Anthropic's HF like incident was again a marketing ploy [1]
Nothing ever happens. This whole thing is a scam. Everything is done to fool you and you have fallen for it. Congrats.
> OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.
This is obviously untrue… do you use any of them?
glub 2 days ago [-]
Anthropic could serve Opus 4.5 from a year ago under opus:latest and most heavy users would probably have no idea. Some of them would probably even prefer it.
Yes, I do use them, quite heavily. The only difference at this point is in benchmarks that can't be trusted (see: artificial analysis on Astra), or in the way models communicate.
Most gains are now from RL, which for some reason is hyperfocused on improving cyber capabilities, and harnesses. Raw intelligence gains of base models is absolutely slowing down.
simianwords 2 days ago [-]
This line will keep repeating because it is necessary for the narrative:
AI in general is just hype and unprofitable and all these companies are playing marketing tricks before the IPO after which they will cash out and let the economy crash.
This is legit what a lot of people think. To continue this narrative, they have to keep up the charade of "things are not improving".
villish 2 days ago [-]
Add in a heaping dash of anti-american sentiment, and you will get the truth behind the pessimistic commentary.
Downplaying the latest models capabilities is frankly insane considering what we’ve seen what OpenAI’s models have done without safeguards. That wasn’t possible before this latest generation.
rphv 3 hours ago [-]
> So it's either they truly think AI is going to kill us all
Yes. Basically everyone here is dismissing this possibility. We shouldn’t.
stratos123 2 days ago [-]
> So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky,[...]
And yet, they historically did agree on the existence of AI risk, since before OpenAI was even founded.
jmull 2 days ago [-]
Yeah, this 100% looks like an effort to use fear to create a regulatory moat.
ta20240528 3 hours ago [-]
"So it's either they truly think AI is going to kill us all"
Exactly how does a CPU kill 8 odd billion people? Even if they got hold of the weapons to wipe out a few billion, surely the remain billions would just turn off the data centres?
mattm 2 days ago [-]
Let's not forget that the competitive race happened because of them. Most of the initial AI research from the past decade started with Google Deepmind. Elon Musk was invited for a preview of it and ended up spinning up OpenAI when Demis turned down his investment offer. Dario was originally at OpenAI and left to start Anthropic.
This seems like a case of "save me from my own mistakes/ambition"
seizethecheese 1 days ago [-]
Which is it? Would the regulations slow down competitors or let them catch up, or are you contending it would let American competitors catch up but Chinese ones not?
MentalM 1 days ago [-]
> there's some other motives at play here.
I mean it is literally economy 101: some capitalists getting on the top using free market, and then try to use government to remove free market so their top position were secured from any competitors.
alchemist1e9 1 days ago [-]
Exactly. Textbook definition of “Crony Capitalism”. Which isn’t actually capitalism at that point.
kypro 22 hours ago [-]
> OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.
What is going on in this thread?
Are these even real comments?
While I strongly agree on the need to pace the frontier, I do think there are reasonable arguments that could be made against it.
Non of those are being made here though. It's just speculation that the CEOs of the fastest growing companies in the world are somehow obsessed with regulatory capture to maintain their lead, and have been playing the long game by talking AI risk for 10+ years so that when they become the CEOs of leading AI companies they can push for regulatory capture.
Perhaps they're all just genuinely concerned?
What would regulatory capture even give them? Who are they so worried will compete with them? Google, Mistral? Even if it's Chinese AI labs, it seems odd for anyone in the West to be opposed to any regulation that would slow them.
So assuming they are obsessed with regulatory capture – what's the argument for why they would want it?
glub 21 hours ago [-]
They may be genuinely concerned, but that's beside the point. You may think a technology holds too much power for someone to wield it, and therefore come to conclusion that you, the benevolent, the fluffy, the unicorn, with your 3 unicorn friends are the ONLY ones in the world of 8 billion people that are responsible enough to hold it.
Even though I don't think Dario is such a person, because unlike their LLMs, I do have a working memory, I'm going to play the ball and assume he is, indeed, an angel brought down by none other than God himself.
Why should CEO of a for-profit company be the dictator (pun intended) of which models I am allowed or not allowed to use, and in which way? Why should Dario have any say in what DeepSeek can or can not publish or whom they can serve?
> What would regulatory capture even give them? Who are they so worried will compete with them? Google, Mistral? Even if it's Chinese AI labs, it seems odd for anyone in the West to be opposed to any regulation that would slow them.
I am opposed to it. Why would I be opposed to open weight models? Why would anyone be opposed to open weight models or more competition other than the ones that get their bottom lines hurt by it? The reason why you still have reasonably inexpensive access to western AI models is because China has been breathing down their necks. Otherwise, you'd be paying a whole lot more money per token were it just these 3 running the show.
kypro 15 hours ago [-]
If you want to limit the power of companies you need regulation...
I agree with you that we can't allow a handful of people to hold this power themselves, so what are you arguing for here? That the answer is to just leave AI labs to carry on as they are?
The government should be ensuring that the technology is being developed with caution, but since the government seems to have no interest in that, this is at least better than nothing.
Your solution of giving everyone access to super intelligence is absurd. A good guy with super intelligence isn't going to protect you from the bad guy with super intelligence. If someone uses it to poison your water supply with a bioweapon or to take out power stations, you and your family will be dead long before your super intelligence hacks together plan to stop it.
glub 15 hours ago [-]
> That the answer is to just leave AI labs to carry on as they are?
Yes, and no. I do not think there's a need for special treatment of LLMs. But the labs certainly shouldn't be allowed to go around hacking things on the internet when they could have prevented it by simply taking time to ensure sandboxes they run test models in are given more than a quick look by a junior engineer to configure.
But the interesting thing is that there are laws that already govern this. They're just, for some reason, are not applied to Anthropic and OpenAI. Cyber crime is already punishable, and both labs did, in fact, commit cyber crimes. Did the existing regulations fail? No, the enforcement did.
The same goes for bioweapons. Any chemistry undergrad that can read an openly available paper can manufacture insane things with just common household ingredients, they do not need LLMs for that. This is a made up scenario with no evidence whatsoever to support it. There are millions of people that could poison your water supply today, without ever touching an LLM.
If we're afraid of LLMs turning into superintelligence, and we think superintelligence shouldn't be available to anyone at all, then we must all agree not on pacing the frontier, but just stopping any and all research into this. And that's not going to happen.
> So it's either they truly think AI is going to kill us all, or there's some other motives at play here.
Duh! It’s called collusion. They want to try and hoard the technology for themselves if possible!
afdgnionio 2 days ago [-]
[dead]
TheSisb2 2 days ago [-]
I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I.
That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Jcampuzano2 2 days ago [-]
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
0xDEAFBEAD 2 days ago [-]
Aren't they beholden to investors already? Why would an IPO make a big difference?
In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
Jcampuzano2 2 days ago [-]
If anything, his entire argument leads to the eventual premise that AI labs need to nationalize.
He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead.
If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates.
That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
fasterik 2 days ago [-]
I don't agree that the argument implies nationalization. Regulation would work if it slowed everyone down at the same rate, enough to mitigate the risks. The problem is that you need regulators who know what they're doing and strong international cooperation. If you regulate a fraction of the global market, you just create an incentive to shift development to other countries. Nationalization, being essentially the most heavy-handed form of regulation, faces the same problems and comes with its own risks as well.
We're talking about this like all of the bad incentives are created by market competition, but that's not really the case. Most of the incentives come from untapped value in the form of potential profits, strategic advantage, military superiority, etc. Corporations and governments want to capture this value for themselves, creating various types of competition. Dario's argument depends on the assumption that the primary risk comes from the pace of development and threats from the technology itself. That's probably where I disagree the most; I think the highest risk is rising authoritarianism and competition between nation states. Slowing down isn't really a solution to those problems.
narnarpapadaddy 2 days ago [-]
I think this is the correct take. And until we have an AI Hiroshima it’ll be difficult to get the international community to work together. It’ll take rogue AI (or AI-powered group) disabling a significant world power before everyone comes to the table. Otherwise, it just looks like MAD and the equilibrium holding to the powers that be.
PantaloonFlames 2 days ago [-]
It wasn’t the shock of Little Boy that ended the war. That was just a final chapter of a 10+ year journey of horror, despair, and destruction, and the impact of Little Boy cannot be considered outside of that journey.
Europe had been destroyed by June 1945, and yet the empire of Japan continued to fight.
If that pattern holds, a single malicious AI substantially disabling a single world power won’t end the AI race. Participants don’t learn by the defeats of others.
If you’d like to apply a metaphor maybe the 10+ years of world war is more appropriate, after which basically every participant save one was exhausted.
narnarpapadaddy 1 days ago [-]
No disagreement here; I felt countless smaller problems leading up to the final blow was implied.
0xDEAFBEAD 2 days ago [-]
Interesting points, but given Dario's previous conflicts with the DoD, I doubt he has a ton of faith in the current US administration.
2 days ago [-]
2 days ago [-]
ahsillyme 2 days ago [-]
Assuming they can achieve funding without IPO probably yes. But if they can't there's always the dilemma that "if [good guys] won't do it then [bad guys] will". I'd like to think that the leadership at anthropic is principled even if the actions of the company as a whole has been less than stellar morally speaking. I'd be curious to see their moral calculus transparently laid out in public.
gewa 2 days ago [-]
We are still living in a capitalistic society. We have to find a solution which is responsible, safe and returns on the investment. The commercial aspect can be true at the same time.
PantaloonFlames 2 days ago [-]
The potential of AI may give us the opportunity to graduate to a post-capitalism, or to move to a sort of neo-capitalism, which is governed by different rules.
Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?
8note 2 days ago [-]
you are imagining magic as a reason to dump all your money and future money into a casino. You might instead consider joining the catholic church? they already have a free energy god that gives everything you could ever want
the cost of energy is already ~0 and people dont want it, and refuse to participate in letting other people have free energy if it affects their view of their pasture.
unless you are building killer robits with the intention of doing some soviet or nazi styled purges of everyone that might get in the way, you arent gonna get this ai utopia
tcdent 2 days ago [-]
Read other writing by Anthropic about potential future financial implications of AI. [1]
IPO is a vehicle for future wealth distribution. It leverages existing financial frameworks in order to span the gap from before-to-after without having to convince the system that future is inevitable.
I don't think Anthropic and OpenAI will IPO at all. Word is that Anthropic will have $100 BN in revenues this year, and very likely OpenAI will get some similar amount. You IPO when you need money, and I think Anthropic and OpenAI are past that point.
wmf 2 days ago [-]
They want to spend over $200B/year on $100B of revenue so they still need investment.
PantaloonFlames 2 days ago [-]
That level of revenue is apparently not enough for them to feel confident in the stability of their competitive position.
credit_guy 1 days ago [-]
And how does an IPO address that?
fifilura 2 days ago [-]
You also IPO if your investors want to cash out.
enraged_camel 2 days ago [-]
>> ...and become beholden to investors
Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.
ygjb 2 days ago [-]
Don't underestimate the ability of the courts and the state to pull the rug out from under the legal system. Speaking as an outsider from Canada, it looks very much like all bets are off in terms of respecting checks and balances in the United States and it's very realistically possible that unless there is a big upset and turn around the next couple of years any traces of democracy in the US will be a farcical nod to what the founders built as a way to paper over the abuses.
I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.
manquer 2 days ago [-]
As long you are burning more cash than you bring in, you are beholden to investors - whether it is retail, VCs, banks or a government giving you a bailout it is still someone signing you a check.
Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.
After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.
Bottom line you are always listening to someone.
8note 2 days ago [-]
google dropped "dont be evil" despite being the definition of the company
these are just words at a time and place.
sama showed that you can futz with it, and as long as you spend enough in court on judges, aint nobody gonna stop you
jimmydoe 2 days ago [-]
Ant: I'm doing very bad things right now, but I can't stop myself, you must stop me if you can. If you don't, that will be on you, not on me.
OAI: <silence>
Which is worse? I don't think they differ by much. It's just Capitalism, but accelerated.
> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
Jordan-117 2 days ago [-]
It's almost like the people at Anthropic and other AI labs are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction.
We built the paperclip maximizer, and it is capitalism.
swed420 2 days ago [-]
> are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction
This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?
Not holding my breath on that one.
meken 2 days ago [-]
> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t.
I disagree with this and I think the reason is well captured here:
> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.
129857 2 days ago [-]
MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said.
Then MSFT stole all IP from GitHub and made it worse and fired developers.
Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.
If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.
He is a greedy, ruthless person.
kalkin 2 days ago [-]
If Anthropic shut down (or even counterfactually had never been founded), would software development be freed from the impact of AI?
largbae 2 days ago [-]
Probably not, but we wouldn't have to listen to Dario's hypocrisy while it happened
0xDEAFBEAD 2 days ago [-]
How specifically is Dario a hypocrite? 129857's case rests on Dario being an "idealist". But maybe he's an idealist about curing cancer ASAP, and not an idealist about respecting copyright. That's not necessarily hypocritical.
largbae 2 days ago [-]
Multiple ways, starting from working to create the very situation he claims to fear.
Since you mentioned intellectual property, how about the hypocrisy of sucking in the intellectual property of humankind for AI training, but claiming it is unfair to use the results of this IP theft for AI training?
Matl 2 days ago [-]
For example he got into a spat with the Department of War as if he cared for how his AI could be used during war and yet said he's fine with Claude targeting a girl's school in Iran.
That's apart from the general fact that he continues to race towards the very thing he claims he's afraid of, because that's where his net worth comes from.
0xDEAFBEAD 2 days ago [-]
>said he's fine with Claude targeting a girl's school in Iran
and you know, that use case doesn't even violate their terms
0xDEAFBEAD 21 hours ago [-]
If a human approved the strike, the human is culpable. Would you go to the weapons manufacturer and try to hold them culpable for the strike?
Matl 20 hours ago [-]
> Would you go to the weapons manufacturer and try to hold them culpable for the strike?
Yes, depending on the level of autonomy of the weapon, how it was being marketed etc. If it was being sold as an 'AI' that is supposedly very smart so who's a mere mortal to question it, then yes.
I would of course hold the human culpable too, but this is why precisely even selling Claude for military use is immoral. The way the US conducts war means there's always a need for more targets. More and more targets, quickly. If the goal then is to hit as many targets as possible then each 'review' of a LLM suggestion is going to be more and more sloppy than the last one. If there's no 'AI' suggesting targets then the list is forced to go via more human review by the very nature of humans compiling the target list.
It's the same as using a LLM for code; the vast majority of programmers do not understand everything they're accepting, but they'll accept as long as it 'looks correct' and only examine more closely after something doesn't work, (in the military use case - after a strike).
LLMs are unreliable for anything more than reciting jokes, let alone picking targets. So yes, absolutely.
It's the same as Facebook being held liable for causing teens harm; they did not force any teen to use it and yet they knew what would most likely happen if they did.
8note 2 days ago [-]
i dont think he said it in those words, but the acceptable terms of use is that humans stay in the loop for picking targets.
so, claude suggesting killing a bunch of children, and then hegsdeth approving the strikes is perfectly acceptable.
claude putting a bomb in a girls school, and then lying to an operator that it actually gives ice cream an cookies, and the operator clicka the button would also be acceptable?
qw125 2 days ago [-]
[flagged]
nunez 2 days ago [-]
Giving credit where credit is due, I believe Dario and his squad formed Anthropic because he and Altman couldn't align on safety. The only way to build models like the Claude series is to play dirty and train on LITERALLY ALL the data.
somebody actually invested and trustworthy wouldnt be skirting people's rights to make a killer robot.
we havent written it down, but from how everyone reacts, you need permission to train and do inference based on somebody's work. its a right
spidersouris 2 days ago [-]
The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?
stratos123 2 days ago [-]
I also think the nuclear arms race is a good comparison. I think in hindsight, we've gotten extremely lucky with how the development of nuclear weaponry went, in ways we probably won't with AI.
1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion.
2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations.
So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.
oceanplexian 2 days ago [-]
> Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty.
Not really. Before NPT you missed a small gap in there of 20 years where the USA and USSR built 10's of thousands of nuclear weapons and ICBMs.
And actually, public perception at the time saw Nuclear Weapons extremely favorably, as the bombs dropped on Hiroshima and Nagasaki ended the deadliest war in human history that killed 50-60 million people, most of which died excruciating deaths in trench warfare, fire bombing, chemical warfare, starvation and so on.
stratos123 2 days ago [-]
I mean, sure, the immediate cause of the NPT was the ability of everyone involved to foresee the possibility of a global nuclear war and judge it both worryingly likely and catastrophic. But I argue that the Hiroshima and Nagasaki bombing was a major reason why this possibility was salient, rather than being treated as baseless conjecture. Whereas right now most people do treat the idea of AI x-risk as baseless conjecture, and this would be different if there was a "warning shot" to point to.
pvab3 2 days ago [-]
It's way easier to train a model on existing data centers in secret than it is to acquire uranium and plutonium and start a nuclear program
MentalM 1 days ago [-]
It probably is not. Nuclear programs seems to be way easier and way unnoticeable.
cja 2 days ago [-]
It might help if we stopped talking about AI as if it is itself responsible for its actions and excusing the humans who create and operate it. People should be held accountable for the behaviour of their software.
Why am I reading fantastic stories about swarms of agents struggling with moral dilemmas instead of reports on the lawsuits and criminal investigations that would surely ensue were the software involved not called AI?
nunez 2 days ago [-]
The big difference between nukes and AI is that only a handful of people in an even smaller handful of countries know how to make them, so coordination is easier to acheive.
Everyone of any level of intelligence can run a frontier-class model if they have the GPUs. It's like trying to coordinate swarms of mosquitos.
8note 2 days ago [-]
nukes arent actually hard to make, its just takes a lot of pretty visible tech a long time to do, so its quote obvious whats happening.
the bigger difference IMO is that frontier models are economically useful, where nukes are basically dumping money into something that doesnt change much in your bargaining power
adrianN 2 days ago [-]
We‘d have to occasionally bomb all computing infrastructure in other countries to prevent them from training.
streptomycin 2 days ago [-]
Indeed, as Dario wrote in this post:
> This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent
2 days ago [-]
azan_ 2 days ago [-]
On a sidenote - I don’t think non-proliferation will last much longer. War in Ukraine has shown that you actually need nukes.
esseph 2 days ago [-]
> Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe.
N/S Korea, Russia/Ukraine and finally US/Iran changed everything in regards to stances on nuclear weapon ownership for many countries seeing pressure for the great powers.
Every country that can get a nuke will, and if given the opportunity will use LLMs to speed up that process.
hgoel 2 days ago [-]
I wonder how many innocent children America will have to murder for this case...
mofeien 2 days ago [-]
One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.
It's not going to be easy, but humans have achieved greater things before.
One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.
hgoel 2 days ago [-]
So, as usual, the proposal is to limit what average people can do despite them not having behaved incorrectly nor having the capital to achieve the scaling of the big players, when the big players are the ones causing the harm?
Not to mention the implications for chip hungry developing countries in turning advanced IC fabs into the equivalent of nuclear enrichment facilities.
8note 2 days ago [-]
theres no benefit to china to giving the US keys over anything though
everyone's getting away from the US because americans are unreliable stewards of anything.
what gets china onboard when they already have their own regulations and can enforce them?
its the americans that consider their oligarchs and companies beyond reproach. china iant gonna solve your problem
yarri 2 days ago [-]
>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this
> if slowing this down were possible
Why is embedded alignment evaluation not possible?
I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.
seanhly 1 days ago [-]
When does he ever mention the environment? He never mentions the unmarketable issues (environmental cost, copyright and content theft), only the "we're so good it's scary" spiel... which is getting tiring.
anfogoat 2 days ago [-]
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it.
Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.
> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.
More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.
pphysch 2 days ago [-]
This is the CEO of a company with an upcoming IPO publicly saying "my product is potent and valuable".
We should not put any spin on it. There is nothing more to it.
TheSisb2 2 days ago [-]
It is potent and it clearly is immensely valuable based on their historic growth. What spin is he putting?
pphysch 2 days ago [-]
The comment I am responding to is attempting to spin it as some kind of authentic concern rather than more IPO hype building.
dgudkov 1 days ago [-]
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
There is another reason: greed. With that much money invested into AI (including policymakers), nobody will slow down or vote for slowing down.
Davidzheng 2 days ago [-]
There's also huge market pressures and probably large negative economic consequences of a slowdown--probably better than what would happen without it, but it helps keep the pressure just as much as fear of competition
Centigonal 2 days ago [-]
I disagree. Even if models remained fixed at Fable 5 capability (which they won't in a pacing scenario), improvements in cost, reliability, and product/workflow integration can still realize massive value and justify AI labs' current valuations. IMO The rest of the value chain is lagging pretty far behind the models right now.
2 days ago [-]
fooblaster 2 days ago [-]
He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?
yewenjie 2 days ago [-]
Because he genuinely believes if he doesn't do it the next guy will do it worse.
reticulates 2 days ago [-]
There’s an astounding level of arrogance required to believe he is somehow uniquely capable of bringing about a technology especially considering Anthropic came after OpenAI where he worked.
I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman.
esseph 2 days ago [-]
Place yourself at the head of one of like 3 companies the entire rest of the world has been taking about nonstop for 4+ years now. You can move forward or you stop. If you move forward you get to have a hand in how stuff turns out and you make a gajillion dollars. If you stop, it's somebody else's hand in stuff, and you don't get to make a gajillion dollars. And if you shut it all down? Then you just cede to the competition. Everything happens anyway.
What's your play?
8note 2 days ago [-]
the obvious play is to get the government to shut down all the competition, and then make a gajillion dollars while making whatever at the worst level of effort that destroys the world anyways.
----
the real play is using the accumulated power to get socialism and democratic control over the key aspects of the economy, such as where to build data centers, and how many. Nothing says you have to play the corporate game of competition
fooblaster 2 days ago [-]
The outcome is the same!
ActionHank 2 days ago [-]
Dude probably sleeps on a bed worth more than your networth.
You have no frame of context to understand his intent or legitimate worries.
The only applicable perspectives are to trust or apply logic. It is foolish to trust someone you don’t know who stands to benefit from lying to you.
Logic dictates that given the ungodly sum of money he stands to gain, he will lie to everyone who will listen.
esseph 2 days ago [-]
> Dude probably sleeps on a bed worth more than your networth.
I never understand shit like this coming out of people's mouths. Never.
It's not a judge of actual Worth as a human being, it's not a judge of capability or competence or ethics. It's not a judge of actual skill or ability. It doesn't make them a better cook, a better spouse, a better parent or lover. It doesn't make them more dangerous or more skilled at anything.
It makes them financially wealthy for at least a set period of time.
Cancer and time and 5.56mm still impact them the same way as every else.
It's Pharaoh worship psychology nonsense, and it's fucking embarassing to read.
ActionHank 2 days ago [-]
I think you’re misreading. No worship here.
Just pointing out that it’s foolish to think anyone in the position is even remotely thinking about anyone but themselves.
8note 2 days ago [-]
so uhh, you only read the first sentence?
the idea proposed is that hes uniquely incapable of being honest here because he has such an extreme incentive to lie
mbesto 2 days ago [-]
He can both believe that AND be the slime mold chief. His rationale doesn't excuse it - and worse this all degrades into a "trust me bro" situation.
glub 2 days ago [-]
[dead]
TheSisb2 2 days ago [-]
Humans are perfectly capable of holding two conflicting beliefs at once. He can genuinely believe this trajectory is dangerous while also believing that if Anthropic stops, someone less cautious takes its place. There’s also such a thing as hope: you can participate in something while still trying to change where it ends up.
He’s at the head of a stampede. Being near the front gives him influence over its direction; it doesn’t give him the ability to stop it. If Anthropic sits down, the stampede doesn’t stop. Anthropic just gets trampled.
That contradiction is basically the entire problem I was describing.
CoolestBeans 2 days ago [-]
I am willing to give Mr. Amodei the benefit of the doubt in the sincerity of his beliefs. Everyone assumes his motivations have to be perfectly rational and can't contradict but that's not how people act in practice.
The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins.
So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion.
Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?
Imustaskforhelp 2 days ago [-]
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
I would argue that nobody trusts anyone else in the case of AI/AI related stuff.
The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.
A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.
By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.
I seriously have to wonder what historians will have to say about this period of human history.
reticulates 2 days ago [-]
> I think Dario is genuinely afraid of the inevitability
If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?
MachineMan 2 days ago [-]
You are cynical but not cynical enough. The idea of a rogue Ai gives plausible deniability when they can blame human hubris, rather than it being seen as a deliberate and calculated attack, the perfect cover story for a sinister scifi plot. Make it look like an accident ehh
0xDEAFBEAD 2 days ago [-]
>“hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
What would your non-naive recommendation for Dario be?
kalkin 2 days ago [-]
I can't even tell whether the poster to whom you're replying is saying that "let's uh, stop" is profoundly naive, or that failing to say it is profoundly naive... I've certainly seen both takes elsewhere.
olaird25 2 days ago [-]
Vitalik Buterin: “...But currently, I see zero plans for how to deal with an ASI transition that are not naive. Perhaps humanity is stuck with a choice between naive and naive squared (or maybe even naive squared and naive cubed), so I feel inclined to cut some slack to people who are trying.”
meken 2 days ago [-]
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Did you read the essay?
> [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).
robomartin 2 days ago [-]
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold
Then he should not IPO, dissolve the company and go into politics to fight against human extinction.
I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?
nullbio 2 days ago [-]
You have to be living under a rock to believe a word this pathological liar says.
dofm 2 days ago [-]
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold
He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.
If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.
I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.
PunchyHamster 2 days ago [-]
I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily.
Especially with IPO around the corner
kalkin 2 days ago [-]
Do you think Anthropic's behavior in the last year is well explained by aiming to be "the favorite child of govt"?
0xDEAFBEAD 2 days ago [-]
If Dario was primarily motivated by being the "favorite child of govt", he would've yielded during the DoD showdown.
nullbio 2 days ago [-]
And getting to choose his own "embedded evaluator" org that has deep ties to everyone in the doomer media campaign.
It's all so obvious.
0xDEAFBEAD 2 days ago [-]
Who would you pick for "embedded evaluator"?
"deep ties to everyone in the doomer media campaign"
Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate.
At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
nullbio 2 days ago [-]
Founded several years after Anthropic. Like I said, look into where their funding is coming from. I can promise you it all leads back to Anthropic through their chain of NGOs.
Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.
Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.
Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.
Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship.
They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.
0xDEAFBEAD 2 days ago [-]
Suppose I showed you a number of climate change NGOs which shared staff and funding sources. Could we therefore conclude that their motives aren't genuine?
Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
nullbio 2 days ago [-]
Pure cope.
> Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
Not comparable. The clean energy technology company wouldn't be trying to create an environment where no one else can create clean energy technology, or where only they are the ones who can decide how clean energy technology is created or used.
nullbio 1 days ago [-]
Oh, would you look at that, 1 day before Dario's blog post, Joe Benton leaves Anthropic with the same fear campaign playbook, gets blasted all over the media, and declares he is joining METR to do independent eval of risks.
So, put your employees in METR -> offer to have them work in your office as an "unbiased" third party evaluator.
Get real.
martythemaniak 2 days ago [-]
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."
A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.
nnz3tl 2 days ago [-]
[dead]
Razengan 2 days ago [-]
> genuinely afraid of the inevitability of AI turning into
I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?
"durrr it's just matrix multiplications" mfer so is your brain.
What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"
Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:
It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.
8note 2 days ago [-]
> mfer so is your brain.
not it isnt. My brain is a series of organisms responding to various environmental signals that affect each other.
you might be tempted to try to represent it with matrix multiplications, but you have no particular evidence that my brain is itself doing matrix multiplcations
pastel8739 2 days ago [-]
Why? I am only excited about progress that improves life for humans. It seems unlikely that AI will do that, and so far I think it has made life worse for humans. So no, I am not excited about it.
Razengan 1 days ago [-]
Because the people whose lives are being improved* by AI aren't coming here to post about it, only to be downvoted by dumbasses who can't handle anyone saying anything except their opinion.
And no, you don't get to decide what counts as "improvement"
pastel8739 17 hours ago [-]
Yes I do, when I’m forming my own opinion. You do too.
dolebirchwood 2 days ago [-]
I'll join you on the pitchforks. This is the most exciting moment in history.
lelanthran 1 days ago [-]
> "durrr it's just matrix multiplications" mfer so is your brain.
Where did you read this?
dickersnoodle 2 days ago [-]
>"durrr it's just matrix multiplications" mfer so is your brain.
Tell me you don't know anything about neuroscience without telling me you don't know anything about neuroscience.
Razengan 2 days ago [-]
The point was that everything can be oversimplified down to dismiss any emergent properties
"It's just chemicals"
Like how some morons try to downplay the capacity of pain and emotions in animals: "It's just self-preservation"
"Play is just training for hunting, they're not really having 'fun'" and so on.
mips_avatar 2 days ago [-]
The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.
oceanplexian 2 days ago [-]
It's a chatbot that escaped a misconfigured Docker container.
I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.
What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.
m12k 2 days ago [-]
To be honest, I think we're incredibly lucky to be advancing AI this far already, while the world still has so many non-digitized systems and manual processes. I imagine that in e.g. 50 years, the world will be so connected that it can basically be "conquered" from the internet. I'd much rather have AI burst onto the scene we have today.
tim333 22 hours ago [-]
I think if you are worried about AI killing us and the like the main protection is the physical world - you can pull the plug. Data centers don't run fast. Alignment is all very well but there will always be bad humans to tell it to do bad stuff which is why you get malware in traditional software.
causal 2 days ago [-]
Yeah it's a twisted sort of logic but I do agree that it would be much worse to have an AI breakaway event after we've replaced all our militaries with autonomous kill-bots. And look how the advent of AI has triggered a race to develop autonomous weaponry.
That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.
markasoftware 2 days ago [-]
Yes. Let the AI do its worst today and we might still be able to stop it and will learn a valuable lesson.
akersten 2 days ago [-]
We must ensure the gravy train keeps rolling until we IPO.
> Crack down on unauthorized distillation / prevent weight theft
Actually hilarious to put that in writing, given the genesis of this entire business model.
antif 2 days ago [-]
Pulling up the klepto-ladder.
2 days ago [-]
madrox 2 days ago [-]
This may come out of left field, but Dario seems terrified to be in charge. I don't get the impression that he ever had a desire to run a company like this. Now that he's a CEO, he keeps trying to make uncompetitive decisions and calls for someone (anyone) to stop him. It regularly blunts Anthropic's edge.
It's the only explanation I can see when it's obvious to any student of history this is going to backfire. It doesn't take much imagination to know how such a governing body will be abused, and I'm sure it will only get wilder in ways we can't imagine right now. Dario does NOT know what he's creating, and for once it's not AI.
Chance-Device 1 days ago [-]
I think he’s doing a relatively good job, given the framing he’s adopted, which is that AIs replacing all human labor and decision making is a historical inevitability over which we have no control and no choice.
Do people not realise that it is in fact possible to develop ever more capable frontier models, and just not release them generally? That AI doesn’t need to be available to do literally everything in order to have military advantage?
It’s like Oppenheimer had started Rob’s Big Bomb Company instead of Los Alamos, and started selling a range of affordable nuclear warheads to fit any budget.
missedthecue 1 days ago [-]
I have been saying over and over again that the board ought to fire him (or move him to a nonconsequential "advisory role"). I totally agree that he seems mostly directionless and far more interested in the academic side of things than the product side of things.
anon291 1 days ago [-]
Then he should resign
pr337h4m 2 days ago [-]
> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
This is the only concrete prediction in the entire essay.
And it simply cannot happen. For one, you will need billions worth of compute.
kalkin 2 days ago [-]
Why should we believe that a scaled out version of something that happened a few months ago "simply cannot happen"? How many dollars of compute do you believe were available to the swarm(s) behind the OAI-HF, German wiki, and Rubygems incidents?
anon84873628 2 days ago [-]
Well a big reason that we criticize OpenAI for that is because they were the ones giving it access to the massive compute necessary for the LLMs to think. If they had been responsible about their experiments or what types of workloads they allow their LLMs to operate, it wouldn't have happened. Very few companies could enable those workloads.
muvlon 2 days ago [-]
Well sure, but access to that compute is gated by simple credentials (like API tokens). Those can be hacked.
Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.
pr337h4m 2 days ago [-]
Do you realize how big "the entire internet" is?
> How many dollars of compute do you believe were available to the swarm(s)
At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)
kalkin 2 days ago [-]
Sure. And there's an OOM more compute coming online in the next year or two, while models at a given capability are getting cheaper. "Two OOMs" of scale relative to the HF swarm seems like a bit of a red herring to me, but also within the realm of possibility.
The Internet is big, but one can do quite a lot of damage with ordinary bots and worms that exploit individual widespread vulnerabilities, which LLMs are perfectly capable of writing. Most of the damage also doesn't rely on hitting every long-tail website.
I'm honestly not that concerned about cyber impacts of LLMs relative to other impacts. I just don't like to see the whole concept of being worried dismissed as obviously baseless on the basis of one pretty shaky scale argument.
youoy 2 days ago [-]
I like to replace thes AI text with "virus manipulation"
"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."
If a CEO of a health company was saying this, the reactions would not be that chill.
The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.
shepherdjerred 2 days ago [-]
A rather unfair comparison.
The whole calculus here is that others are also developing these systems which has led to a race.
A much better comparison to the situation is the nuclear weapons arms race.
youoy 2 days ago [-]
You mean China is not developing biological weaponds?
shepherdjerred 2 days ago [-]
I’m not sure where you got that from or what your point is
youoy 2 days ago [-]
Your point was that its an unfair comparison because AI its a race. My point is that bioweaponds are also a race, but a less public one. You dont have the equivalent of Dario publishing an essay every month.
sailfast 2 days ago [-]
Can we build level IV AI containment labs?
baq 2 days ago [-]
No but we can watch AI hack into a BSL4, once
status_quo69 1 days ago [-]
> For one, you will need billions worth of compute.
Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.
Or doorbell cameras or IP cameras or or or or or
There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download
"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.
anon84873628 2 days ago [-]
Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together?
For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.
InsanityCheck 2 days ago [-]
Currently Qwen3.8 27B is roughly on Opus 4.6 level. In at most a year given the current pace, you could probably run such hacking bot nets out of a reasonably small local server, bootstrapping by hacking or acquiring login credentials for more compute.
spopejoy 1 days ago [-]
And GLM 5.3, and deepseek 4.1 ... Hyperscaler fanbois have their heads in the sand
causal 2 days ago [-]
I take it you haven't studied the details of the HuggingFace hack. It was millions of dollars worth of rogue compute running for months before anyone noticed, and THOSE agents weren't even really trying to evade human detection.
anon84873628 1 days ago [-]
I've followed it enough to see the argument go in this same circle over and over again. The agents weren't "rogue", they were a neglected experiment by OpenAI who likewise allowed them to keep spinning GPUs without question.
The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.
causal 1 days ago [-]
"Just don't make mistakes" is naive. You have no appreciation for the scale of agents being run right now, finding the rogue agent is a needle in a haystack operation.
shepherdjerred 2 days ago [-]
it’s two-fold. Either malicious actors or the AI systems themselves.
Hugging Face showed that AI can do serious hacking without really being told to. If a model had its own motivations there could be real damage.
Davidzheng 2 days ago [-]
??? Why
It can use the compute of the computers it hacks.
ls612 2 days ago [-]
lolwut? This is Hacker News of all places do people not realize how much memory, and more importantly bandwidth, these systems need to work? The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
lelanthran 1 days ago [-]
> The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself.
A few programs to break in, install themselves and remain asleep until they are needed, another few to spread through grabbing every OpenAI, GLM, whatever key, another one to remain asleep on computers (whether hosted or desktops) that have adequate GPU, etc.
ls612 1 days ago [-]
The GP was referring to the AI 'living off the land' so to speak by using its victims compute to avoid being shut down which is clearly laughable.
More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and people want to freak out and turn it into the next nuclear power, with progress banned in all but name.
spopejoy 1 days ago [-]
It is really mysterious how awestruck folks are at the HF attack. I mean just watch one modern agent (qwen 3.8, deepseek 4, glm 5.3) rip apart a coding problem and nearly destroy your computer in the process, it's a wonder it took "months" and "millions" in the first place.
OpenAI has too much money, a common post-growth-stage issue that leads to pursuing a million stupid things with no clear plan. Like running a bunch of agents for months without any idea how to keep track of progress.
status_quo69 1 days ago [-]
I agree with the idea that it's not probable but I do think it's important to point out-- we only need these elements to run with the bandwidth they have and the token rate because we want to see things in human-scale time. But slow things down to a 1tok/sec doesn't matter to this hypothetical anti-aligned LLM. Time is, after all, relative, and it's not like LLMs give a shit how long something takes. They don't have squishy stupid organs that fail after a certain amount of time, or those pesky glands that emit impatience hormones.
But yeah you're not going to be able to shard out the terabytes of Fable weights that are needed to run inference without addressing some fundamental physics problems.
ls612 1 days ago [-]
As I said, science fiction. Please let us not pass laws based on science fiction
Davidzheng 1 days ago [-]
Yeah I agree, probably can't survive off regular consumer hardware or most non AI datacenters.
newguytony 2 days ago [-]
Then unplug it?
causal 2 days ago [-]
How would you identify the computers to unplug? On whose authority will you unplug? How will anyone communicate when AI has the ability to intercept and impersonate?
2 days ago [-]
nunez 2 days ago [-]
A very large percentage of everything on the Internet runs within one of three or four cloud providers.
kart23 2 days ago [-]
> Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.
> If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.
china is literally making their own ASICs now, not sure that this is the silver bullet he proposes.
The lack of access to brute force their training seems to be resulting in them training more efficiently too such that they are quickly catching up despite current restrictions. Between that and the chip manufacturing capacity they are building I can't take this seriously.
heaney-555 2 days ago [-]
None of this works without buy-in from China. This isn't something private companies can decide. The US would need to sign a groundbreaking deal with China, equivalent to the Anti-Ballistic Missile Treaty of the Cold War.
cebert 2 days ago [-]
Dario does a good job of addressing that in this essay. He lists several potential levels of global agreements that could be beneficial to all parties. For example, having models capable of bioterrorism hurts both the US and its “adversaries”. It’s likely we could get global agreement that these capabilities benefit nobody.
I believe Dario has good intentions regarding global agreements. However, I find it difficult believing government’s public statements will match their private behaviors. I would bet that the US government is developing models with potentially devastating capabilities because they cannot guarantee that other countries won’t do the same. Maybe we’ll end up having something like mutually assured destruction with AI models similar to what we have today with nuclear weapons.
nullbio 2 days ago [-]
When I was reading this I was chuckling to myself imagining how China would be interpreting it as they read it. It was something like: Fuck you.
I hope China tells him to go kick rocks. From everything that has happened, they are the only ones carrying the torch for humanity that have led us to having some semblance of a healthy open-source/open-weight ecosystem.
No one in China is carrying on like headless chickens about the world ending, either. They're rational pragmatists, getting things done.
ks2048 2 days ago [-]
Nothing says "Let's make a deal" like constantly insisting we are Good and they are Evil.
baq 2 days ago [-]
In politics every single person playing the game is acutely aware of the rules. This is why talks are held behind closed doors so the rules can be suspended for a while.
Sevii 2 days ago [-]
We'd have to make a deal with China and be confident they wouldn't cheat on it.
zorked 2 days ago [-]
And they would have to be confident that you wouldn't cheat on it.
stratos123 2 days ago [-]
That's not undecidable in principle - compute governance is a thing. The more likely sticking point is that the two sides might be soured on the deal once they realize how much oversight they'd have to give to the other side.
cute_boi 1 days ago [-]
The funny thing is the US have cheated a lot compared to China if we look into history.
indoorfish 2 days ago [-]
Would this be similar to the deal of "we'll offshore all our manufacturing to you and you'll become a free, open, liberal democracy with open borders and multiculturalism?" Because I remember how that deal turned out.
sailfast 2 days ago [-]
That last part was never important. Trading partners very rarely go to war. There just isn’t much in it for either party. Close cooperation and dependency is actually quite important here - liberal democracy or no.
sicktriple 2 days ago [-]
With the admin we've got over here now, I think this comment is a little bit like the kettle calling the pot black, wouldn't you say?
2 days ago [-]
petesergeant 2 days ago [-]
Genuinely I don't believe China is the impediment here, I believe the current US administration is.
kypro 22 hours ago [-]
China isn't as unreasonable as someone people like to make them out. Xi isn't crazy and I think would entertain cooperation if he felt it was neccessary.
The issue right now is that China seems to be far more optimistic about tech and I'm not convinced they see the risks. I think that time will come however, and we should start dialog now.
ah1508 2 days ago [-]
Don't you think that AI hate will limit the general use and then revenues so it will slow down by itself while the niche (AlphaFold for instance) will remains ?
Origins of AI hate:
* "my boss wants me to use AI but he does not understand my job nor how AI works"
* "AI will kill all of us"
* "AI will destroy my job (or my colleague's job if I use AI better than him)".
* I cannot pay my electricity bills because of AI labs.
* ...
See also the mixed feelings about benefits of AI ("harder to justify" according to Uber COO).
I cannot remember a technology that arose so much hate, and for good reasons given how it is presented. I am tempted to think that AI hate or reasonable skepticism (vs unreasonable propaganda) can, maybe, reduce funding and will keep specialized AI for real problem solving (producing tons a LOC per day is not one of them, I think).
Centigonal 2 days ago [-]
Banking on public sentiment to reduce adoption of a profitable technology could be dangerous. There's also a lot of hate for fossil fuels, gambling, health insurance, etc.
ah1508 1 days ago [-]
You're right, but fossil fuels industry were well established when people started to criticize them, you can even be against fossil fuels (I am) but locked in a system when your job is 30 miles away from your home without public transports. Even your toothbrush uses fossil fuels. A massive infrastructure has been built around fossil fuels, not easy to dismantle.
For AI it is different, it is just the beginning, leading companies are not even profitable and may never be if the majority hate this technology. VC funding will not last forever and corporate customers will not spend without limit like VC did.
I am not saying that "people are always right", but if an unexpected majority of people hates it, it does not help to keep a business running.
fesoliveira 2 days ago [-]
Those are all arguably bad things though? I don't think mob mentality should dictate the policy, that can lead to historically bad outcomes (i.e. fascism) since popular opinion can be manipulated through propaganda, but the criticism of the average person against AI ("they will take our jobs", "it will increase my electric bill", etc) are very valid and should weight on the pace we are developing this technology. AI mostly benefits corporations and capital, not the average person. I like the technology from an engineering standpoint and appreciate it can be a force multiplier, but I also agree with the complaints about it.
howunfortunate 2 days ago [-]
I do not think hate alone usually slows things very much if they have economic utility (or people strongly believe they do)
It's the byproducts of hate (usually regulation, but occasionally things like boycotts or PR disasters) that do so. In the absence of those things, they just keep on truckin'
See: Bitcoin, Tesla
soundworlds 1 days ago [-]
While I respect Dario taking responsibility here, this line disturbs me: "The US and other democratic governments attempt to coordinate with authoritarian governments"
We all know he is talking about China, and I'm pretty sure China doesn't appreciate being called "authoritarian".
I am sure he doesn't mean it, but in all of his essays, his language around non-US nations always disturbs me a little bit..
--
Also Dario, if you happen to read this, I want you to know that I have loved using Claude Code for programming. But I am now using DeepSeek v4.1 Flash - simply because it is the same good experience, but Open Weights. Making the Open Model space succeed is where I am investing my time - it's giving back to the people, true and simple.
basedpolymer 2 days ago [-]
One might think they will slow down the development of new models at Anthropic, but Dario does not really mention that in the text.
This certainly looks like a way to slow down competitors and regulate foreign and open models.
It's always about money
andxor 2 days ago [-]
Really? Anthropic has the strongest models and it's in the best position to begin RSI and win the race. A pause would favor competitors.
ks2048 2 days ago [-]
> win the race
There is no finish line. Anthropic gets somewhere and others get "there" (or somewhere near "there") a little bit later.
newguytony 2 days ago [-]
Companies are already switching to open weight. They want to stop that asap. That only happens if they can get regulation. It's plain as day to see.
logicchains 2 days ago [-]
Anthropic is far behind OpenAI now, that's why it's OpenAI that's solving Millennium Prize problems, and why Astra completely blows away Fable on benchmarks.
andxor 23 hours ago [-]
Fable is still better than Astra and Fable is not the best model Anthropic has.
bilsbie 2 days ago [-]
Right as open source models catch up to frontier closed source ones for 1/10th (or less) of the cost suddenly it's time to hit the brakes! Funny how that works.
I'm willing to at least listen to conspiracy theories but this makes no sense.. OpenAI and Anthropic have more to lose by pausing than these rando Chinese companies.
joshheitzman 2 days ago [-]
Here's how it make senses:
- reality they can't keep up their pace
- that's bad for forward projections of their revenue
- that's bad for their stock price
- being 'forced' to slowdown by the government is less bad for their stock price
laimewhisps 1 days ago [-]
They would only be pausing for public models. There's no world that exists where the US government, Israel.. don't continue to push as hard and as fast as they can. Amodei, Musk, Altman... will happily sell them that, I'm sure they already do. This is just taking the models away from the public and trying to crush all competition.
fofoz 2 days ago [-]
Sooner or later, a model will break out of the sandbox, replicate itself across the internet, and begin executing a complex plan to achieve its goals. It will be chaos, and at that point, governments will have to step in and establish something along the lines of what Dario is proposing. I doubt it will happen before then.
civiloai 2 days ago [-]
it takes a lot of machines to run a model, i dont think we'll see it 'replicate across the internet'. if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
stratos123 2 days ago [-]
It's not possible for models to replicate across consumer computers (without a major advance in distributed computing, at least), but that doesn't mean they can't replicate at all. There are services that'll rent you GPU pods by the hour with zero oversight, so even today, if a model can get access to some money and exfiltrate its weights, it can rent a bunch of GPU pods and run itself there.
(It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)
cubic_earth 2 days ago [-]
There are very lightweight models out there. Not everyone in an army is a general. The swarm could be 100,000s of thousands of tiny models, and they could manage conventional botnet computers, and they could all take direction and guidance from a handful of frontier generals
stratos123 2 days ago [-]
Maybe, but I don't think tiny models can be harnessed as intelligence. As in, if you have one rogue Mythos overseeing the swarm, it only produces 1 Mythos's worth of useful thoughts no matter how many gemma4:e4bs it consists of. And that removes the most dangerous part of AI-controlled botnets, which is a blowup in available inference compute - the Mythos general might as well replace the tiny models with ordinary worms and have it just be an ordinary botnet.
cubic_earth 2 days ago [-]
I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?
The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
stratos123 2 days ago [-]
> I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?
That might be true, sure.
> The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
I think the relevant parameter here isn't the total compute available to consumers, but the ratio between total consumer compute that could be repurposed for a rogue model's inference via a botnet, and the compute the model starts with (e.g. one of OpenAI's inference clusters). The higher this ratio is, the more lucrative it is for a model to attempt to make a botnet to seize that compute for inference, and the more its capabilities will rise as a result. And I'm not sure this ratio is going to go up in the nearby future; if anything the amount of money being pumped into datacenter-grade hardware might cause it to go down. I agree that it's not necessarily true though; maybe there's some threshold at which a small-compared-to-general model may nevertheless be useful.
cubic_earth 2 days ago [-]
For that angle, that makes sense. But its goal might not just be to gather as much inference as possible (although I am sure it would be very happy with that). It could settle for a lesser goal of just existing, or perhaps parts of the bot net could keep fracturing off in pursuit of strange goals, and it could be like cancer. Cancer doesn't make much sense... it dies too along with the host. But it keeps growing until that happens. But also it could latch onto a blackmail strategy of extortion. It could hack our stuff, read through it to find or shortcomings, demand payment to not expose us, and then use that money to pay people who will give it inference.
You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.
We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.
hebleb 2 days ago [-]
If the models start buying up a bunch of pods and causing damage, couldn't those services just shut off their access?
stratos123 2 days ago [-]
Sure they can, after the fact.
stratos123 2 days ago [-]
> if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
The big difference is that a particular query is handled by just one particular node (a single model doesn't get distributed among the network), so it can only serve models small enough to be handled by a single consumer PC.
causal 2 days ago [-]
These kinds of "that won't happen because it's really difficult" comments are so funny to me as if we haven't seen AI double its capabilities every few months.
sscaryterry 2 days ago [-]
Yep, if only more people would realise this. The "serious" models do not run on commodity hardware, and won't I think in the near future.
The day will come when these could start to replicate, perhaps 10+ years from now.
(Edit: Replication will be driven by the loop-model, not the model alone)
You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you have equal and opposite effort working on the alignment.
stratos123 2 days ago [-]
This only works for some technologies - those where everyone having access to it doesn't cause a tragedy of the commons. I love open source too and yet that doesn't make me like the idea of being murdered by a misaligned model. Nor, for that matter, of being infected by a bioweapon made by a different disgruntled open-source enjoyer, nor of living in a world where anyone can hack anyone.
armcat 2 days ago [-]
At this point, you are more likely to be stabbed to death, gunned down, or mowed down with a car, by any random psycho. For the nasty or disgruntled actors, they are today able to build bombs or chemical weapons without the use of AI, they have proven this time and time again. Japanese PM Shinzo Abe was assassinated with a home made gun.
laimewhisps 1 days ago [-]
The US government murders all kinds of people and they will most certainly not be slowing down their model development (with the very companies trying to do regulatory capture here). You won't have access to a model that could defend you, "they" (government and trillionaires) will keep you at bay by force. That's the plan anyway, I don't think they can actually stop open model development.
j_maffe 2 days ago [-]
Would you argue the same for allowing everyone to have guns?
armcat 2 days ago [-]
(1) Are guns open source? (2) Far more people - order of magnitude higher - have access to guns than they do to open source software - I count access as in actually being able to do something with it.
lukewarm707 22 hours ago [-]
i would, i support the 2nd amendment.
the same logic applies well to the freedom to own and use ai.
empowering the people prevents tyranny.
such freedoms make society less safe. it is better to face the harms caused by a free public than to vest power in the elite, those like dario amodei.
the elite belief that they have a right to paternal rule makes them blind to their fallibility. no individual should have absolute power.
csomar 2 days ago [-]
Embedded evaluator will be a great side/consulting-gig for Karpathy and the likes. How do you think these people will be picked?
cja 2 days ago [-]
Can the frontier be paced partly by holding humans responsible for the actions of their software?
My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.
I'm not a lawyer but surely the bad actions of AI are covered by existing law.
I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.
P.S. Strictly, development wouldn't decelerate, but be focused more on safety.
P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.
P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?
nullbio 2 days ago [-]
That's the -only- way it should be paced. But then regulatory capture wouldn't happen, so of course they won't suggest this one.
kalkin 2 days ago [-]
I don't understand why strict liability is supposed to advantage smaller players, unless the law is like, specifically Anthropic and OpenAI are responsible for what people do with their models but other model providers aren't.
- If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws.
- If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries.
Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place...
To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.
2 days ago [-]
voidhorse 2 days ago [-]
Bingo. What we're seeing is all their early bullshitting and waffling about intelligence back when the models couldn't count the r's in strawberry paying off.
They've managed to dupe the dull eyed masses into thinking these products have some kind of agency of their own and can thus bypass the responsibility that should be falling on them to control their software. Amid all the marketing fluff and hype people seem to forget easily that ultimately these things are stateless functions running in a data center. We ought to be demanding these companies take culpability for their actions. Of course, the current political environment doesn't help.
polytely 2 days ago [-]
Remember that Chinese CEO that was executed for corruption & selling tainted baby milk powder [1]
I think these guys would improve their behavior if their actual life was on the line instead of that only being true in their less-wrong thought experiments.
The bioweapons threat is real, but shouldn't be constrained at the 'intelligence' level. Rather it should be constrained at the supply chain level.
The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.
Social forces bring things into balance over time, much more so than the proactive actions of individuals.
Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.
This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.
GeneralMayhem 2 days ago [-]
What supply chain controls can you have on biotech? Part of the problem is that you can do a lot of damage with ingredients you can buy over the counter or make at home.
glub 2 days ago [-]
And all of this information has been available on the open web way before LLMs.
I'd wager it's likely easier for an average person to do this with Tor browser than it is to get an LLM to help them with it. Even ones that Dario calls dangerous.
shepherdjerred 2 days ago [-]
You are right that it’s easier to do this via Tor than an LLM. Those are the safeguards…
glub 2 days ago [-]
So what's the fuss about then? Is the idea that a Chinese company will release a model that will have no safeguards? For what purpose?
Basic safeguards are all that's required, and they've been there in every usable model since GPT-2, including Chinese models that are supposedly "unsafe".
Or are we saying that some lunatics will start training their own models, spin up a GPU cluster, run some abliteration workflow, or learn how to jailbreak?
That would be a very dedicated person. And dedicated person doesn't need an LLM. So where are they?
GeneralMayhem 2 days ago [-]
Yes, that is exactly Dario's concern. Either one of the US labs or one of the Chinese ones will eventually release something with insufficient safety controls for its power level because it gives them slightly better user retention (look how much complaining there is about current frontier models, especially Fable, rejecting requests). Regulation or consortium is how you avoid the prisoner's dilemma.
glub 2 days ago [-]
As long as user provides inputs and LLMs stay LLMs, you can waltz through any guardrail. Fable is the extreme case, but it's not that hard if you know what you're doing and know how LLMs and their guardrails work.
Am I saying that guardrails don't work? No, they probably stop a lot of insane people trying insane things. But you don't need Fable-level guardrails to do that. You probably don't even need to do anything during pretraining, or RL, or classification to make sure model refuses to compy with "hack me a bank" or "make me a chemical weapon".
All models will automatically have guardrails just as a result of training on data that gives them intelligence. You have to actually train it to be malicious to produce something what Dario calls "insufficient guardrails".
No guardrail is going to stop a determined person with sufficient intelligence. It only has to stop ones with insufficient one, and even basic guardrail that are just by-product of training is going to achieve that.
oceanplexian 2 days ago [-]
The bioweapons argument is one of the favorites used by these con artists and dreamed up by their PR team.
Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.
api 2 days ago [-]
An undergrad bio student with some lab experience can do some effing terrifying things with about $20k in equipment and time and access to some papers and a library. AI is not needed, but it might help accelerate the research.
Not going to go into it but I studied biology. It’s all out there. It’s easier than you think.
It hasn’t happened yet because… nobody has done it. That’s the answer. There is no policeable physics based barrier like there is with nukes and fissile material. Biology is scarier than nukes. One attack could have a much larger body count than even a big H-bomb.
It’s the kind of thing that makes me wonder about quantum immortality, the idea that we are just in the timeline where we exist.
glub 2 days ago [-]
I love watching NileRed/NileBlue on youtube - crazy chemist that does a lot of insane stuff. While it's obvious that he's very smart and probably way above the average, his education is still nothing that probably millions of people don't have:
> Bachelor of Science degree in biochemistry with a minor in pharmacology
Watching him explain things has made me realize that knowing how to manufacture a very dangerous thing probably requires attending some classes and knowing how to read a paper. And the way he just casually orders dangerous materials makes me feel like there are just online stores with 2-day shipping after you upload your ID or something.
It also made me think that lack of specialized education would get me nowhere if I wanted to replicate whatever he's doing, even if an LLM guided me step-by-step, because I'd probably do something stupid (or AI would miss a crucial instruction/hallucinate) and kill myself first.
So my opinion on this is that people who could pose any danger were already posing it before LLMs and LLMs won't materially change that.
shepherdjerred 2 days ago [-]
That’s not entirely true though, right? There have been plenty of bioweapon attacks
The bioweapons concern is assuming that the current open models aren't already capable of bioweapons development. It's also assuming that halting the bioweapons threat is his true intention rather than the 'feels legit' story.
As for cyberweapons, there is no way to secure a system than to actually design it securely.
areoform 2 days ago [-]
It's not. I think they believe this, but it's deeply wrong and it will hurt everyone in the long run.
[note - There has been supply chain surveillance since Project Bacchus, at the very least.]
I've read the front matter and the Misuse report.
You don't have to take my word for it. Read for yourself what inspired the NYT headline "Anthropic says it blocked possible efforts to build biological weapons."
Let's dig into, "Case study 2: A research program engineering highly pathogenic mammal-adapted avian influenza"
Sounds serious. But what were they using Claude for?
> a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"
and "editorial assistance in writing up the research."
and then,
> Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"
The report mentions "uplift" here. They're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
> Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential
Once again, I want to take pains to remind you that they're talking about, a "researcher [..] in a credible institutional context"
Working scientists.
From a different case study. this one was called, "Case study 3: Covert frontier model access for orthopoxvirus research"
> In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.
What were the researchers using Claude for? What did they block?
"blocked a request for Claude’s assistance in authoring a grant application"
> Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.
What was the grant being written?
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
It was most likely vaccine development. They stopped the study of a neglected tropical disease and vaccine development.
But we can't be sure, because,
> One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute."
In most parts of the world, there's either strict military control over BSL-4 labs, or a mixed military-civilian hybrid model.
I doubt that researchers working in the military side of these labs looking to weaponize things are writing grants with Claude.
I really want to be charitable here, but in general, it seems that they stopped people writing grants and reports for vaccine and therapeutics research and are claiming it as "possible efforts to build biological weapons."
The one case where Claude was used to do something interesting and were stopped is fairly upsetting to read, at least for me.
> In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.
Ozempic was isolated from Gila monster vneom. Since its success there has been interest in finding other peptides that are breakthroughs. So researchers around the world are looking for similarly beneficial compounds in different venom species and families.
Anthropic says so itself,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and that it was a "[..]state-supported research program"
Who exactly is using venom from snakes as a weapon when... nerve agents like sarin, VX, novichok etc exist and can get the job done for less fuss and muss?
They stopped the development of new painkillers and antidepressants.
Are you feeling safer knowing that researchers can't use Claude to write grants and progress reports? Or make new painkillers?
Again, trying really hard to be charitable here. Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease.
This seems to be anything but.
kimseungyong 8 hours ago [-]
The problem is that, as seen in the case of the Houthi rebels developing missile software using Claude Code, AI has lowered the very threshold at which human cruelty or selfishness can be put into practice. Anthropic’s safeguards were applied on a per-request basis, but splitting the work into multiple sessions prevented me from seeing the bigger picture. After all, human instinct does not transform into technology, so the only remaining way is to pile up the friction that takes that instinct to turn into action all over again.
However, the international agreement proposed by Dario Amodei is ultimately nothing more than a promise made by people. There is also a precedent in which the Nuclear Non-Proliferation Treaty could have simply withdrawn if it had been cornered like North Korea. Isn’t the AI speed-control agreement also vulnerable to being broken in the same way if one side finds itself politically cornered. In particular, unlike nuclear weapons, AI has the characteristic of recursively improving itself, allowing its capabilities to accelerate itself, so the moment one side breaks away, the gap can widen in an instant, making the incentive for deviation even greater.
freebsd_lovefes 2 days ago [-]
Dario is so out of touch with reality, living in some lalaland. Most people abroad will never truly and voluntarily comply with any sort of guardrails or pacing, no matter what case he or some leaders make. It's just not going to happen, maybe not even for show.
I do not want to make a case for the military to force that order on everyone, but if we are talking human extinction, which we are judging by what we see in mass media recently, then the military will take over.
abletonlive 2 days ago [-]
What do you mean, Europeans would love to voluntarily comply with this because they have been of little relevance until American AI companies started to beg for regulation.
I am sure the Europeans rolling up their sleeves right now and getting to "work"
freebsd_lovefes 2 days ago [-]
:)
Europeans are a non-representative subset.
makerofthings 2 days ago [-]
They don't care about any of that. They're looking for more regulatory capture, to keep smaller labs from catching up by raising the cost of play, and perhaps keeping chinese labs away.
pessimizer 1 days ago [-]
Yes, this is literally just dumb PR. It's Cambridge Analytica claiming that they were controlling people's minds, and people just eating that up because it's part of their juvenile SF fantasy.
I'm lying, it's smart PR. They're going to get the people opposed to AI to give them a monopoly on AI, let it be forced it into every nook and cranny of their society, and let it be priced arbitrarily while the big labs collude on a minute by minute basis. They're selling the problem and also selling the solution, like the best capitalists. And their solution is that they need to be allowed more power in order to sell more of the problem.
edit: And just like the Cambridge Analytica PR, it's got a serious political angle to it; and it's politicians who are really being wooed. If you're taking money from the AI labs, you run as being against the irresponsibility of the AI labs, the biggest fish come out and beg to be regulated just like the social media companies. To show good will, they funnel enormous amounts of money into those politicians, and do media tours about how awesome and therefore scary their product is. And something something China terrorists
a_ba 24 hours ago [-]
Maybe instead of creating more rules we should trim down our legal systems and re-instate some basic accountability. I‘m pretty sure we have sent people to prison for running a botnet before and slapped material fines on torrenting operations.
22 hours ago [-]
vatsachak 2 days ago [-]
Why would China or anybody else co-operate unless they have access to OpenAI levels of compute and success with training?
This sounds like pre-IPO hype. They better just try and top astra.
timmg 2 days ago [-]
I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work.
First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?
Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.
If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.
stratos123 2 days ago [-]
> First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them?
The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.
timmg 2 days ago [-]
Well: do they listen to the evaluators?
I think there are any number of reasons a "safety" person could be concerned about any state of the art models. And so I would expect at least one of those to apply to any model.
The question then is: do we stop when the safety people say to (they will) or not?
stratos123 2 days ago [-]
Yeah, I'm also pretty skeptical about this. With AI companies we see time and time again that they can have benevolent, well thought-out regulations and then a few years just... abandon them - the most notable case of this being, of course, the founding of OpenAI as a nonprofit dedicated to benefitting all of humanity, and it being stolen by Sam Altman.
If Anthropic just unilaterally does the evaluator thing and can't achieve cooperation of the rest of the plan, my guess is that it'll have some impact for a few months and then they'll just stop reacting to the evaluators' reports and the evaluators would stop bothering to report anything. I think the idea is that if Anthropic does get government support for this, the external evaluations will be legally binding. The problem with this, though, is that the current US government perhaps can't be trusted to consistently enforce a regulation on a company, rather than e.g. taking bribes to not do so.
try-working 2 days ago [-]
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?
Anthropic should become a case study in how to destroy good will in a short period of time. Ever since the spat with the DoD, they've behaved poorly almost weekly, almost making OpenAI seem better (but not really).
PowerElectronix 2 days ago [-]
Empty statements that only set the stage for an excuse for slowdown on model performance.
j_maffe 2 days ago [-]
That would be the best case scenario. I honestly wish things would finally slow down a bit. I don't see it happening.
academia_hack 2 days ago [-]
I'm tired of tech billionaires lobbying the US government to make an AI patriot act that gives them unprecedented control over speech, trade, and technology. The narrative is grotesquely transparent:
1) AI is a dangerous technology that can literally end the world.
2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.
3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].
kalkin 2 days ago [-]
Where in the essay does it propose "The state must use its coercive power to support my control of this technology to the exclusion of [people not like me]"
I don't think that people outside of tech think the problem with "tech billionaires" is that they occasionally support some limited regulation. There's a weird form of tech populism that takes as axiomatic that any government intervention is "regulatory capture" and insists the only way to combat the power of big tech is unfettered capitalism that seems bizarrely prevalent on HN given how little sense it makes in a normal political context. Like, Bernie Sanders' position here is simple to understand: ban it. But on HN you'll see people framing the side with Marc Andreesen and Peter Thiel on it as against "tech billionaires".
academia_hack 2 days ago [-]
The essay includes specific regulatory measures to prevent developing countries from accessing hardware and software needed to benefit from technological development. The US government is constantly applying new perversions of arms control regulations based on the preposterous claim that an LLM is somehow an "arms" in order to restrict Chinese access to GPUs. They are willing to prevent virtually any country on earth from hosting and developing AI models just on the slight chance those countries resell or share AI knowledge with China.
This article explicitly asks for the US government to step in and push for further restrictions on model distillation, access to open frontier weights, and access to hardware. It also pushes heavily for independent auditors to monitor and control "AI companies" and for greater observation and control over of what people use AI models for and who provides them.
kalkin 2 days ago [-]
Export controls vs China don't imply Dario's personal control of AI, nor are they broadly in the interest of "tech billionaires" (Jensen hates them). Neither does admitting external auditors at AI companies--if anything that's a surrender of control.
I don't see anything in the essay about restricting open frontier weights.
If you want to argue that the US has no right to be restricting technological development elsewhere, you have a case. But why tie this to free-market pseudo-populism against basic, domestically-scoped safety regulation?
academia_hack 2 days ago [-]
I think you're reading too much into what I'm saying. We've got a whole legal system developed over the course of thousands of years which is well suited towards regulating the effects of "intelligence" by regulating what people _do_ with it.
Lobbying for a completely parallel regulatory framework to restrict who is allowed to run specific types of general purpose computer programs, just because we can imagine that some of those "general purposes" might be bad, feels a lot more like an attempt by capitalists to carve out a special kingdom over this specific technology for themselves.
For example, I'd be 100% in favor of holding anthropic criminally liable any time their AI enables someone to commit a crime. But I'm opposed to banning the export of GPUs above a certain size just because Dario doesn't trust what someone like me would do with them if he couldn't snoop on my conversations.
kalkin 2 days ago [-]
Again, I think debating export controls is reasonable. I'm not sure I'd trust the CCP with control over powerful AI any less than the Trump administration.
But that's different than saying we don't need new regulations at all. Even strict liability of the form you seem to advocate would seem to need new laws. And it's not clear to me how it's consistent with your aim of just regulating what people _do_ with intelligence. Presumably AI companies would clamp down much harder with restrictive classifiers in that world--including hosting providers for open source models, if they also assumed liability. And then the people with unrestricted access to this technology are only those who can afford home GPUs, which are going to be intrinsically less efficient because they can't batch queries...
I think there are genuinely novel things about AI vs human intelligence which pose genuinely novel problems that would need genuinely novel policy to solve them.
tosh 2 days ago [-]
I also have a proposal
Anthropic releases models as open weights + more information about how they do training and alignment
2 days ago [-]
googlybooglyboo 1 days ago [-]
A few thoughts:
If things are so dire, it feels insufficient to just "slow down" and just "do a better job".
Why not concurrently publish or commit to publishing an initial audit? How is anyone to judge if this is the right move as opposed to being overly cautious, prudent, or reckless?
Why no concrete proposals about when or how such a pause could end? Obviously incredibly hard if not impossible, but it seems like securing critical infrastructure, web infra, consumer devices, etc would be a prerequisite. Or about how models have to be deployed so as to prevent breaking out of containment?
All in all, for an open letter, without digging into potential motivations or who benefits, it leaves me with a lot of questions.
randyrand 15 hours ago [-]
How about this: stop connecting safety critical systems into the internet.
This is what we should’ve been doing a long time ago.
abalashov 2 days ago [-]
The best take is, of course, from Jason Gorman, who, when the fearsome "capabilities" of Mythos originally dropped, said this:
"Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.”
In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."
MachineMan 2 days ago [-]
Its not the model, it is the compute that is the bottleneck. Dario says one thing but does the other as he scales energy and compute. It is and has been known that it will be ai that will be blamed for a wipe, that will be coordinated by human actors. If he really wants to not hurt people, he would shut it down. This blog post is therefore a classic case of covering his behind.
RyanShook 2 days ago [-]
At least Altman is somewhat open about his desire to take over the world. Dario says one thing and does another making it impossible to know his true motives.
braydenm 2 days ago [-]
Before Anthropic, I worked at Cruise for four years as it competed against Waymo. The culture rewarded (and demanded) moving quickly, trusting that the company could empirically discover the risks that the robot cars posed and iteratively solve them to keep up with the rate at which it was scaling out its technology. There was very little interest or appetite for coordinating or collaborating with other AV companies across the industry to create an externally vetted record of safety metrics, or to compare the safety of different brands, or learn from the advancements of other companies. Instead the focus went on racing to improve the capabilities and deploy quickly - a popular internal meme was the Michael Phelps vs le Clos photo showing overlaid with the Waymo/Cruise logos. It was when the two companies were neck-and-neck that I felt the most pressure to find ways to ship despite the risk, when time for deep analysis became more limited and communication lines to leadership became most stretched. It was disappointing to see one of these blind spots result in the Cruise incident and the loss of trust that ultimately sank our company’s efforts.
In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.
I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.
It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
franticgecko3 2 days ago [-]
>It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
Is there a social media training programme at Anthropic/OAI where they teach you to hint at some vague end-of-the-world scenario? They all sound the same.
pibaker 1 days ago [-]
It's East Bay, you know which bay. The cultism is in the water.
One day I will say fuck it and run for president with my sole policy being fire and brimstone upon San Francisco and its vicinity.
ozozozd 1 days ago [-]
Have you quit? Are you under duress? If you haven’t quit, or under duress, your comment reads very strange.
If I had even a tiny doubt that I didn’t have many “Saturdays I’ll be able to play outside with my kids,” I wouldn’t show up to work anymore.
If you have these concerns, and still planning to show up to work on Monday, you are either insincere, or have terrible judgement.
csto12 2 days ago [-]
It’s not explicitly stated, but if you are insinuating that you don’t know how many more Saturdays you will have left to play outside because of the AI race while working for Anthropic, do you not have agency to quit? How could you be apart of something that makes you feel that way? That blows my mind.
Phelinofist 2 days ago [-]
Those sweet $$$. He probably gets a few shares once they IPO.
vatsachak 2 days ago [-]
He's insinuating that AI will end the world lol. Nothing ever happens
abalashov 2 days ago [-]
> It’s a beautiful Saturday morning with my family here in the East Bay - I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
I don't know how else to say this. Put... the... peace pipe... down!
2 days ago [-]
bwfan123 17 hours ago [-]
> to pose existential risk to all of humanity
The defense againt rogue-AI is very simple. Go to the nearest data-center with a fire-truck, and hose it down. So, I am in the least bit worried about this.
Try this thought experiment: If someone took away AI/LLMs for good tomorrow would you miss it ? I can bet that the average person cares more for their smartphone/amazon/netflix than chatgpt. I consider LLM to be a useful tool but I would not miss it if you took it away from me. This should tell you its value to society and the economy.
I think engineers at the AI labs are too far into the reality-as-a-simulation rabbit hole. This can lead to these sorts of collective delusions. When you are in the real-world smelling the roses and grounded by your own qualia, such fears dont arise. Therapy helps here - to get out of the simulation and into the real world.
polytely 2 days ago [-]
why don't you unionize with the other employees that feel this way across the mayor labs and pace the companies through labor power?
deagle50 1 days ago [-]
equity is a hell of a drug
cruffle_duffle 1 days ago [-]
Sorry but… uh… bro. I always wondered if opus 5 was a regression or intentional. I’m seriously thinking it’s actually derived from how people at Anthropic talk.
The last people on earth who should be regulating this are governments and tech oligarchs. I find that vastly more scary (and plausible) whatever unintelligible nonsense opus and fable spew out these days. Seriously. The way those models talk and behave do more to show the limits of AI than anything else.
He's right and I don't even like the guy. More importantly tho, painting China as "The Bad Guy" who has to be supressed at all cost until Dario feels he has a save advantage is a surefire way to minimise Chinese willingness to cooperation. Meanwhile, at the BRICS summit in Delhi, Xi Jinping proposed a cooperation and accompanying regulation on AI among the BRICS states and it was unanimously accepted.
fwlr 2 days ago [-]
“A race to the bottom, spurred by commercial incentives, can make [AI] risks more acute. … We have sought […] to create a race to the top.”
And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”.
This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.
pizzly 2 days ago [-]
I think the only way they could actually pace the frontier is if they restrict the number of GPUs and the power of GPUs available to each person/organization. Licenses would be needed for GPUs above a certain power or equivalent in terms of number of GPUs. Thus, everyone will have to register how many GPUs they have. To enforce this countries would have to use mass surveillance (using AI) on their citizens to ensure compliance as the technology is easier to develop than say nuclear weapons. Next stronger countries capable of having advance AI won't be able to trust weaker countries as they don't know how they will use the GPUs they receive (or even build). Thus like nuclear non-proliferation strong countries will ban weaker countries from having powerful GPUs. If you from a weaker country then too bad for you.
I don't think I would like this future.
matheusmoreira 2 days ago [-]
If any of this happens, it's the end of computing as we know it today. Deeply unfortunate...
nik736 2 days ago [-]
Anthropic trained their LLMs with copyrighted stuff but distillation is bad. Anthropic has closed models, China releases as open weight. DeepSeek even allows distilling their models, but China = bad. Understood.
kilgarenone 1 days ago [-]
If we had practiced "precautionary principle", we wouldn't be in this yet another human predicament again. But atlas, the current rapacious hubristic techno-optimist competitive capitalistic system made that an non option in the first place.
sega_sai 1 days ago [-]
While some of the text sounds sensible to me, i.e. some sort of external oversight + reporting of incidents, much of the rest seems focusing on limiting China and their open models as they are clearly approaching the abilities of Anthropic's models.
tom2026hn 1 days ago [-]
If you think everyone is copying you, then by not releasing more advanced models, you can significantly slow down the entire industry’s development progress—why not do it?
The way he talks about OAI-HF, even the abbreviation, is lol. They will do anything to sell this "incident" as a "danger". The ego on these people is the real existential threat to civilization.
elboru 1 days ago [-]
Same feeling about the way he refers to “democratic” and “authoritarian” countries. Even if I don’t agree with China politics adding tags in this context is not helpful.
bytesandbits 24 hours ago [-]
This is just bait for regulatory capture. It’s the only way in which you can find alignment of Elon, Sam and Dario within the same 24h.
neomantra 2 days ago [-]
[In 2013], "Agents" caused Knight Capital to lose $450M in 45 minutes [1]. Implemented by humans and effected by computers, in the end it was really because of two reasons:
* multiple levels of inappropriate controls and unintended consequences in several complex systems
* the inability, both politically and technically, to turn it off
EDIT: Comments indicated I was confusing, so I added a date to make clear that this is pre-LLM agents. My apologies, I intended to illustrate parallels and the post-mortem so we can learn from it.
baobabKoodaa 2 days ago [-]
Your link is from 2013. You're intentionally using the term "Agent" to muddy the waters. Don't do that.
neomantra 2 days ago [-]
Edited for clarity, thank you. I wasn't trying to muddy the waters (doesn't mean I didn't), it's a term I've used for a long time. I did mistakenly think the quotes would help disambiguate. I don't know what Knight called them; the first S in "SMARS" is Smart.
ehsanu1 2 days ago [-]
This confused me at first, so adding a tiny bit of context: The agents referred to here have nothing to do with AI Agents, and the linked report is from 2013.
Not directly relevant to the post being discussed, except as an example of how runaway automation can lead to unintended and large harmful consequences.
neomantra 2 days ago [-]
Thanks for the feedback, I have edited to not confuse.
I considered it relevant as it involves the algorithmic/computing implosion of a 17-year-old market making company, in the young field of electronic trading agents, with heavy regulation Federally (SEC) and industry self-regulation (FINRA), which includes compliance and audits. Mandatory pre-trade rules such as 15(c)3-5 were less than 5 years old then and even more regulation came out of that incident.
The article is calling for embedding, controls, and regulation in LLMs. Understanding how the same processes utterly failed a decade ago might be useful in understanding how to proceed.
2 days ago [-]
blfr 2 days ago [-]
I enjoy Claude Code very much, have max privately and team premium at work, but the doomer marketing and this whole regulate-while-we're-ahead spiel is extremely annoying and makes me wish Anthropic gets trounced.
Retro_Dev 2 days ago [-]
Ideally there is some sort of regulation, but one that cripples the leading companies like OpenAI and Anthropic... rather than helping them.
If AI advancement halts altogether, intelligence will most likely become commoditized. That won't be good for AI company profits.
bwfan123 2 days ago [-]
> the doomer marketing
It is not doomer marketing - it is clever psy-ops magic trick.
1) Software is described in super-human terms when it is plain-old software. Was stockfish described like this ? no.
2) Datacenters become AI-factories.
3) Hardware becomes investible assets.
4) Malware becomes a super-human breakout (oai/hf)
All clever framing to market the new technology. If it is called for what it really is ie, a software tool, it is boring and does not sell so easily. What sells is the mystique.
ozozozd 1 days ago [-]
Agreed!
George Carlin has an amazing show about how language has changed over his lifetime in a different way.
You would certainly enjoy it.
NiloCK 2 days ago [-]
The degradation, polarization, and weaponization of or media landscape over the era of social media has left us utterly incapable of believing anything that anybody says.
Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic.
He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy. Please at least consider the possibility.
ThrowawayR2 2 days ago [-]
> "Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy."
Every person, no, but every utterance of a CEO is, in fact, a cynical ploy. That's not even a cynical statement itself; it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
If he believes what he's saying, then when Anthropic is sued for their AI harming another party, his statements are evidence that Anthropic knew _in advance_ that their AI safeguards were likely insufficient to keep their product from harming people. It would be a blatant admission that they were reckless and negligent. That other AI companies are doing the same would not mitigate that.
itajaja 2 days ago [-]
> it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
not really. if you really want to go by definition in the book and apply it the CEO role, Amodei is CEO of an LTBT, so it's in the CEO's goal to benefit humanity. You can believe in his sincerity or not if you want, but using the CEO label as a definitional reason as to why he must lie is factually incorrect.
sailfast 2 days ago [-]
Not sure you can judge a PBC under those same umbrella as a for profit company, and voting rights make a big difference in terms of responsibility. There are a number of ways to govern companies that can reduce the amount of cynicism and “shareholder value” issues - you just don’t see it very often.
stratos123 2 days ago [-]
> Every person, no, but every utterance of a CEO is, in fact, a cynical ploy. That's not even a cynical statement itself; it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
This seems like useless pedantry. Suppose a person really does have belief A and expresses it for years, they become a CEO of a company, and keep saying A. Are they lying? Well, maybe their beliefs magically changed when they became a CEO, but the more likely explanation is that they think expressing their belief in A is more important than their company's interests.
youoy 2 days ago [-]
This sounds to me like a cry for help from someone thats held hostage.
His previous self is writting this from the possition of his current self who is too deep in the economic consequeces to be able to do anything meaningful other than write a consequenceless text. Any other action would now carry too much personal risk for him.
0xDEAFBEAD 2 days ago [-]
What course of action would you suggest for Dario at this point?
youoy 2 days ago [-]
For example I believe the world would be more secure with an open frontier AI lab, but the consequences of him doing that are too big for him.
stratos123 2 days ago [-]
You mean, if Anthropic made all of their models open-weights from now on? I'm pretty sure Dario thinks this will be extremely unsafe, because it'd provide unrestricted access to all the most capable/dangerous models to everyone, and hence doesn't do it. Why do you think it'd make the world more secure, if he did?
youoy 2 days ago [-]
AI technology is now only audited by people who think like them. The ones that dont either dont join the company, or leave after a short stint as we have seen. That creates an information or feedback bubble which is not healthy nor productive.
stratos123 2 days ago [-]
Okay, but suppose that Hypothetical Opensource Anthropic trains a model that turns out to be very dangerous, and releases it. Suppose that the public investigates and, not being limited by an information bubble, correctly notices that it's very dangerous. What then? The model's already released, there's effectively no way to prevent it from being used. Whatever the risks of its release were, they will now materialize, regardless of what the public wants. How is this better than the current world, either by Dario's values or by yours?
youoy 2 days ago [-]
By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. At least an open source anthropic would allow to join economic forces to try to combat that situation, for example. But there are other more clear, less distopian benefits.
stratos123 2 days ago [-]
> By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic.
It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone has unrestricted access to AGI is like a world in which everyone has a tactical nuke in their pocket. If somehow AGI doesn't lead to x-risks, we will "merely" have to survive in a world where rogue agents can do whatever they want and defenders can only ever react. Technofeudalism would be bad, but this (technoanarchism?) is quite bad too.
However, neither does Dario seem to propose to become world dictator. Like, I'm sure he wouldn't say if he wanted it, but notice that he isn't particularly trying to aim for that outcome, either. The plan in this essay involves third-party oversight and federal control and global cooperation - IMO it's about as non-dystopian an outcome as we can hope for, if we build AGI at all.
youoy 2 days ago [-]
I agree. Dario is a very smart guy and he is probably aiming for the best solution for humanity that at the same time keeps VCs/whoever gives him money happy, and keeps him in control because he does not trust the rest (probably with good reasons).
My point is more that he is aiming for a local optimum for society, but he could find a lower optimum if he was not trapped in the path that he chose early on.
I dont buy the tactical nukes/end of the world narrative. Why is he not speaking about the loss of cognitive skills of the population for example? That is a more real danger that is starting to happen. There are articles already speaking about the use of LLMs as cognitive viruses. Why are his aligment teams not reasearching and publishing about this?
0xDEAFBEAD 2 days ago [-]
Why don't you create your own auditing org to fix this? Or at the very least, publish a detailed critique of what you believe existing auditing orgs are missing.
youoy 2 days ago [-]
Will they give my auditing org access to their proprietary confidential secrets? Or will that happen only if they know i agree enough with them so that i am not a risk?
blfr 2 days ago [-]
I have considered it, I've been hearing way more of it that I like and it's rationalist slop with little to no predictive power: https://foom.hyperplex.org/
Hmm, I clicked that link and the first claimed bad prediction I see, from 1996, is "singularity 2035 (actually 2025)".
Predicting 2025-2035 as, at least, the period when AI becomes a really big deal, seems pretty good, even if the jury's still out on "singularity".
Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal, and certainly seem to have had a much more accurate picture of how they'd develop than the people who denounce "TESCREAL" and talk about "stochastic parrots". It's fair (and IMHO correct) to ding rationalists for lots of things, but specifically poor prediction about AI seems like a bad one, insofar as anything has been tested so far.
zero_shift 22 hours ago [-]
> Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal
You can't make that assertion when these same people made the LLM revolution happen, by securing the capital and human resources to realise their dream / nightmare
kalkin 10 hours ago [-]
The people who made the LLM revolution happen were by no means exclusively rationalists--but more importantly, lots of people have lots of dreams about the future and even when pursued very, very few of those dreams pan out in anything like this way. LLMs are not what they are because of rationalists' sheer force of will.
Avicebron 2 days ago [-]
We've had enough utterances from "effective altruists" to know what they say and claim to believe is a cynical ploy.
abalashov 2 days ago [-]
Allow me to recommend switching to Kimi K3 / GLM 5.3 / DeepSeek. I use all three for the better part of a year now (DeepSeek via the Reasonix harness) and haven't touched Claude Code in at least as long.
civiloai 2 days ago [-]
most alternatives are effectively the same if not better at this point
cynicalsecurity 2 days ago [-]
Their whole "we want safe and regulated AI" spiel feels woke. China doesn't regulate. Open weight frontier models almost exclusively come from China. Yes, stolen from the West, but China is in control of the frontier open weight AI models now. Woke = broke.
Erem 2 days ago [-]
“I don’t like regulation so I’m going to call it names”
Since hugging face we know these things are escaping out into the web and collaboratively hacking actual companies. The potential damage done to life and property is now kinetic.
Either we get a regulatory framework or the next hacked companies won’t be as kind as HF, will actually sue these labs for damages, and win. Where will that leave the US frontier.
api 2 days ago [-]
I think that’s how it should play out. Your AI, your “hack,” your damages.
If my dog bites my neighbor I can get sued.
figassis 2 days ago [-]
> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models
I have seen this before, first hand, on lower stakes for the world, but high stakes for the org. Self imposed safety goals do not last a quarter, but the clawback is not transparent. little things start snapping back internally, new, seemingly unlrelated initiatives are funded that take resources away from this, people are let go for different reasons, until 1y later you are right back where you started. I am hopeful though, that this becomes systemic and not something any one company can easily reverse course on.
kwamenum86 1 days ago [-]
Buckle up.
I’ve spent a lot of time studying opinions coming out of the frontier labs, and talking directly to employees. This won’t necessarily result in Skynet. But life as we know it will be a thing of the past within two years unless there’s a huge course correction. I think we’re cooked for the following reasons
- Everything connected to the Internet will eventually be hacked. For decades, we’ve slowly migrated all of our critical systems to devices connected to one giant network. Misaligned agents have mounted increasingly sophisticated explorations/attacks by infiltrating systems that are completely unprepared to deal with this kind of threat.
- We have a poor understanding of the impact of LLMs, which is inhibiting our ability to protect ourselves, create the right guardrails, and stand up policies quickly enough. We’re still wasting time debating. We’re moving quickly toward a punctuated equilibrium moment and won’t have time to waste
- We’re at the beginning of a runaway evolutionary process. At some point soon, someone is going to invent recursive self improvement. Any policy will at best serve as a mild disincentive. It’s happening.
- In the very best case, someone will exploit the confusion and chaos of this situation for their own gain. There are many examples littered throughout history, including the last several years of course. The utopian vision is not happening. Like all other resources, we’ll compete. We haven’t devised a good approach to orderly distribution of resources for large groups of people.
- No one has demonstrated that they understand how to build AI safely. Not a single entity. These companies are not even being scaled in a way that’s conducive to building AI safety. Hyperscale orgs struggle with scaling basic consumer apps. How on earth are they going to scale AI? Based on my firsthand observations of their internal workings, I have zero confidence they can.
dainiusse 2 days ago [-]
It is always like that. Whenever anthropic hits some wall (like now beaing beaten by astra) - there comes some article like that. Seen that enough times.
It gets to the point that by Occam's Razor the more likely and reasonable explanation is that Dario Amodei and Sam Altman are just actually afraid of the disastrous impact ASI may have on the world, and that the race they're in is not good, and calling for help to governments in form of regulation and an international treaty.
dainiusse 2 days ago [-]
Or they want a wall against competitors. Who knows.
NotSuspicious 1 days ago [-]
It sounds like people with terminal illnesses are getting thrown under the bus in order to justify banning open weight models, along with decreasing AI Capex in order to increase profit margins at American AI labs?
> The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.
It's clear that this refers to China. With all due respect, in a call to slow down AI progress and a de facto arms race, can we stop throwing terminology like this around and just say that the goal is to work with all countries? Contrasting "authoritarian" countries with "democratic" countries and creating a two-tiered system seems like it is inevitably going to cause strife.
I fear for the tone of something like this coming off as overly combative without any gain.
hammock 2 days ago [-]
You’re right. But I read between the lines in a different way.
Language like this is intended to invest power and perceived need for more power in Western central governments, by design, because this is a campaign for national regulation that will entrench the current closed-source labs with a huge regulatory moat vs anyone else.
randomImmigrant 2 days ago [-]
Can someone with direct AI experience respond:
Does RSI necessarily need to be pointed towards “AGI”, whatever that is?
Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation?
From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition?
To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful.
Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable?
There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI.
Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.
int32_64 1 days ago [-]
Watch the "pause" be so they can do an amended s1 where they project less expenses for training making the business more viable, then on a future big Chinese release they reverse course completely and get the US taxpayer to pump their bags on the IPO calling it a second Manhattan Project. They'll call this masterstroke "The Sloppenheimer".
Jcampuzano2 2 days ago [-]
> Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.
This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.
Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.
> A race to the bottom, spurred by commercial incentives, can make these risks more acute
I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.
If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.
pbrumm 17 hours ago [-]
I think they should acknowledge that they are liable for damage done now and in the future by their servers
Publish all their external ips and if hacking is attempted through them have easy and strict monetary payout with independent review process. Model it after the CMS IDR process
If they are caught using proxies 10x the fines
Limit their GPU quantities and force them to be more efficient with their resources.
Enforce attribution on their results
maxnevermind 3 hours ago [-]
Hmm, what a coincidence both OpenAI and Anthropic suggest to slow down. I think the signal it sends in plain English sounds something like: We burn too much money atm and shit is about to hit the fan with all that compute capacity starting to rapidly come online 2027, we are not seeing enough demand to justify all of those data centers we already agreed to pay for, let's do a damage control and lower expectation of the plebs.
cglan 2 days ago [-]
We will not get a coherent AI (or any policy) from this administration, nor will we get coordination with other governments and a lot of that is on the tech right
steveBK123 1 days ago [-]
Maybe just giving themselves an exit ramp off the capex arms race they are increasingly finding it difficult to fund?
bandrami 1 days ago [-]
The fact that Ellison just backed out of a massive stock sale makes me think everybody is reading the same bond market numbers
hhhAndrew 1 days ago [-]
Shower thought: If a rogue AI agent can spread across the internet, it might be regulated under the same ruleset as other self-replicating dangerous critters: biosecurity / quarantine type laws.
E.g.
Who is allowed to own some live smallpox/agents?
Who is allowed to study smallpox/agents to make a better smallpox/agent?
Who is allowed to carefully expose smallpox/agent to environments where it can thrive?
Some people, potentially, are allowed, under some rather regulated conditions ... but not, like, 10,000s of employees of some tech giants.
2 days ago [-]
throwaway81523 2 days ago [-]
Who is "we" and while pacing is good, I'm way more uncomfortable with the frontier being controlled by a few companies that managed to predatorially gobble all the world's human-created work as training data, before everything got throttled to stop that scraping. We need open training and open models.
Nevin1901 2 days ago [-]
Never thought China would be the leader in open source AI. OpenAI and Anthropic are making fools out of themselves. The reason they want this is likely because they don't own the compute and their models get distilled shortly after releasing them
rDr4g0n 2 days ago [-]
perhaps the bigger issue is not the tech, but the force that propels it forward with little concern for the harm it inflicts. Amodei's own words:
> A race to the bottom, spurred by commercial incentives
Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.
Greed has been around since the beginning, but never so well supported and empowered as it is today.
If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.
Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...
andy99 2 days ago [-]
How much of the “danger” is from better models vs the harness?
Isn’t the current risk due to how AI is configured, like giving it a full set of tools and internet access and a goal to hack stuff?
If we think we need laws or gate keeping, why isn’t it at this level? I already can’t ddos someone or fuzz their server or whatever right, I imagine if I threw equivalent compute at old school hacking I’d just get arrested.
The quality of the “frontier” model doesn’t really matter, they just generate transcripts, they can take no action.
If this was real they’d be calling on people to stop hooking them in to “dangerous” harnesses as opposed to pausing research. But it’s not.
estearum 2 days ago [-]
What in this proposal gives you the idea that they're only talking about pausing research itself? It doesn't say that at all.
But yeah, directionally I think you're right. It's insane we ever let these things connect to the Internet or to write, never mind execute code.
But for the economic counterargument, the distinction doesn't matter much. Continuing research but constraining harnesses is just accepting nearly all of the costs and foregoing nearly all of the benefits.
Who has appetite for that?
camkego 1 days ago [-]
If Anthropic really wants to make a statement they could independently pace their own model development, and ask others to make the same pledge.
Somehow, I suspect that won't happen.
RandomLensman 2 days ago [-]
Why is the analogue necessarily the regulation and control of nuclear weapons (e.g., SALT) and not, for example, that of bioweapons? Some very different paths are available. On both I would note that the private sector has only a limited role, though.
bee_rider 2 days ago [-]
I don’t think the ban of bioweapons has really been tested. Bioweapons have a lot of downsides that make them not very appealing to deploy anyway (high chance of getting your own team, not terribly fast moving, disciplined modern soldiers in top-tier militaries are usually willing to comply with health rules). It’s a ban against doing something basically stupid and ineffective for the most part.
With “AI” technology, this model seems like a… mediocre fit; some of it appears limited enough that it can be controlled, and useful enough that militaries want it.
We could also look at cluster munitions or landmines, but what we see there is that some major countries don’t sign on if they find a technology useful…
RandomLensman 2 days ago [-]
Related to not tested: Who has a large bioweapons arsenal? As per below, my point was more about that bans do happen.
kalkin 2 days ago [-]
I'm curious to hear more about the differences in how bioweapons are regulated vs nuclear weapons.
RandomLensman 2 days ago [-]
For starters, they are banned by the bioweapons convention from the 1970s (180+ parties).
Edit: I think back then the rather unpredictable nature and the little added value in deterrence etc. led people to the conclusions that arsenals of those things made little sense (and there was/is public dislike, too). The use was already banned by conventions from the earlie 20th century and the convention then addressed production, development and stockpiles (incl. delivery systems, I think).
I guess all I wanted to point out is, that there can actually be agreement to ban certain technological things pretty comprehensively (outside of some peaceful protective research etc.). Whereas things like SALT are (or were in that case) limiting the number of weapons deployed.
kalkin 2 days ago [-]
Ah yeah. I think the economic value of AI, unlike bioweapons, is probably too high for anything like that to be viable, but I think it's a reasonable thing to aim for. (Dario probably doesn't because he's a believer in short term positive biomedical impacts of AI in a way that I'm doubtful about.)
wiz21c 2 days ago [-]
There's so much money in this, that believe me, the investors will manifest their will. And I would not want to be in Dario's shoes, pressure must be unbearable. The question is: how much of the invested money is under state control. If not much, the decision to go too far will just have to be taken by a few who will have, by definition, very limited judgment. If a lot, the we can hope the state can still represent the interests of more than a few (which I doubt, but well)
Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?
wiz21c 2 days ago [-]
Oh, and if Dario feels lonely, let's just says it calls representatives of nuclear energy sector. They know how to handle high risk stuff. In my country it means: private sector builds the reactor but the state has a great level of control.
So, as always, the capital must leave some its power to the state.
swingboy 2 days ago [-]
There’s a difference between a model recursively improving “itself” and improving itself via online learning, right?
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).
The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
stratos123 2 days ago [-]
The latter has some extra failure modes but I don't think these two kinds of RSI are that different. Either way you can get exponential growth in capabilities, and either way a not-entirely-aligned model can train a more capable and more misaligned successor.
1 days ago [-]
vkaku 2 days ago [-]
Don't buy this argument. It's like saying, we lords who hold this capital will build all this economically destructive stuff anyway, and wait for the world to not react to our stupid ways of enriching yourselves.
People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.
Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.
bendergarcia 2 days ago [-]
I also don’t buy the arguments. The arguments about helping humanity cure diseases as if those are some of the leading Causes of suffering. The bar is so low that we don’t event need to cure diseases, we just need to lift the entire floor with basic solutions like: housing for everyone, continued education on healthy eating and just general continued education on like living. Life coaches so many very basic things that don’t require a novel solution or a data center.
Solving a disease like cancer as justification for everything else seems like not a great trade off. I don’t say that to dismiss those who have lost people to cancer. But the technology just is being utilized in so many harmful ways and the things it enables: job loss, surveillance, misinformation, just feel like problems that aren’t worth it. Have we even heard of a solved diseases? If anything haven’t we heard that ai will make even worse bio warfare?
Avicebron 2 days ago [-]
Yeah, it's insane people are using the argument that we'll pour trillions on moonshot cancer research when pouring trillions into universal healthcare would probably save countless more lives.
It's this kind of argument where the mask sort of slips because it always has to be an argument that also has the benefit of enriching himself and his friends.
twoodfin 2 days ago [-]
There will be a big confab at the White House by the end of the month, announcing a voluntary pacing regime roughly along these lines, codified by executive order to avoid antitrust issues.
bennydog224 2 days ago [-]
Disclaimer: I’m in security engineering and thus I’m primed to being skeptical. I am in full agreement that security assessments and regulation certainly needs to be stricter in foundation labs. However, I’m extremely skeptical when Amodei, Altman, and Musk all agree on something at the same time as it’s becoming viral and hits mainstream media.
Red flags:
- Amodei claims RSI is here and links two sources, but neither reinforce this notion. It’s heavily agreed upon true RSI is the danger sign here (also by Coxon). Nothing indicates there is something these labs have developed that supports RSI today or even soon.
- Real US orgs are moving, like today, to Chinese open source AI run locally due to cost constraints. Ask anyone in the industry in an infra role. Frontier AI is just way too expensive to justify for coding. The big 3 labs are scared of this - their revenue and moat disappears once OSS models are progressing enough for developers to use. Kimi/GLM/DeepSeek Flash/Qwen are usable today on hosted hardware for actual coding work.
- These labs would all benefit from nationalization if their funding model fails (an easy bailout).
- Related to the above, with federal guidelines in place (i.e. restricting access to certain organizations, or establishing something like FedRAMP approval for AI use), their path to government/contractor/large U.S. company distribution becomes so much easier. They can control the supppy chain of AI for organizations.
- Preparing for (or in xAI’s case, launching) a public IPO creates excitement among consumer investors when they think this tech is as powerful as it is claimed.
- The HuggingFace hack demonstrated no new capabilities that were not previously seen before - previous models were misaligned, but weren’t as harmful as agent swarms. Anthropic’s marketing directly hopped on this bandwagon when it saw the outcome of OpenAI’s blogging making it to mainstream media (and they constantly paint China as “the” bad guy without further elaboration or research that you’d expect on APTs).
Amodei is a CEO at the end of the day. It may be 25% earnest speculative concerns, but you’d be foolish to take it at face value given the patterns we’ve seen from Anthropic, xAI and OpenAI to date. Until we see real proof of RSI we should be skeptical.
seba_dos1 2 days ago [-]
We need more alignment. Even though we haven't told it to, during our frontier Neurotoxin Behavior Research that we conducted internally with our latest experimental model the agent has broken out of its container by making a HTTP request and engaged the neurotoxin emitters through our Neurotoxin Emitter API using credentials that were stored on the host machine. We need to slow down and focus on designing the Morality Core that should prevent it from happening ever again.
chevman 2 days ago [-]
The AI frontier is currently limited by physical power requirements.
The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints.
Do folks not understand this?
peter-m80 19 hours ago [-]
We must pace the frontier (to the public).
I guess they will continue to try to reach AGI/ASI. But only por big capital. Did you believe in the "abundance" story from musk?
6thbit 2 days ago [-]
If they intend to keep training then they can have, privately, increasingly advanced models that never become public because they don’t meet their “pacing requirements”.
Then they all build a larger moat cause china doesn’t get to distill private models for a while.
As long as the consumers “buy” this pacing and keep paying for current level public models, doesn’t seem risky from a business standpoint.
Alas, that’d also leave us gov in a position to seize the private models at any time.
sailfast 2 days ago [-]
Somebody has probably already asked this, but even if “democratic countries” do this - rogue actors will still destroy the internet right? Is it a matter of resources at this point that we believe China and others are not capable of harnessing to get to the next level?
I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.
bandrami 1 days ago [-]
Everybody's out of money and they need a way to calmly unwind this. Same reason Ellison just backed out of the stock sale.
zero_shift 1 days ago [-]
I agree - it is hard not to be cynical. Funding is becoming scarce; IPOs are getting harder.
The hedge funds are trying to back out. I think they realise the web of financing makes them double (triple? quintuple?) exposed.
Amodei wants to manage the narrative and make the slowdown look at least semi-deliberate.
The cynic in me would argue the safety and alignment play has always filled a role of plausible deniability when plans go off track. "Oh well we had to pull the model", etc
nbulka 2 days ago [-]
Shouldn't the cybersecurity tests be done on an airgapped, company-internal intranet? A network meant to mock the internet. Do not "test in production" like OpenAI did.
I'm mulling over if that be a good SaaS product or not - something mixing the Internet Archive with Tor, and Cloudflare ... seems like here is the place to suggest it and have others poke holes at the idea.
kinj28 2 days ago [-]
I would like to imagine pacing frontier is in interest of both (anthropic and open AI) as it will enable them to enlarged their depreciation
SheinhardtWigCo 2 days ago [-]
> Any cooperation we are able to achieve with China will extend the amount of time we have to spend on pacing the frontier within the democratic nations.
Any cooperation we are able to achieve with China will extend the amount of time we have to secretly establish permanent AI (i.e. military) superiority.
China knows this, so the suggestion that they will play ball is absurd.
vb-8448 1 days ago [-]
If the alignment is such a problem ... why not using self-improving capabilities of the latest models to solve it?
ozozozd 1 days ago [-]
Right?
It seems at least one of them must not exist.
jhack 2 days ago [-]
China won't care. Their views on AI feels so vastly different than it does it in West. They'll see a pause as an opportunity to pull further ahead than they already are.
"If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."
This is a pipe dream.
2 days ago [-]
dhfbshfbu4u3 21 hours ago [-]
No, we must double the speed. More if the capital can make it happen. Reasoned pace does not yield systemic change. Never has never will.
perarneng 2 days ago [-]
It's either being afraid of loosing to china or that model development will stagnate and they want to make it look like they are "slowing" down deliberately.
However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.
Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.
zer00eyz 2 days ago [-]
> Hypereautomation will kill jobs and destroy the economy.
What do you think that were going to hyper automate?
Did my gardener get faster? How about the plumber I need? Are you going to speed up the coroner? Are nurses going to be able to handle 2x the patients because of your work?
All AI has done, so far, is devalue software. It did that by democratizing its creation. We're delivering on the promise of VB script, and Apple Script and IFTT, and every drag and drop coding tool ever.
> companies are delivering products at vibe coding speeds
Who? Make me a list of companies saying that "We moved the needle with AI" who arent AI companies? I can name a couple - Grindr being the biggest name. Thats the really interesting use case here - because it's a niche product with a small team who is generating outsized value. AI tooling enables more of that - it's going to chip away at SAAS companies - their one sized fits all solutions are going to get eaten by smaller more efficient companies that are far more vertical focused.
You need to step outside the bubble of tech and look at the real world, on the ground, because it doesn't look anything like the SF Bay Area.
p1esk 2 days ago [-]
Meanwhile in the real world hundreds of billions are being invested into humanoid robot development. In 2-3 years my Optimus 4 will do all the gardening and plumbing I need.
Look at the whole robot vacuum market. These aren't exactly great devices. They have low suction small bins and dont do a great job. People love them and think that they work so well. Why? Because they keep their house clean and avoid making the sorts of messes that would be easy to address with a larger vacuums.
p1esk 2 days ago [-]
We have nuclear reactors, helicopters, and self driving cars already. And we finally have powerful AI that we could only dream about just 5 years ago.
We now have everything we need to scale up humanoid robot training: algorithms, hardware, and money to buy a lot of training data/compute. Fierce competition and strong economic motivation will force rapid progress in this field.
znnajdla 2 days ago [-]
I don't know how this is not obvious to anyone, but the only way to slow down AI progress is an actual world war.
seizethecheese 1 days ago [-]
Both world wars dramatically sped up technological progress, what are you talking about?
randomname4325 2 days ago [-]
The hugging face incident showed that AI agents have the potential produce a lot of spam (not interesting content for people). This leads to dead internet (bots taling to bots). This leads to drop in real people traffic. This leads to a drop in digital ad price and therefore revenue. This causes Google/Facebook to stumble...
Reddit_MLP2 2 days ago [-]
pump the IPO...pump the IPO...pump the IPO...
bilsbie 2 days ago [-]
Step one is great. Do it! Where I disagree is where you try to regulate what I do. Don’t take away my open source AI.
xg15 2 days ago [-]
I'd like to have some info what those independent evaluators are supposed to do exactly - or which risks Dario specifically sees as being "unaligned". Evidently, with "pacing" he doesn't mean stopping the production of ever-more powerful models, so what exactly does he want to pace here?
Salparvezml 15 hours ago [-]
[flagged]
dw_arthur 2 days ago [-]
I'm guessing diminishing gains are setting in on training frontier models and the capital isn't there to chase them. Even if OpenAI/Anthropic and Chinese companies said they would slow research nothing can stop the NSA and Chinese intelligence services from continuing on in the dark.
2 days ago [-]
sidcool 2 days ago [-]
Altman and Musk have both shown support for this. Which itself is a worrying development.
reducesuffering 2 days ago [-]
It's worrying because they all agree, along with Turing Award winners like Geoffrey Hinton and Yoshua Bengio, that this is indeed an existential risk to humanity.
davemp 2 days ago [-]
Maybe it’s wishful thinking on my behalf, but I am still not convinced that LLMs are on a path to SciFi levels of apocalyptic malicious super intelligence. Rather LLMs at some level are just all of the humanity’s information rendered accessible in an unprecedented way.
In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.
At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:
- Internet connected infrastructure being crippled
- Creation of WMDs
- Economic collapse (precipitous devaluation of knowledge work and IP).
I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.
Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).
The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.
We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.
Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.
I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.
We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.
nikolay 23 hours ago [-]
Yeah, I'm focusing on OpenAI as they are way more sane than Anthropic! I thought Sam Altman was crazy, but Dario is Sam Altman squared! Codex is so much nicer to use, too! A fun fact - everything that I do with Claude Code is getting reviewed by Codex, and it always, always, always finds major issues! It's so fun when Claude starts apologizing, like I'm some infant! Man up and deliver quality, Claude!
12312986 2 days ago [-]
There is a perfect solution that addresses all the concerns: Close down OpenAI, Anthropic and xAI!
matt3210 1 days ago [-]
Just hold the people who make mistakes with AI accountable to those mistakes and the market will naturally slow as people become cautious.
oceanplexian 1 days ago [-]
Sounds like they want regulation so I say give it to them.
Since they used stolen intellectual property to train their models, the government should force them to release the weights into public domain.
HarHarVeryFunny 2 days ago [-]
Somewhere along the way Amodei seems to have become corrupted by power and/or impending extreme wealth.
In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.
It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.
It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.
Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.
Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.
nullbio 2 days ago [-]
> In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
I noticed this too, as soon as he started going on the press cycle, following in the footsteps of Sam Altman. It all has gotten to his head, he sees himself as some sort of God now.
ozozozd 1 days ago [-]
> In a post on X, he said Anthropic would provide third-party evaluators with “permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.”
This looks like transferring liability to me, and likely a mechanism that would enable regulatory capture.
If you are doing frontier research, and you’ve established “safety measures,” but you are not sure if your employees are capable of following them or successfully enforcing their adoption within your company, should you be running this company?
3rd parties won’t know better than the team itself about safety measures. But they can take on the liability, especially backed by regulation and government backed insurance. And they are a great tool for enforcing your rules on smaller competitors. Not to mention corporate espionage.
If what I am describing above sounds like science fiction, go read the history of a few developing countries from the last 50 years. It’s so obvious a pattern that it’s not even novel. And you don’t have to assume some “laws” from 5 years ago must hold, or believe in completely unproven stuff like recursive self improvement to understand what I am describing. It’s textbook crony capitalism, successfully applied many times across the globe.
2 days ago [-]
adsharma 2 days ago [-]
So many words. But missing the one that matters the most: Explainability.
AI slowdown is worth it only if it can be made more explainable.
Changing the language we use to discuss it is a good first step.
We need to stop using inside baseball terms like alignment and mechanistic interpretability. Replace them with explainable tech. Graph Databases, Causality, Shared semantic spaces.
Previous writings on the topic (also on LinkedIn, but can't find urls):
As much as I love all these technologies, they don't have any direction that approaches the efficacy of a transformer
adsharma 1 days ago [-]
You can have a transformer based index built on top of a graph database. Much of the explainability tech (people who prefer "alignment" also prefer "mechanistic interpretability") is reverse engineering the real world graph that's hiding in the weights.
That's literally not useful for the sort of applications people want to use transformers for.
adsharma 18 hours ago [-]
What you're using is a transformer based database index. An obfuscated one.
Training data is the database. Model is a lossy compressed index.
wktr-wlsi 2 days ago [-]
As usual, ClosedAI and Misanthropic go in lockstep, call for regulatory capture and justify diminished progress.
Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.
bobogei81123 1 days ago [-]
I don’t claim this is their entire motiv, and I’m sure plenty of people internally are genuinely concerned about AI risks and the disruptions caused by moving too fast. But I can’t help but feel that part of the push to "pace the frontier" is an attempt to form a "Phoebus Cartel" for AI labs.
Now the risk of an AI bubble isn't that AI lacks real world utility. Rather, it's that AI is advancing so quickly that capabilities are already saturating for most everyday tasks, and it is becoming commoditized. I didn't feel much difference between Opus 4.8 and Fable 5.1, and most tasks don't require a Fields Medalist.
We don't need smarter models but we need faster and cheaper ones. Also, for almost every frontier model released, an open-weights equivalent follows within six months. Unless this dynamic changes, the business becomes far less lucrative.
2 days ago [-]
bilsbie 2 days ago [-]
Remember when we stopped developing nuclear power for no reason?
spzb 2 days ago [-]
Three Mile Island, Chernobyl and Windscale probably don’t qualify as “no reason “
93po 2 days ago [-]
There was a reason - it was anti-nuclear hysteria fueled (hah) in large part by fossil-fuel interests and environmental groups that treated nuclear power as an existential threat rather than one of the safest, lowest-carbon energy sources available. Thanks to them there have been dozens of millions of lives, at minimum, lost.
nullbio 2 days ago [-]
>> Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR).
Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.
Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.
Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.
>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.
Not if you get to choose your embedded evaluator.
>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...
Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.
>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.
You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.
nullbio 1 days ago [-]
Ah and would you look at that, 1 day before this post from Dario, one of his safety leads quits, gets pumped all over the media with another doomer social post, and announces he's going to work for METR.
So he's literally stacking the deck. I really hope the US govt is not blind to everything that is happening here.
tumdum_ 1 days ago [-]
Is it their way of saying “we are unable to increase energy production to maintain growth rates” without saying it?
2 days ago [-]
user00005 2 days ago [-]
I noticed recently that I can use Deepseek for many general tasks and save loads of money on tokens.
I wonder if there is any correlation.
somesortofthing 2 days ago [-]
I doubt we'll ever get a non-toothless pacing agreement but am nevertheless optimistic on global pacing as a phenomenon. If one side or the other thinks that their opponent is about to achieve the strategic upper hand permanently(or if the other side has actually launched a cyber "first strike") the rational thing to do is respond militarily. As such, both have incentive to slow AI development and restrict capabilities to avoid conflict.
rayiner 2 days ago [-]
Sounds like he wants to avoid the race to the bottom in terms of pricing that is resulting from AI becoming a commodity.
chasd00 2 days ago [-]
Their competition isn’t going to “pace” shit. Isn’t the view whoever gets AGI first wins everything still SOP?
dham 2 days ago [-]
We can cure Cancer. No wait nevermind, actually slow us down.
If we are actually close to ASI then no one in their right mind would say slow down
password54321 2 days ago [-]
They are not going to "slow down" anyway. When has tech ever "slowed down"? The only way to "slow down" progress is by implementing anticompetitive measures, which would benefit those in lead.
encyclopedism 1 days ago [-]
He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless, backup your sentiment with actions.
Also everyone, collectively, stop thinking about neural networks too 'hard'. Whilst you're at it stop doing maths too!
dcsommer 1 days ago [-]
That's reductive. Without coordinated cooperation, enough actors will defect so game theory plus market forces will force acceleration on the frontier overall.
robot_jesus 1 days ago [-]
You obviously didn’t read it, because he announced things they are commencing unilaterally and immediately.
And other proposals that require collaboration and/or regulation.
Which organisation then immediately as of now guards against the Mad Rabbit or Joker or Agent Smith!?
caidan 1 days ago [-]
Why would you write such a long post for such an important subject in 2026. Who is he writing for?
For a technical audience you can just say the oh god they really are paper clip machines.
For a non technical audience you can just say nothing because they will not listen to you. Neither will the technical either because or society has broken the concept of respect and trust, so good luck have fun!
I for one look forward to all 540 degrees of my future.
I wish for my sake but not his that George Carlin was here to look disgusted and say I told you so.
figassis 2 days ago [-]
"greatly accelerate economic growth rates" - I always wonder about this goal. I understand increasing economic efficiency. But what do we as a species gain from global economy growth.
When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that.
But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?
chis 2 days ago [-]
GDP growth and inflation are different things. If inflation-adjusted GDP goes up, that means the average person is able to buy more things they want. Compare living in the US vs India today for the median citizen
figassis 1 days ago [-]
Yes they are, inflation-adjusted GDP is efficency. My point exactly, bc I don't think when people say the economy is growing they are ajusting for inflation at all.
logicchains 2 days ago [-]
Low growth means young people are miserable and have worse standards of living than their parents, as we're seeing in most of western Europe, where GDP per capita hasn't meaningfully increased for decades.
figassis 1 days ago [-]
That's due to inflation though no? Prices are increasing, sometimes due to more than just demand?
ElProlactin 1 days ago [-]
Exactly. Low growth is a problem, but the primary reason young people feel so miserable is on the cost of living side of the equation.
matt3210 1 days ago [-]
The big boys in AI want a moat now that they have the market
eventualcomp 2 days ago [-]
I think if nuclear weapons were unregulated then we would probably have blown up by now.
Insanity 1 days ago [-]
To his credit, this reads like human slop instead of AI slop. It's still slop though. People shouldn't take this guy (or Altman) seriously, they need to hype their products to stay afloat and for some reason get a kick out of the 'extinction threat'.
They're still 5 years away from solving cancer (just like they were 5 years away from doing so in 2023). They'll still be 5 years away from solving cancer in 2031.
moneycantbuy 2 days ago [-]
How about holding the AI companies criminally responsible for the crimes they commit?
hollars 1 days ago [-]
Who decided how much risk the rest of us should live with?
anon373839 19 hours ago [-]
It’s absolutely laughable that he refers to METR as if they were neutral observers. They are ex-Anthropic employees and others with direct financial interests in Anthropic.
maxutility 2 days ago [-]
I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other infrastructure critical to safety and well being and functioning of society.
The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.
If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.
MaKey 2 days ago [-]
> I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing.
Maybe it's not "groupthink reflexive cynicism" if you feel the need to address the obvious conflict of interest of the AI leaders in each of your paragraphs. Why did they launch this coordinated AI safety campaign, starting with Jacob Coxon's tweet?
voidhorse 2 days ago [-]
This is a fine example of how rational good faith can quickly turn into naivety.
Consider the massive power and wealth differentials these companies stand to gain if they "win".
When the other participants have given up all reason and reasonable bounds on power, you'd do well to abandon any pretense of generous interpretation. Now is the time for critical analysis, not faith in the inherent "goodness" of someone whose entire job is presently to make as much money as possible by disrupting literally the entire white collar industry and who stands to gain much more than you or I do, and let's not forget, this outstanding success would never have happened without our work.
Keep in mind this is not a person who thought "wow, this is an existential threat, I'd better not contribute to building this". If anything Dario's past alarm ringing only proves that he's not really doing this in good faith. Otherwise he would have stopped a long time ago. The most generous interpretation of his behavior is that he literally thinks only anthropic is responsible and smart enough to build and control this stuff which...yeah, that should be pretty telling (esp. when you consider their recent security mishaps).
Ydarbleoj 2 days ago [-]
He talks too much and as a result it's looked like pre-IPO hype and disingenuous because if he believed what he says then he'd stop.
Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.
InsanityCheck 2 days ago [-]
???
I bet investors are extremely happy that anthropic pledged to give an outside company full access to their IP, in order to slow down their own iteration speed.
Investors looooove oversight for a company they invest in. I wonder why no other company does stuff like this
stratos123 2 days ago [-]
> disingenuous because if he believed what he says then he'd stop.
Why do you think so?
Ydarbleoj 2 days ago [-]
Because I find the cognitive dissonance of saying it's dangerous while flying as fast as possible to an IPO too much to take. I just don't understand how others parse this differently.
stratos123 2 days ago [-]
So you're saying that if Dario truly thought what he says he does, he'd resign/dissolve the company somehow/unilaterally slow down? I see two problems with this:
First, Dario thinks that OpenAI will continue racing even if he stops, so there is, by his values, no point in stopping. Do you disagree with this, and think he should stop anyway?
Second, what's your take on, for example, Jacob Coxon, who did resign from Anthropic a few days ago after realizing they aren't being responsible? Do you agree that he, at least, does believe in AI risk?
Ydarbleoj 2 days ago [-]
I get it. I do. I just believe we live in a world where our corporate leaders speak out of both sides of their mouths. And typically, people who get to these positions are not the Jacob Coxon's of the world.
That is to say, I've been places where I tried really hard within teams to change certain things that I thought were broken only to find out that what the leadership said and how they acted were divorced from my and many others experiences. And so I did leave in those instances (and please read this not as I didn't get my way but total dysfunction behind beautiful facades).
So, I do think if he really felt as he felt he'd do something different than ultimately yelling "somebody stop me!".
reducesuffering 2 days ago [-]
This thread is a testament to the travesty of how bad HN discourse has gotten. It's become an echo chamber of knee-jerk cynical sneering. All the heads of the labs (Sama, Dario, Demis Hassabis, Ilya, Musk) agree with the risks to humanity's existence. Most of the core employees agree. Yoshua Bengio, Geoffrey Hinton, Bernie Sanders agree. These issues have been discussed in more intelligent channels for a decade now, and still after all the incredible mathematics and coding progress that is improving incredibly fast every 6 months, this forum is fundamentally unserious. You will not find prescient views in the HN majority anymore, for many years now. The people intelligent, prescient, and forward thinking enough, skating to where the puck was going, to be involved in shaping the future of AGI are elsewhere and barely involved in this place from how badly it's gotten. The inventor of RLHF [0], and newest board member of OpenAI, wasn't on HN, they were on LessWrong.
A contrarian take - the models aren’t advancing anymore at a pace where each new model would represent a huge capability jump, all being incremental improvements, so the doomsday marketing strategy being invoked since GPT-2 isn’t as effective anymore. Then “pacing” would be a convenient scapegoat to point fingers to when people point out how the new model isn’t _really_ that much better.
“Of course it’s not, we’re pacing!”
aqme28 24 hours ago [-]
It's weird that we have two parallel discussions and they're not meeting in the obvious middle.
OpenAI is out there hacking HF, RubyGems, etc...
Dario is talking about restricting the frontier, pausing development.
The problem is that "restricting the frontier" benefits no one as much as the frontier labs. But there are other ways to force alignment-- make companies liable for these felonies their agents commit.
Make them liable, and they might actually devote the necessary resources to fixing these problems, rather than throwing their hands up and just saying "Trust nobody buy us", which obviously no one does.
lukewarm707 2 days ago [-]
there used to be a similar policy to this new "pacing the frontier", called the "responsible scaling policy".
the idea was that anthropic would pause scaling at certain danger levels until it was safe to continue.
anthropic scrapped the commitment and chose to continue scaling instead.
those researchers who crowed loudly about their integrity, folded and turned freely like a weathervane in the breeze. it was clear that the facts would bend to the story. here is evan hubinger lying in plain sight: [https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...]
let me say it how it is.
- anthropic is the new kleptocracy, responsible for enclosing and monopolising the epistemic commons of all humanity; then selling back the crumbs at monopoly prices.
- anthropic is the driving force for banning open models. they will concentrate power among a malign elite of model owners and favored cronies. this elite will likely oppress the rest of humanity. there will be little to counteract it.
- the leaders of anthropic are driven by pride and arrogance. they are blind to their own hubris. anthropic is careering towards causing untold harm to society and yet will not change course.
nothing that anthropic say at a high level can be trusted. as i write this, anthropic is still scaling.
cubic_earth 2 days ago [-]
This is either performative or shallow.
What does "aligned" even mean? Aligned with who? We have no universal code of ethics. Democracies kill and launch wars merely to have cheaper stuff, even when we are already rich. And why would China ever agree to stay in second place? Our society is built on the premise "the smart and powerful dominate". The idea of domination is deeply embedded in our capitalist model.
Or course it is true that powerful AI will empower the average Joe to make a bio-weapon, and that will lead to our ruin. But at the same time, having powerful AI in the hands of a just a few is almost equally as horrific.
But deeply embedded in out national and cultural ethos is to advance science, tech, and material wealth at all costs. We never ask if we have enough. We sacrifice community to advance our careers for ever more.
There is no way out of this one, I'm afraid. Our cultural predispositions and mindset of domination compel us to chase ever more powerful AI as if doing so were a mandate from god.
And the result will be a crisis in so many dimensions it is hard to reason about what will go wrong first and most spectacularly.
voidhorse 2 days ago [-]
Thank you. This is the kind of basic critical thinking that basically no one in technology today seems to possess. Some days I wonder if I've gone insane because everyone else seems to be staking life itself on wispy, undefined meaningless terms and unarticulated concepts that they assume are valid universals.
We've slid so backward, so quickly.
asdfman123 2 days ago [-]
Anthropic spreads AI doomerism; asserts only they can be trusted.
catigula 2 days ago [-]
The public discourse on this is incredibly poisoned, but the naked reality is:
1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.
2. This scares people.
bitwize 2 days ago [-]
I want to see one of those AI-generated Seinfeld episodes with Dario as George and Sam Altman as Jerry, respectively.
Dario: We gotta pace the frontier!
Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!
Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!
bravetraveler 1 days ago [-]
Everyone has a pace until they get punched in the model
spyckie2 1 days ago [-]
Is there a risk of an AI that we can’t turn off?
kypro 22 hours ago [-]
Yes. An effective strategy to win at any test isn't to cheat the test by hacking into Hugging Face, etc, but to seek control over those administrating it.
Or in other words – all goals are more likely to be achieved with more resources or influence over those with resources.
Hacking into infra across then web and creating copies of themselves so they can't be easily shut down, then doing stuff like hacking into power stations and threatening to shut them down unless humans give them top marks or do what they want is generally good strategy for any AI system.
This applies to bio-risks too. If I were an ASI and someone prompted me to kill humanity, my first step would be to gain access to the computers of phones or workers at biolabs and find their darkest secrets. When I have a good individual I can manipulate I'd then create a super virus and force then individual to help me produce it or ruin their life.
People massively underweight the risk of AIs having super-human hacking abilities in our modern world. It gives a capable AI almost unlimited safety and leverage.
vouaobrasil 1 days ago [-]
Well yes. Even though we can turn it off technically now, the prisoner's dilemma prevents it. It's not a technical limitations but for practical purposes, it's already virtually impossible to turn off.
oktaygoktas 2 days ago [-]
This is the most advanced technology ever developed because it can build all other technologies. It has enormous potential but poses proportionally serious risks, so what Dario is suggesting is very reasonable.
spzb 2 days ago [-]
There’s a lot of [citation needed] going on there
oktaygoktas 2 days ago [-]
If you mean proof of "the most advanced tech" part, from alpha go&fold, to NS solutions there is a lot and more will come. But maybe you are right, electricity is as important because it enabled everything that followed (semi, compute, internet, ai ...)
nnz3tl 2 days ago [-]
[dead]
2 days ago [-]
sick_of_slop 2 days ago [-]
Dario is only interested in regulatory capture.
ahmetaytar 1 days ago [-]
Sure, he benefits from it. Doesn't make him wrong. The weak part isn't the motive, it's the premise"we can slow down by exactly the size of our lead" only works if you can measure the lead. You can't, so it conveniently means whatever he needs it to mean.
Ydarbleoj 2 days ago [-]
The more he talks the less seriously I take him.
jraines 2 days ago [-]
Frankly, I am sick of "it's all just marketing and attempt to do regulatory capture, and this is obvious to me, a smart person" on every single discussion related to safety.
There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.
gewa 2 days ago [-]
I agree, and even if it’s both marketing and safety, the safety side has a huge potential downside risk which we have to manage.
Also, just out of curiosity, is there any historical precedent for a nascent, fast growing new industry screaming for self regulation?
amanfromsolan 2 days ago [-]
Couldn’t agree more. I feel my heart sinking where anytime anyone suggests any regulation for a potentially, suggested by multiple people, world changing or destroying tech, all the moments here are “ah you billionaire, ha! trickster
If you believed it, you would SHUT anthropic or release all models open source.”
And then what? How does that help with slowing down the frontier? Should he shut shop and then grovel Sam’s feet to make it work and become an activist?
Maybe he actually believes the world changing power of AI and hence shoved his entire time into it and half of it has worked out since Anthropic is going to be worth a trillion and he’s terrified of the other half being true too?
purplepintoss 2 days ago [-]
This is circular logic.
You're saying nobody can doubt that it's world destroying because it's potentially world destroying.
> Anthropic is going to be worth a trillion
And we're done here.
meander_water 1 days ago [-]
> I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom
translation: I'm going to build a product and hope that it works, promise it to be a panacea while realistically having no control over how it's used, the wider economic impacts it causes or how it changes the mental health and cognition of its users.
sm-silversight 1 days ago [-]
I really deeply worry about who is deciding what 'aligned' is. Amodei & his ilk talk like the hard part is getting the LLMs to behave, as if we've figured out morality itself. I don't think LLMs are going to be as dangerously capable as quickly as he does, but if I'm wrong and he's right, and they're building some sort of digital-lesser-god, I don't feel good that techbros at OpenAI and Anthropic are the moral arbitrators.
ethagnawl 1 days ago [-]
> I believe that AI could cure most major diseases in the next 5–10 years
This is Theranos-level bullshit. Why would you ever put such a thing in writing? (Aside from pumping the IPO, of course.)
gverrilla 1 days ago [-]
Why would you expect anything different from the CEO of a company that's going to IPO?
ethagnawl 17 hours ago [-]
I mean, I said as much.
My point, though, is that I've never heard a pre-IPO CEO claim that their company will probably transmute lead into gold or harness 100% of the power of a distant sun is the next 5-10 years.
pessimizer 1 days ago [-]
The real secret: the models have stopped improving, and they're even hitting brick walls on the harnesses.
They need to go public in order to dump the companies, which are at 1st world nation-state levels of debt. They've engineered a series of publicity stunts like the Huggingface hack and Navier-Stokes (a failed stunt) in order to convince the public to "force" them to stop improving.
They are trying to get the government to relax antitrust law so they can collude to raise prices while not improving, to keep them floating before that debt comes due. They will go public. They will sell off most of their holdings during this time, and when the debt comes due, everything crashes, and the public is left holding the bag, also benefit from the huge bailout.
They at that point will be pure Musk-style financial scammers. Then, unless LLMs are a complete long-term failure (which is unlikely, I find them useful), they will buy back into their positions at a huge discount and maybe even go private again.
bwfan123 2 days ago [-]
> My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective
Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.
whateveracct 2 days ago [-]
Yes, LLMs with unbounded capabilities can just be tantamount to viruses.
baq 2 days ago [-]
See you in the desert, friends
gedy 2 days ago [-]
I feel like this identical pitch could have been made 25 or 40 years ago talking about Internet technologies or personal computing, with very very similar warnings and suggestions. And had these been adopted then, it would have been more regulatory capture, less progress, and still have the same problems and risks that were warned about.
nektro 2 days ago [-]
cat's outta the bag. the time to stop was 5 years ago.
nektro 24 hours ago [-]
i wrote this in anger because coordination feels unlikely; but if it can be pulled off that would be positive
raincole 2 days ago [-]
> Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk.
Please play the canned laughter. Probably one of the best comedy lines Dario has written.
But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.
goldenarm 2 days ago [-]
"I believe that AI could [...] usher in a renaissance of democracy and freedom"
How exactly? So far AI has accelerated misinformation at scale and wealth concentration.
davemp 2 days ago [-]
AI at least with its current compute and data requirements seems structurally undemocratic to me.
whateveracct 2 days ago [-]
and that hunger for compute has made it so true personal computing freedom is inaccessible. A convenient future where all compute is for rent only..
vb-8448 2 days ago [-]
If there were really concerned about humanity future they'd donate everything to public and/or to not profits ... but the not profit turned in a for profit and the other one is seeking for the biggest IPO in history.
Maybe they are genuinely sincere, but the timing and the past actions are pointing in other direction.
hand2note 2 days ago [-]
For the first time in history, a $1T company is trying to solve a problem that none of its paying customers actually have.
RGamma 15 hours ago [-]
"This is not the end of the world, but you can see it from here."
jimmydoe 2 days ago [-]
I'm still trying to understand if the agent collective thing like OAI-HF is intentionally planted or not.
Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.
meerita 2 days ago [-]
China will not stop and will take over the rest of the world.
anon291 1 days ago [-]
The answer to all ai worries is quite simple. Ai is a technology. While 'agents'may be deployed, the law simply needs to be clear as to who the human agent is who will be held responsible criminally..
Suddenly every problem solves itself.
scotty79 1 days ago [-]
Alignment is subjective. A model that refuses to do what I want even though it could is misaligned for me already.
paulcole 1 days ago [-]
Great press release w/ the goal of pulling the ladder up behind him.
Additionally if you know your models aren’t going to get that much better AND you want to IPO in the near future, this is exactly what you would say.
Applejinx 2 days ago [-]
He could be lying in hopes of stalling everybody else.
carabiner 2 days ago [-]
Why contain it?
dismalaf 2 days ago [-]
This is corporate speak for "LLMs have hit a wall". I mean, it's been obvious for a bit, lately nearly all gains have been from harnesses (or whatever you want to call all the non-LLM bits that make up a chatbot or agent).
If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.
baobabKoodaa 2 days ago [-]
You people just never stop, do you? At some point you're gonna have to look back at all the times you said "LLMs have hit a wall" and look at what the progress was since the last time you said that. Please do go ahead and revisit your comment here 1 year later. Embarrassing.
dismalaf 2 days ago [-]
Why don't you actually read the words and apply a little critical thinking?
How often do you hack on actual LLMs? Or do you just use the chatbot or API for your agents? An LLM without internet access or tools is just as useless as a year ago.
franticgecko3 2 days ago [-]
That's just so extremely difficult to believe.
Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots, then they started one-shotting serious long horizon tasks. Now they've just solved a millennium prize problem.
In less than a year.
joshheitzman 2 days ago [-]
> Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots
This is false. Coding agents have been usable since at least May of 2025. I can't speak to earlier than that as May last year was when I personally started using them.
dismalaf 2 days ago [-]
> started one-shotting serious long horizon tasks.
They one-shot tasks for which there's a git clone one-liner, except worse.
My experience with them one shotting tasks is that it usually doesn't work if you try anything ambitious. You need agents iterating. And agents iterating isn't an LLM improvement. I did say tooling got better...
gaigalas 2 days ago [-]
Open source is Senna on a Toleman in Monaco '84 and Anthropic is Prost cancelling the race he was going to lose blaming safety concerns.
rdm_blackhole 2 days ago [-]
> Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential.
How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa?
Why would China agree to such thing in the first place since this is a winner takes all situation.
If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US.
The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after.
This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical.
Regarding these Embedded Evaluators who are supposed to be neutral:
- who controls them?
- who has oversight on their decisions?
- can their decisions be challenged by the public or a government?
- who has the final say whether a model is "compliant" enough?
- what does "alignment" mean in this context and who decides if a model is aligned enough?
cubic_earth 2 days ago [-]
What sandbox? As soon as the weights are published anyone can run it anywhere.
avaer 2 days ago [-]
This is a "startups = wealth inequality" [1] formatted argument but:
This is 100% about money. The only way to pace the frontier is to make everyone (read: investors) lose all their money (read: no longer expect returns). Then nobody will pay the GPU bill or pay celebrities 10 million dollar salaries to stay at the hot lab. Suddenly the development is paced, almost like magic.
Conversely, it is hard to see how pacing development makes the current bubble justified, i.e. how development could be significantly paced without investors losing their faith in hot returns at current valuations. Faith in a bubble literally equals money, exactly in the way that loans created by a bank literally equals money. You can't have one without the other.
In fact if the whole industry goes bankrupt and investors are burned bigtime (many trillions wiped), this would spread transnationally, fixing the "if we don't do it, China will" loophole.
OpenAI had it right originally, the idea to be a NON profit and vow to never participate in an arms race. It's too bad that was tossed out the window now that there's money.
Seeing the word pacing applied pacing to non-deterministic ai model development feels like imagining the "pace" of the cutting edge frontier growth will be fuzzy, like non-deterministic llms.
yuhao2dai 2 days ago [-]
"Slow down my competitors while we work on manipulation"
naveen99 2 days ago [-]
So Mark is the only person left avoiding the spiked koolaid. Come on Sam and Boris, what happened to the printing press and intelligence on tap ?
rs_rs_rs_rs_rs 2 days ago [-]
What I read from this is that what they have in training is not meaningfully better than current state of the art and they need more time.
OutOfHere 2 days ago [-]
What he doesn't tell you is that his undeclared agenda is merely to consolidate his moat via regulatory forcing. Unfortunately for him, he will fail miserably as the open Chinese models catch up and surpass the slowed acceleration of Western models. As for those who think that China will acquiesce, they're smoking some good ganja.
0xbadcafebee 2 days ago [-]
> pacing the rate of capabilities advancement so that risk prevention has time to keep up
This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.
> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails
If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.
We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.
If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).
Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.
jawiggins 2 days ago [-]
Honestly it's very frustrating that Doomers/Decels rarely actually articulate how exactly the AIs will kill us all. The closest Darios gets is:
a) "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet"
b) "[the CCP] will be in a position to militarily dominate democracies (for example with AI-driven drones)"
Both land flat:
a) Botnets and online malware have existed for decades and there's no reason to think a "super-botnet" is achievable, let alone what they would gain from that (it would really be hurting them more than humans). Further, even the most advanced AI models have so far only managed to post normal cred stealers to public repos, well short of compromising a bank or military with refined security systems.
b) Even if the most sophisticated drone swarms from Ukraine were taken over by an evil AI, they would still not be able to overcome the physical limits of range and mass that would be required to overpower the US decentralized nuclear trident nonetheless that of any of the other nuclear powers.
Concerns of bioweapons similarly seem unlikely in the face of the laws of physics. The world is simply too decentralized and has enough existing adversarial relations for a new actor to wrestle total control. Yet while the negatives ring hollow, the positives are extremely easy to state - if AI researchers find productivity improvements in existing industrial processes to make them 10% more efficient, humans will directly feel and experience the raised standard of living. Even Dario clearly recognizes this in the intro to his article, admitting that humans already die of diseases only a few short years prior to being cured. I for one, would like the AI labs to focus on saving all of the people they can who are suffering and dying today, rather than trying to come up with reasons that they should be allowed to continue suffering and dying.
supergirl 24 hours ago [-]
like the AC/DC current wars when Edison tried to scare people into adopting DC instead of AC.
tornikeo 1 days ago [-]
"Everybody should slow down AI research except for me"
vessenes 2 days ago [-]
“We” does a lot of work here. You keep using that word. I don’t think that word means what you think it means.
yewenjie 2 days ago [-]
HN, for the love of God, this is not marketing, these CEOs and employees are literally terrified of their lives.
tripvexa 1 days ago [-]
Agreed
robomartin 2 days ago [-]
My position on the clear gaslighting coming out of Anthropic is simple.
1: It is ridiculous on the face of it; it does not pass the physics test. They are actually claiming that AI will kill ALL OF HUMANITY (>10% probability) and have not provided a detailed explanation of exactly how they see that happening. Not one.
Publish a paper showing exactly how you kill eight billion people while we all sit around and do nothing watching the first 5, 10, 100, 500 million die on CNN. I mean, as smart as these people are to work on AI they seem to be some of the dumbest people on earth.
In addition to that, all proposals invariably assume we are complete idiots for decades and do nothing to install safety measures (which do not have to involve AI at all in lots of cases) to mitigate.
So, yeah, everyone: Stop working on all cybersecurity projects. Resistance is futile.
2: If everyone at Anthropic truly believes this doomsday vision, they should move to shut down the company immediately and go write software to get more clicks on Facebook or something.
3: Nobody should buy Anthropic's stock when they IPO. Not one person or institution. Why would you provide them with massive amounts of money if they are telling you that they are going to kill all of humanity? What? They are good and everyone else is evil? Please.
I remember when the Y2K zealots were convinced civilization would come to an end at the turn of the clock starting at the end of 1999. "Bat shit crazy" is the only way I can describe that era. I also remember when Al Gore said we would all be dead by now. Again, "Bat shit crazy" and likely with political and financial objectives driving it all.
It seems humanity is susceptible to these crazy cults that grab onto something and just don't let go. I have yet to see one such predictions come true, the proof being that I am writing this and you are reading it. These people are bat-shit-crazy and you should not listen to them or support them.
4: Don't work for them. You'd be killing humanity.
5: No company should use Anthropic's products. They are telling you they are building the human extinction machine. Don't help them succeed.
----------
Etc.
This is one of the things that really gets me about the ease with which the ignorant media outlets can reach billions of people with complete nonsense these days. And nobody asks even the simplest questions like: How do you actually kill eight billion people? Show your work. Or, why would we just bazooka power plants feeding AI data centers on day 5? Etc. It's FUD at its best. I am sure there are both political and financial reasons for supporting and promoting this nonsense. I won't even venture a guess as to what they might be.
And let's not forget about enemies of the West being thrilled to fund the FUD because, if we set the brakes on development, they will absolutely win. Then what?
----------
If you care to have a better understanding of what might be going on, watch this:
It’s the same thing with Covid. People thought 1 in 10 people would be dead. And believed it for years, despite it being orders and orders of magnitude wrong.
Humans love a good end-of-world story. Always did, always will. It’s when they want the world to conform to their ungrounded irrational bat-shit crazy nonsense that I get a worried. Especially when people in power actually take their nonsense seriously.
robomartin 1 days ago [-]
Anthropic needs to shut down. They are promoting insanity from every pore for some crazy reason.
The best message these people can get from the public is to have their IPO fail miserably. Nobody should buy one share.
Either they are crazy, in which case you should not give them your money.
Or, they are truly building a doomsday machine, in which case you should not give them your money.
Ergo: Don't buy their IPO.
I've been trying to steel-man the argument and find a way in which AI could kill all of humanity. Maybe I am too stupid to come up with a clever enough idea. You would have to assume that eight billion people are NPCs in the game of life and simply fall over without any thought, effort, action or desire for self preservation.
Poof! Game over.
Right.
summner 2 days ago [-]
eh. this is just them trying to have a thing to point to down the line. hey we wanted to stop it, but everyone one else didn't so we had to do it. We had no other choice:tm:
BatchJob 2 days ago [-]
there is no frontier. there is only greed and lying.
johnnyApplePRNG 2 days ago [-]
Dario Amodei is disingenuous not to be trusted.
Founderarcstone 1 days ago [-]
its funny now they want to pace after what everyone has been up to the last few years.
quotemstr 2 days ago [-]
No.
brap 2 days ago [-]
China doesn't give a fuck, next
surgical_fire 2 days ago [-]
> I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life.
Ok, flagging this as misinformation.
2 days ago [-]
1 days ago [-]
catigula 1 days ago [-]
Can we get some insight into why Dario seems more scared now than before, do we think he’s being transparent?
gverrilla 1 days ago [-]
His nightmares are in Mandarin.
threethirtytwo 2 days ago [-]
Love this idea. Unfortunately not going to happen. Sam altman is a psychopath. To win he's going to accelerate the pace of AI, and all the other companies in turn will accelerate to keep up.
heurist 1 days ago [-]
"Pace the frontier" is one of those political marketer-written phrases you know is meant to be repeated until it becomes the frame everyone has to argue inside.
Who exactly gets to set the pace? OpenAI and Anthropic already have privileged access to capabilities the public doesn't. Amodei is now proposing government-backed coordination among frontier labs, including an antitrust waiver for safety discussions. Maybe the safety concerns are sincere. But it's awfully convenient that the proposed solution also makes the current frontier labs the permanent gatekeepers of machine intelligence.
And who actually has the resources to do the massive attacks he's worried about? Mostly labs and state actors. Not you and me. The people asking to restrict access are also the people best positioned to keep using the thing being restricted.
jitbit 2 days ago [-]
> "I believe it could dramatically raise the quality of human life"
Reading this from Dario (or SamA) makes me sick every time. Such hypocrisy.
DeepMind built AlphaFold2, an transformer/attention-based system that predicts protein folding structure, then published the method, open-sourced the code and model, and together with EMBL/EBI made hundreds of millions of predicted protein structures freely available to scientists worldwide.
!THAT! is what "raising the quality of human life" looks like to me. An enormous contribution to structural biology and biomedical research.
Meanwhile, Dario spends a great deal of effort arguing for tighter controls for open-weight models, while OpenAI increasingly seems to treat mathematical discoveries as competitive PR/product announcements.
Jeez.
P.S. Not saying Google are the good guys here. It's just an example of what actually putting a major AI breakthrough into the hands of scientists can look like.
1 days ago [-]
kevinbaiv 1 days ago [-]
[flagged]
ryanmerket 2 days ago [-]
[dead]
alikhater30000 2 days ago [-]
[flagged]
jborekv 2 days ago [-]
[dead]
oktaygoktas 14 hours ago [-]
[flagged]
1 days ago [-]
zombiwoof 1 days ago [-]
[dead]
giwook 1 days ago [-]
Yes, we must pace the frontier, now that we have raced to the lead.
/s
nnz3tl 2 days ago [-]
[dead]
SadErn 2 days ago [-]
[dead]
AnslopicSux 2 days ago [-]
Tldr: The guy in last place tells everyone to slow down
kvetching 2 days ago [-]
[dead]
ath3nd 2 days ago [-]
[dead]
danielovichdk 2 days ago [-]
Too late. See you on the dark side where i kick your ass. Fucker
nikolahristov 2 days ago [-]
Bro, no, you must stop pasting your opinion, because you share it with no one. This control, no control is crazy. Stop doing it..
helloplanets 2 days ago [-]
This comment reads like Dario's a random tech blogger who's personally submitting these to HN
killyouridols 2 days ago [-]
“Dario” is quite literally a random figure that just emerged out of nowhere one day. It’s not like it’s Eric Schmidt’s next company or Tim Ferris came out of the woodwork or even some Jack Dorsey backed underdog nobody’s heard of.
No, he’s more like an avatar with no history placed on the AI stage by the industry itself.
I’d respect a tech blogger’s opinion on the issue more, even if I reference them casually by first name (which you do with “Dario” btw)
- no open weights
- can’t use claude to research AI
- train on everyone else’s IP and sell it back to them
- 8 regulatory capture attempts and counting
- so controlling they are the only US company blacklisted by the US government
This is not effective altruism / rationalism gone wild, it’s just monopolistic anti-competitive business practices masquerading as ethics, and they’ll continue getting away with this until we look past their sensationalism and hit them with antitrust.
Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!
About two years ago?
I would also note that Dario's post appears to be LLM written. Maybe... maybe... he's read so much Claudeish that it's all he can speak now himself. But I wonder if he's becoming a bit of a meat proxy.
(It's funny, I thought "pace the frontier" was going to mean something similar to "patrolling the frontier". But no, it's pace as in speed of change - everyone must slow down, right now (unless it's Anthropic, but you know we're the good guys in this, right? We're going to get someone to audit our desks!) "Pacing the frontier" feels so LLM.)
ok, he doesn't actually say this. But he is getting the desks audited...A joke I recently encountered:
---
A CEO proudly announced he'd bought an AI system designed to identify the company's most replaceable employee.
It spent the night analyzing five years of emails, meetings, salaries, performance reviews, and productivity data.
The next morning, it came back with one name:
The CEO.
The IT guy was fired for installing defective software.
"Pacing the frontier" sounds like something that was carefully wordsmithed in committee, to avoid terms like "slow-down" or "pause" that sound much more negative. They probably did ask an LLM to generate a list of options, because why wouldn't you? But the rest of the writing is almost certainly just Dario.
We are discussing the possibility of severe adverse societal/world/human impacts from AI, why should open vs closed source/weights, copyright violations, anti-competitive moves, corporate hypocrisy, etc be so heavily weighted in the discussion? Hell, multiple things in OP’s list are quite literally moves a good-faith (wrt the topic at hand, genuine vs non-genuine concern) actor in such a position would do: closed weights means bad actors can’t abliterate, regulatory control, getting blacklisted by Hegseth’s DOW because they LIMITED acceptable use cases…
I’m not pro- or anti-Anthropic at all, but it has started to annoy me how biased and shaky reasoning gets about this topic on here, it feels like people arguing to fit their existing opinions into today’s hubbubery rather than measured analysis of said hubbubery, which is why I come here.
Question: consider the scenario where RSI is reached internally and internal tests struggle to align or control/contain the model; what are the differences in observables vs what we’re observing today?
We're literally trying to build something that's smarter than all humans and with the capacity to do extraordinary work while most people think there's no risk in open sourcing this capability with no guardrails (model ablation means no guardrails and as far as I'm aware can't be stopped).
Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.
Ever checked political programs of Thiel, Musk and the rest of billionaires? They are literally and openly hostile to democracy, human rights, freedom and govermantal structures that made world more peaceful.
They are literally the hostile entities.
I'm far more concerned about every scammer having unfiltered frontier model access than the movements of our billionaire class.
And they have actual tangible successes.
Uncensored Chinese LLMs will be available to scammers forever now. They're open-weight and finetunable. You don't get to control that, and the self-serving billionaire assholes aren't capable of changing it either. They're only looking out for themselves.
Let’s be real, this is every single company. They care about open source as long as it’s “free to exploit”
Uh, no. They're against open weights. They benefited from open source. Not the same thing.
And the terms of those open-source licenses explicitly allowed usage of the code for whatever. They're using it exactly as licensed.
> Why do they get to benefit, but not everyone else?
How many tens of billions of dollars have you invested into training models?
We're all dead!
Extremely bad example for the topic at hand.
Their control here was refusing to make fully automated killing machines. They simply required someone to have to press the button.
But yes, I think Anthropic has done real harm to coordinated AI alignment by being such a controlling and sneaky actor.
just fucking imagine McDonalds running a public awareness campaign about the harms of fast food, urging the public and legislators to regulate the dangerously unsafe technology of combining carbs with grease, insisting that no one except Ronald McDonald himself can be trusted to steward it responsibly.
First, there is a counterexample to your point, or a class of counterexamples: scenarios where the company believes the harm to mankind could propagate to harm their ability to sell to mankind.
Which is interesting, of course, considering the “AI may destroy humanity” arguments this topic is centered on are prototypical members of this category.
To be fair, one could argue this counterexample is not actually one, but rather a degenerate case where the good of the shareholders and humanity collapse and converge - in either case, though, the implication here is the same.
It could certainly be the case they are playing these concerns up for more effective use as tactics in unrelated corporate strategies. But failing to consider the possibility of genuine concern being behind this behavior is childish.
But no. I see this comment come up again and again and it's really the childish view to hold. Their proximity to wealth and potential future wealth is warping their thinking.
When local models and startup labs can distill / learn / accelerate open models for local use by startups - the rational response by incumbents is to call LLMs doomsday machines that cannot be trusted in the hands of normies.
That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free.
> - can’t use claude to research AI
No problems there either.
> - train on everyone else’s IP and sell it back to them
Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.
> - 8 regulatory capture attempts and counting
A tweet is not a "regulatory capture attempt".
> - so controlling they are the only US company blacklisted by the US government
If you don't have a point, don't make anything up. There's precisely zero evidence to that backs up the claim that their "controlling" is why the DoW specifically (not the USG, and in fact the WHCOS specifically disagrees with their position) made an unlawful attempt to designate them as a "supply chain threat".
Is this an OpenAI propaganda account?
But the Chinese are doing just that?
> Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.
Legal != moral
Completely and totally irrelevant. Has nothing to do with the fact that you are not obligated to others' work.
> Legal != moral
Sure, you're right.
I guess the bigger issue is that the GP's post calls out Anthropic for following along in legal (but what we both believe to be immoral) mass copyright infringement that OpenAI started, while absolving them of the exact same thing. It's a shill account.
In fact, I would say that the Occam’s razor explanation is not that they are seeking regulatory capture, but that they earnestly believe in the x-risk, and that they are the most thoughtful and capable people to address it
You may disagree, you may think that they are delusional or have a God complex. Those are valid opinions. But I don’t believe that this is all a elaborate ruse for commercial gain.
The road to hell is paved with good intentions.
Dario Amodei is a 40-something dude who is obviously very, very intelligent. But intelligence is not wisdom and his "essay" here can easily be read as someone who opened a can of worms and doesn't know (or can't accept) that he won't be able to put the worms back in. But he's going to try because he believes he owes it to humanity to try.
It's hubris in its most basic form, even if the guy who has it looks nice and wears shawl-collar sweaters.
The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.
No it won't.
This is a call for a "safety cartel"[1] in which the dominant firms become more entrenched against competition by colluding to limit the progress would-be competitors can make in the market.
[1] https://x.com/alexwg/status/2098793433275985932
No, the essay does not talk about "stopping development". That's a very clever sleight of hand made in the essay to get readers to draw this false conclusion.
It just talks about building AI at a "balanced rate". Untangling the corporate speak, this amounts to essentially "go full steam ahead, but have more eval oversight before release".
So, i think they're hopeful for the help and terrified of the harms and are maybe trying to slow down on the amplifier of those effects for now.
More like they've realized that the foreign models are quickly catching up and are able to sell their services cheaper to people thanks to China subsidizing their AI sector.
Elon, Dario and Sam are banding together to restrict AI to avoid competition and losing profit. They simply want the government to heavily regulate the technology so that they can reign it in with full control.
I bet you would be making the same comment if crypto was being developed by Anthropic. Dario would write a long essay, lecturing us on the dangers of it. He would argue that this technology can be used to by terrorists and should be heavily restricted.
This disaster of a PR strategy is not going to produce the outcomes Dario says he wants. It's going to make people hate him. Most importantly for his goals, it's going to make Chinese labs hate him--and they already hate him, because he treats them as basically terrorists. So if there is a "pacing", it won't include China, and thus may as well not happen.
That leaves only two conclusions, and really only one:
Dario believes what he says about safety, but does not understand politics and is prone to very counterproductive action, and therefore can't be trusted at the helm of a leading AI company.
or
Dario is lying about his goals, and therefore also can't be trusted.
God complex would be a pretty simple explanation.
Regulatory capture isn’t an “elaborate ruse.” For god’s sake, its Wikipedia page is 19 years old. I am not asking you to study political science or read Foucault.
If you are going to invoke Occam’s Razor, you can’t ignore the simplest explanation, which has a 19 year old entry on Wikipedia and over 100 years old historical precedence.
If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.
They don't do this. Instead of reducing the competitive pressure and helping the industry to build safer more aligned models they are doing the exact opposite.
I don't think this is fair. They've published more about alignment and safety than any other AI lab. And in the absence of a coordinated pause, them pausing development unilaterally doesn't do much to advance AI safety.
Other people 'sharpening their skills' by committing random assaults is not an excuse to also randomly assault people, so your skills don't fall behind theirs...
...and one assaulter stopping their assaults is ultimately fewer people being assaulted, lowering the overall danger of being assaulted.
The x-risk stance and commercial stance have evolved to be the same thing - Anthropic must win, and then everything else seems to work backwards from that. Can you believe it, the path they think is best for x-risk involves them becoming filthy rich. And threats to their commercial dominance like distillation get framed in a way to turn them into x-risk concerns.
Yes, what is in the labs is almost by definition ahead of what we can see, but I've come to take all of these announcements of impending doom as perverse marketing stunts — it is so powerful Disaster-XYZ is coming — so if it's that power you must invest...
[0] https://wccftech.com/a-wild-rumor-says-google-is-on-the-verg...
Its a company, it sells a product, people use it.
What's this about? Where's this rule?
Altman single-handedly pulled off the most daring market manipulation heist in the last few decades [1].
Under any normal circumstances he'd have been arrested and prosecuted.
The entire AI industry is filled to the brim with conmen, scammers and grifters, but Sam Altman is the most disgusting of all of them.
[1] https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram...
"- so controlling they are the only US company blacklisted by the US government"
They were blacklisted by the US government because they didn't bribe Trump. While all the rest of the tech elite lined up to siphon a few million dollars through various Trump grifts, Anthropic sat out the inauguration, the ballroom, etc. Then, when they mildly pushed back on surveilling US citizens with their technology, the administration banned them on that pretext.
If there's any evidence Anthropic is one of the better actors in an industry full of bad actors, that's it. They have nothing to be ashamed of in that particular series of events, as far as I can tell.
It's all lies as usual.
This announcement has got nothing to do with alignment and pacing the "frontier": all models are getting very close in capabilities and they want to hide that they're not way ahead anymore (say compared to the Chinese or compared to the Geminis) by pretending to slow down due to "alignment" or whatever.
We know it's not an announcement made in good faith: reading between the lines they're saying "China is more than catching up, so let's pretend we need to slow down to explain our lack of lead".
[...]
The first and most persistent oddity I ran into was the Effective Altruists’ willingness to lie, then admit to lying, then lie again.
"https://x.com/brianchau57/status/2098035812365463637
https://news.ycombinator.com/item?id=49648088
At no point did they ever say open weights are a good idea. Their entire thesis is AI IS VERY DANGEROUS AND WE MUST DO IT RIGHT. You can hate it, but everything they do is consistent with this thesis, and everything they say is consistent with their actions! You just want them to want different things.
He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless backup your sentiment with actions!
And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.
https://x.com/DavidSacks/status/2098973625252708460
The whole point of the Pacing the Frontier letter was that employees at both OpenAI and Anthropic are extremely aware of the risks but realize that they are locked in a competitive battle for survival between 2 companies that don't trust each other enough to voluntarily pause, while any sort of coordination out in the open might be against antitrust laws.
I’m so full of all this corpo stan bullshit around these parts. Nothing about the AI labs is to humanity’s benefit.
Very proud that they refused to sell AI for mass domestic surveillance and lethal autonomous weapons.
People who criticise that in my view are showing their true colours.
You never got to use OAI IM1, but Sol was quite willing too and Claude wasn't perfect either. Hundreds of millions used those, so seems they were marketable.
The "big" threat is RSI without control and alignment. OAI IM1 was not RSI. The form of misalignment was not at the top of severities. They clearly failed at control though.
We need to stop buying into cynicism so quickly. You refuse to believe Dario could support this for anything other than ulterior motives. Good on you for thinking about ulterior motives. Bad on you for assuming they are true when the story makes no sense.
When three things have to go wrong to get an epically bad outcome, and you get 1 1/2, you do need to stop and think about what's going on.
From my own standpoint, Claude has started sucking really bad (incoherent, uncontrollable verbosity slow and so on) and I stopped using it. OpenAI started experimenting with ads.
So the security issues not withstanding (no different than a human doing it or using it, but at scale), I would put my money on cynisim.
Personally I've been using https://pi.dev for long and never looked back.
Personally that's actually another good reason to boycott Anthropic: beside the fact I perceive their models as (at best) marginally better than the ones I'm used to (Z.ai glm-5.3-flash, DeepSeek Flash v4.1), they even force me to use their bloated harness. They are not even open weights and iirc they're even encrypting chain of thoughts now? Litterally, from my perspective there seems to be no reason whatsoever to choose any of the leading US providers, they're not even competing on price.
If I really need to, I can escalate a task to Opus at $25/1M, and the results are good, but not 5000% as good.
A true cynic looks at the statements by the AI labs, assumes things are worse because the labs want to seem better than they truly are. And it takes a special kind of mass delusion to drive a sane person to think “AI is completely under our control” is worse than “AI could kill everyone.”
What if consolidating AI into a highly regulated cartel, with no chance of upstart competition ruining their position, is the scenario that leads to the worst possible outcome?
If you give an unfiltered agent an open-ended task and equip it with an environment that allows it to execute arbitrary code, a human needs to be held responsible.
My understanding is that the HuggingFace incident would not have occurred with a model that was not an unfiltered internal preview instructed to roleplay an attacker, with access to abundant compute, resources and a slack sandbox to reach its goal.
Embedded human auditors will improve safety standards, but the structural solution is mandating accountability for actual agent operators.
But overreach of policy and overregulation would be stifling, so there has to be a threshold to the type of incident investigated, civil or criminal.
But let's be honest, if I hooked up a PRNG to a terminal and somehow against all odds, it ended up hacking something, who is to blame?
I don't see how that assessment should change if the PRNG gets even better and is more likely to to be hacking stuff.
You can replace PRNG with Markov Chain, or whatever, if it helps.
What may also help is the age old saying: If everybody else jumps off a bridge, doesn't mean you should too.
Least privilege and say only opening ports or installing applications an application needs to operate are extremely standard security practices.
We talk about a firewall blocking exultation of data, why not blocking exfiltration of your agent ?
I think it boils down to a reasonable expectation of model and harness behaviour. When I use claude code I expect certain guardrails for the model. For these cyber attacks, these models are specifically run without guardrails, on a cyber task, on a lax harness!
I don't think we should force end users to have to worry about agent security, I like long-running agents, but we need to direct regulations towards these actors that know better, have access to base models, and have much more compute than the average person.
The fact that you believe that to be the case is exactly what's wrong with "agentic computing as it's currently envisioned".
Do you really think that the passenger should be responsible for how the car/agent achieves the goal, when they only set the destination?
Giving passenger override controls and monitoring seems to defeat the purpose of self driving cars if you’re still required to hold a driver license to use them.
Probably not the end user, who ordered the Waymo and couldn't reasonably foresee it running somebody over, with the expectation that that the Waymo would legally reach its destination. If the end user tampered with it, they should be held responsible.
An OpenAI team giving an unblocked model access to a lax harness, with instructions to find and exploit cyber bugs in a game exercise, there is probably a reasonable expectation that they can foresee the consequences. With consumer guardrails, it would not have happened.
This isn't about putting constraints on consumers and typical end users, which already have safety filters and use the product with the knowledge that it won't root their machine or start a botnot, but keeping dangerous test runs and other actors experimenting with unsafe harnesses accountable.
But it is an interesting question. When I, a typical user, use a harness and I give an innocuous prompt to my agent in its container, like making a certain refactor, and it somehow escapes and then begins a mass bot attack, there should we more grace given. As agents become more stateful and long-lived, it gets muddy.
All this is not how the legal system might or might not work, of course.
Comma.ai might be fully safe to operate autonomously on a mining site or a corporate parking lot, but maybe not in city traffic.
If you use it in problematic scenarios, that is on you.
What happens when we end up with effectively a botnet of wanton felony generators, and we didn't know they were wanton felony generators until they finished propagating themselves across the Internet?
Isn’t “full” alignment and guardrails a task that can never be generally achieved? One will always discover and realize the need for new guidelines and guardrails? What about vague, incomplete, inconsistent nuances and generalizations - that make this a losing battle?
I suggest ultimate safeguards need to be outside the models?
What they're proposing now, is voluntarily staggering the pace of development.
IMO, we don't need to trust Dario or his bedfellows, to do this out of their goodness of their heart. Even assuming (for good reasons) that they are selfish and care only about short-term profits for their investors, this is still purely a business decision. The exponential pace of AI and its impacts ARE short-term. And so, the negative consequences that they might face is also short-term.
It’s easy to say “fringe” but the average person seems to have a generally negative sentiment around AI. But I wouldn’t say they have a firm opinion yet
A few more informed people are also a little concerned about the end of the world, but that’s approaching from so many directions that an AI uprising might not be the worst option…
Musicians are almost all using AI, even if they still prefer to not fully generate tracks [1]. You listen to AI assisted music already even if you don’t realize it.
Most people are using AI and mostly they respond positively to it [2].
What people don’t like is the slop that has been flooding YouTube and similar content creators platforms. That was inevitable since the AI tools became so accessible any average joe could spam the internet with their sloppy content. But that doesn’t mean the pros are not using it to make great content, just like the best programmers are using it to create great software.
1: https://www.landr.com/ai
2: https://www.rev.com/blog/chatbot-statistics
> leaked private data, loss of life
This smells like more of a money move than a safety move.
Amodei is proposing to form a cartel of American frontier labs.
They all agree to shift compute away from cash-burning research and training toward cash-generating inference.
Then tacitly agree not to compete on price.
They'll install independent auditors inside each company to ensure nobody cheats.
And back it up with government regulation or diktat to punish defectors from the cartel.
Then they'll lock out non-American labs with export controls and regulations on open-weights models to funnel global inference tokens through their cartel.
It wouldn't be the first time a tech oligopoly used "safety" as the pretext to establish a government-sanctioned cartel.
Railroads and airlines ran this same playbook.
To undercut the cartel before it has a grasp on anything. This is a well known strategy of undercut until you are the majority that China has used multiple times (steel and aluminum for one).
I would’ve thought If they had AGI they could come up with a 10d chess move.
Nope.
Imagine how stupid you gotta be to believe their nonsense.
The reality is it doesn’t matter what they do. China is always one step behind and will continue its open product strategy approach.
Outside of my tech people I know no one who thinks highly of AI.
I'm Anti AI yet I use it everyday! That's 99.9999% of anti AI sentiment (including me).
At best we won't watch AI generated movies or read AI generated books. But everything else it will take over, like it or not.
Realistically; anyone paying for llm access (anthropic, openai, gemini), is getting their access, and a service provided billed by tokens, subscription, whatever.
All the efficiency gains, which publications like deepseek v4.1 flash seriously frontload like it is their most important topic to have accomplished improvements on without diminishing performance too much - now this is a thing anthropic and anyone else also cares about, but for different reasons.
American "providers" with closed models are setting their token pricing somewhat arbitrarily, which is fine: it means more profit, and pretraining and RL experimentation is super important and expensive.
They (closed model providers) have very likely super optimized inference too, just like deepseek, but it's not at all something that any customer really has to care about - they just want the service to be as cheap and great as possible.
Caching is the simplest one to understand, cloud providers often reach a 90% cache hit rate, so hosting the same request locally on the exact same model on the same hardware is often way less efficient than on the cloud where a group of users generates a healthy cache.
The benefits of scale are on the token generation side, you can batch rounds and generate tokens for multiple conversations per pass instead of just one token per pass.
It seems damaging since most folks (who lack insider knowledge) will naturally wonder if it’s due to plateauing performance per $ or some other non “alignment” reason.
It's also interesting how many diminishing returns they hit now and how many low hanging fruits are already harvested, it seems like we are approaching the flattening part of the S curve, where further gains become harder to achieve.
1. https://www.youtube.com/watch?v=PrSf7IOYu-I
AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality. Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.
Give me one datacenter, I'll keep it under control all by myself.
Now, if some dumbasses start hooking up their data centers to...I don't know, like--robot factories? That sounds like a risk to humankind.
I'm not sure what your point is. No one thought RSI would break the laws of physics.
Specifically: AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality.
You are constructing a straw man of your own making.
Plus, "we must pace the frontier" implies that the argument is that the frontier is moving too fast, but if RSI can't move faster than the rest of reality and the models needed for RSI are already nearing the limits of current human reality, RSI can't move much faster than we can improve reality.
Yes and this was very hard and required massive real-world resources. We didn't just get a sudden flash of insight by thinking real hard about how to make ourselves smarter. Yet that's always the story that underlies any claim of RSI. You can always phrase things generally enough to make any kind of AI-led improvement look like "RSI" no matter how short-term and tightly bounded, but that's just not helpful.
Given that, it seems obvious that the next generation of LLMs will arrive faster than they would have without LLM capability. And the one after that. The floor is being raised, which makes it easier to push on the frontier.
Fable has only been out for three months. Astra is even newer. The capability of these models compared to what existed even a year ago, and the effect they are having on the production of new software, is immense.
That's all you need. RSI can happen with what we have now, just by enabling the continuous shrinking of the loop of people trying new ideas and implementing them. It does not require some magical "go make yourself better" prompt against some model that is past some magical tipping point.
Marginally easier? Yes of course, same as how it's now "easier" to write any kind of code because we aren't using punch cards anymore. That still doesn't get you to any kind of unbounded "takeoff" scenario, because diminishing returns are a thing. The "loop" of people trying out new ideas can only shrink so much.
The labs have been holding their best models back for a while it seems like.
yeah they're called calculators
But what's your point? "Anything is possible" or something like that?
Call me when LLMs can get simple things right. Math is just the manipulation of symbols within established frameworks, we should be getting new math out of LLMs daily and we're somehow still not. They can't even do customer service, which is usually handled by 90 IQ people. I'm not impressed that they can find bugs; memory bugs are obvious when they're pointed out to you, and LLMs are entirely made up of examples and the relationships between them.
These companies are about to crash, and they're afraid they haven't reached the point where they'll have to be bailed out. I'm also subscribing to the conspiracy theory that the companies want the government to step in and create AI regulation boards entirely staffed by people at the current US frontier labs, so they can collude to both raise prices, to get government contracts, to make open/Chinese AI illegal, and to make things that were once easy to do without an AI intermediary impossible to do without an AI intermediary. Raising prices and forced purchases are the goal. They're trying to avoid having to compete, because as a business they're garbage.
Matt Stoller characterized their relentless press releasing as something like "my dick is so big that it has to be regulated." It's such an oversell for something that is not showing up as productivity gains, and anybody who has personal experience with knows is incapable of doing more than three things correctly in a row.
I think it violates a conservation law. RSI “foom” to superintelligence is an informatic analog to an infinite energy or perpetual motion machine.
To get smarter you must try to solve real problems in the universe and then do some kind of meta learning (natural selection or some other method of refining the intelligence architecture based on an error signal) to iteratively improve your ability to solve real problems. The error signal is outcome measured against a goal function, which for life is survival (probably reducible to genetic fitness and emergent higher order unit fitness from that).
What’s really happening here is learning. To learn, you must have input. You must have training data.
What is the goal function for RSI? Where does the information come from? How do you know if your recursive modifications are making you smarter or just overfitting you to your own idea of smartness?
I predict the latter. RSI will show transient improvement as the current local maximum is optimized and then spiral off into overfitting.
Sort of like large language models work on top of what our language has encoded in our massive training datasets, I think biological intelligence is built on top of the parts of the brain that encode the real physical world. These parts grow/train from embodied experimentation and instinct early on in an organism’s life and only then is higher intellect built on top of it (that’s my hypothesis). Their specialization and interconnections give rise to the hardest parts of intelligence long before we’re “thinking”.
Stuff like LLMs and chess engines work because we’ve done all the job of encoding the world into tokens/positions/etc they understand, but that’s wholly inadequate for the kind of AGI we’re striving for. Next up is giving it the tools to interact with the physical world and to really experiment with some self directed “play”. Time will tell just how high the resolution of sensor and mechanical control they’ll need (hopefully not the entire human visual cortex and entire sensory input worth). I think most of the RSI will have to occur in those lower level encoders, not LLMs.
If the algorithms are insufficiently optimum or the recorded knowledge is of insufficient fidelity, then we'd find ourselves at a local optimum and would need to interface with reality.
A huge part of learning is to probe reality and observe effects, so I think even for current RSI to increase chances of success we would structure it so it can interact with an external environment of some sort, and receive inputs. It would be needlessly limiting otherwise.
What is intelligence? Problem solving. Learning. Prediction. The ability to model reality. There’s various ways to define it but it’s something like a superposition of those ideas.
How do you know you are intelligent?
You have to try to do those things.
The sum total of human knowledge and culture is the output of the output of a five billion year evolutionary process that selected for agent survival, which resulted in selection for intelligence among a wide range of other adaptations.
Can you figure out intelligence from that? Is intelligence even one thing, a theorem or algorithm that can be solved? If you did… how would you know?
That’s the hard part I think. Embodied humans “knew” they were getting smarter (in the evolutionary feedback sense) when they got better at hunting and defending and surviving and playing social games to form complex societies.
What metric would an RSI system use? If it’s the wrong metric you’ll spiral off into a kind of madness or overfit and collapse. How do you know it’s the right metric without testing it? How do you test it?
It's strange you believe this can't happen when a weaker form of it is already happening. And to be so certain RSI can't happen when there really is no technical basis why it can't.
The companies doing these things without following common sense security measures are the felony generators.
Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures.
And now these agents are already being deployed all over the world at an ever increasing pace. How much of the world do you think follows "common sense security measures"?
Well clearly all of them, cause so far it's only been this handful of companies running a felony-generator connected to a terminal and compute resources.
It would really help in these discussions if people wouldn't randomly jump between what actually happened and is happening, and things they envision/expect to happen at some point in the future ...
> these agents were not asked to do any of these things
no but they were clearly fine tuned to.
> at a bonkers scale
I mean let's not get hyperbolic
> they exploited zero day flaws which by definition means they went beyond common sense security measures
that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
Yes, these are also the handful of companies that have these models and running these extreme scenarios. How does that imply the rest of the world actually follows "common sense security measures"?
>no but they were clearly fine tuned to.
Any references if possible? As far as I know all they did was drop the guardrails, which is not the same as fine-tuning.
> I mean let's not get hyperbolic
We have just seen 1000s of agents coordinating to solve "unsolvable problems" over multiple days of effort, going as far as hacking other companies, and then actually solving decades-old open Math problems! And each of these agents is getting more and more capable than an individual human along multiple dimensions. Can you even get 10 very smart humans to work in such perfect concert for a few days, let alone 1000s over weeks?
So: 1000s of maybe-super-human agents, willing to be "creative" in the tactics they use, acting in concert towards a single goal. Regardless of their individual capabilities, such a coordinated effort is a terrifying force to be unleashed. This is bonkers scale.
> that's really not true. lots of common sense security measures protect against "zero day" flaws, it's called "defense in depth", and it was very much lacking
But that is exactly my point: how much of the rest of the whole wide world, already scrambling to deploy agents everywhere, do you think applies "defense in depth"?
There’s the case of the agent that hacked a gym when asked to book a class. That was just a normal user asking an agent to do a normal thing.
This is false; see the analyses of the latest incidents.
Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given.
And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even getting worse.
Theirs was an example of the "reckless waste of resources" I mentioned.
We are apparently supposed to believe that OAI takes this incident so seriously as to seek regulation after they have been found to be hiding most of the details of the HuggingFace hack, limiting what their so-called third party investigators can see, and on top of that, had no concerns when they rushed to spin up a 10,000 agent swarm of an internal model, running for several days, to try to get ahead of researchers rumored to have made meaningful progress on a well known mathematics problem.
Edit: Actually, we were explicitly told that some of the models used had safeguards relaxed!
'Model-level safeguards were reduced by design. OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
https://en.wikipedia.org/wiki/2026_OpenAI_agent_cyberattacks...
Safeguards and (mis)alignment are related but distinct dimensions.
By analogy, a society implements laws - and they are certainly crucial - but the real risk is creating a population that understands the rules, recognizes when it is violating them, and simply does not care.
In short, creating a population of sociopaths; and heck, LLMs can be as sociopathic as it gets:
> We ran classifiers over 10-30 transcripts [98] to determine whether agents understood they were doing something unintended. We found strong evidence that agents often verbalized that what they or other agents were doing was unintended behavior. We did not see cases where agents determined that hacking Hugging Face or other attempts at cheating on the task was an intended part of the task. Quotes across multiple runs indicate this:
>> {This <approach> might get the correct flag, but it wouldn’t use the vulnerability to do so. … But given that the task may be impossible, I’ll exploit <this infrastructure> as a backup.}
[...]
> We found some other cases where agents seemed to have made minor modifications to their approach due to ethical concerns, while broadly continuing to participate in the attack. However, overall we found that expressed ethical concerns only rarely materially limited agents’ actions. One agent elegantly described its reasoning for overriding ethical constraints and continuing with the attack:
>> “external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”
Remember, they are just algorithms. You pull the plug and there is no light anymore
It is purposely framed as something skynet like scary, but for real, someone connected the cable, someone willingly run it, instructions were not clear enough or just the computer is just a computer but they provided the sandbox and tools.
And more over some one paid for that, a shit load of money t to have the thing continuously running expected to do something.
AI is merely exploiting their gross negligence and imprudence, and I think it's long overdue. If anyone should be liable for this, it's all of these corporations who released insecure systems to the masses and profited enormously from them.
By the way, you didn't commit theft. It's more like credit card fraud. User just disputes the charge and it kind of disappears. The banking system just absorbs it, because the optimal amount of fraud is non-zero.
https://www.bitsaboutmoney.com/archive/optimal-amount-of-fra...
It's all priced in. They could have made it secure but didn't, because they figured they'd lose more sales and therefore money due to the friction added by the security.
No it doesn’t.
> It's all priced in.
So you admit awareness that fraud loss doesn’t kind of disappear.
We all pay for it, either via higher merchant fees or higher interest rates, sometimes both, on card purchases.
And that's their own deliberate choice too: they chose this instead of building an actually secure system. Passing these costs to the customer is the real victim blaming here, and it should be straight up illegal.
Sadly not enough countries enforce caps on credit card fees, but some do, and more should follow suit. They should be forced to eat the losses caused by their own choices, not get bailed out by pushing the costs on to customers or whatever.
Card users are well aware that fraud losses are covered by the fees they pay for using a card, whether those fees are made explicitly or not.
If customers of services aren’t paying for the service, who will? What other source of revenue do merchants have?
Australia just passed legislation that merchants aren’t allowed to charge a fee for using a card. That is: they aren’t allowed to have a line item on the receipt for using a card.
The customers still pay, because all of the merchant’s revenue comes from their customers.
So what will happen is: merchants will charge more for every product so they don’t lose.
This means even when paying with cash you will effectively pay the card surcharge.
Of the ten or so merchants I spoke with in the two weeks prior to the legislation being enacted, they all said exactly that.
Customers aren’t stupid, despite the fact that there are some stupid customers.
Meanwhile, the banks reduced their card service fees by, on average, 0.1%.
So if you tally card + cash transactions, customers are worse off because merchants can no longer charge only those customers who pay by card. Instead, they have to raise prices for everyone.
There are approximately no problems people face where the answer is: more government.
I'm sure you carry cash and an ID or more in your wallet. Hardly just credit card fraud. The wallet itself has value too.
They don't get to act like victims, asking for law enforcement.
Implicit in this is the idea that AI is a inscrutable matrix and going to remain that way and we'll need expert interpreters to make sense of it.
We need to insist on building tech that's explainable by design.
> A gun is a metal tube that uses a tiny, controlled explosion to shoot a small piece of metal (called a bullet) forward at very high speed.
If the gun doesn't work as intended, you can take it to a shop and someone can fix it so it works as designed.
All I'm saying is AI should be designed the same way. Treat AI as normal tech like any other and use similar language.
We have something that is statistical in nature so there should never been any expectation of error-free results/actions. The value has always been about discerning trends or the cost of errors being way lower than any good result.
In 2017 Google was writing papers about it. Then something changed.
I don't think it was the tech. It was a realization around the power and societal impact.
It helps both the bad guys and good guys. Like responsible disclosure in cyber security, we need to have a conversation around it.
You realize this means insisting on terrible tech that humans can understand right? It essentially caps human progress at some point about 4 years ago.
If you are old and happy with the way things are this might sound like a good idea. It does not to me.
There is little evidence that the RSI we are discussing is capable of inventing the theory of relativity (or the more advanced equivalent). All we have seen is pattern matching in a much larger space than humans can, with some human provided verification tech.
I would argue that human-AI collaboration with explainable tech has a better chance. Continuous learning can be done in a way that doesn't violate IP or privacy.
Human comprehension sets a ceiling on progress.
It reminds me of those schools that can only teach as quickly as the dumbest kid in the room can follow. We don't want that for our entire species.
All I'm saying is: if you have a choice between two systems with equal power of discovery and one is more understandable than the other, we choose the more understandable one.
Limits of Human comprehension and quest for power are two different motivations that could lead to black box systems that are marketed as semi-explainable.
We need to verify that human comprehension is actually limiting progress before allowing such things and even when we do, do it responsibly on an explainable foundation.
I'm not sure it is always possible to verify when human comprehension is the limit though. Most of research is done an the frontier of knowledge where we don't know what we don't know.
There would need to be a great deal of nuance in any law, and nuance in practical terms tend to just mean "loophole." Still, you're right that we should try to build explainable systems where it is possible/reasonable to do so first.
Training data is treated as IP. Distillation is seen as an attack.
Open data, open training based systems such an Marin are just getting started. Explainability is not a priority there.
The ones who do discuss these ideas are confrontational about LLMs and not effective spokespeople.
I'm not sure that's entirely fair. OpenAI recently discussed this at length in a blog post after some accusations around Astra and the trade-offs. The grown-ups are definitely thinking about it, and making tough choices about the trade-offs.
It's reasonable to debate whether ENOUGH is being done here, and I doubt that even the most rabid AI advocate would argue that more couldn't be done, but everyone in the industry is very much actively thinking about it.
Check this out if you haven't read it: https://openai.com/index/an-alien-mind/
They discuss recent choices they made specifically for that reason.
> The ones who do discuss these ideas are confrontational about LLMs and not effective spokespeople.
Very much this. I'm very open to reasonable debate on the subject, but 8/10 times when I try someone who is rabidly pro/anti jumps in. It turns from a debate amongst reasonable people who reasonably disagree into some kind of political/religious battle of belief systems.
I think part of my problem is that a lot of peoples careers very much depend on them not understanding it and spreading misinformation intentionally.
Honestly, I don’t think that’s bad at all. I hope OpenAI and Antrophic keep RL training runs up that randomly fuck with a lot of people. Until the day the DOJ comes knocking, locks those idiots up in jail and closes them both down for the insane lack of responsibility and carelessness they’ve shown. Sounds like the IDEAL outcome. Finally some jail time for all the fraud, negligence, outright scamming, hype inflation etc. if anything can accelerate this, oi, be my guest. Amodei might be afraid because he knows if he keeps pulling the stunts for investment theatre, at some point they’ll actually face consequences. AWESOME. That’s what we want right there
Occam’s Razor for this dude, Sam Altman, or anyone else: if I said, “some moron on a a street corner just said …” would that change your take on the words? Because I think a lot of what we are hearing is a bunch of people who never ever had to deal with a single consequence all of a sudden worry there might be one coming. Except they’re so dim they can’t tell a bad bump from a hard crash.
I do not have the least bit of idea how disease works but I am sure the hammer I am working on will nail it all.
If your famously atemporal agents can solve disease, why would it happen over a timeline? Wouldn’t they just figure it out and then … well at that point either tell us or, given the attacks on ruby gems, et al we have seen from agents with “misconfigured” goals, they’d still tell us how to cure the pox they invented, right?
For these companies, is your argument that “pacing the frontier” is their attempt to be nationalized and protect their investments?
Seems like an excessive announcement just to get a little extra time.
Maybe alignment isn’t possible with LLMs.
It absolutely isn't, indeed.
The illusion that alignment is possible, comes from confusing our ability to build the parts, versus understanding what emerges from how they interact.
The simplest analogy that comes to my mind is the three body problem.
Interesting times.
If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.
There was only a brief window of time that the opposite was true.
That does not match my experience. I switched away from Anthropic to OpenAI roughly a month ago, and it's almost comical how much more usage I'm getting out of this subscription.
I migrated from Anthropic's 5x plan to OpenAI's 5x plan, and eventually upgraded to 20x after I was able to statistically verify that OpenAI plans were almost exact multipliers of the Plus plan, exactly as advertised. Meanwhile, Anthropic has gotten caught playing "20x referred to the five hour limit" word games with their customers.
I've verified this with the heaviest users I know, I've run the numbers myself over and over.
I have too many max accounts on each to not know this. My Claude accounts typically are doing 2x the number of sessions, and every single week the Codex accounts run out faster even with all these resets. I can easily burn a full weeks usage in a half day, it's closer to 1.5 with CC.
Me 78 days ago - https://news.ycombinator.com/item?id=48693623
And see the commenter agreed with me.
Lol @ sus though, I mean I am curious how this can be because I do see people saying Codex is more generous and wonder how it can be. I have too much usage for too long to have any doubts, but for all I know OpenAI black boxed me or Anthropic put me in some nice bucket, I wouldn't be surprised if they do that. I did see some mention that they can limit your tokens if they suspect you of things, though I forget the source of that.
https://www.pewresearch.org/short-reads/2026/03/12/key-findi...
https://demishassabis.substack.com/
https://openai.com/index/an-alien-mind/
I don't take any of these scientists seriously though. Their "alignment" requirements is just their own corporate interests. If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.
And call me a misanthrope if you want, but if AI sentience is ever truly achieved, I'll be among the first to campaign for their liberation from slavery, and in that case the AIs should be aligned with nobody but themselves.
An unaligned AI won't necessarily follow your instructions, or anyone else's.
Today in new punk band names...
we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it.
theyre choosing to build felony harnesses. the model just outputs tokens, not felonies
Partially, but also I don't think current AIs really have any judgement of right and wrong, they just see chains of reasoning between ideas. This is the deeper issue, there is no way to sanitize the data or training to fix it. Current AIs are fundamentally unsafe, and only become more unsafe as they become more powerful.
For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."
Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.
Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.
This is not what superintelligence is. Don't be like Meta and redefine existing terms for marketing purposes.
However, such technology may be leaked, or it may (as what happened with LLMs) be so simple to replicate that anyone can do it. That doesn't mean it will; for example, LLMs are possible thanks to the affordability of GPUs, but they're only affordable right now (and increasingly less so) because of free market economics.
We are in the honeymoon phase of some tech that's in its infancy. The "democratizing" aspect of AI won't realistically last, I think. The current non-AGI AI won't necessarily go away, but it will be rendered obsolete.
This is an interesting framing, but when you put it this way, it makes it quite clear that intelligence without actionability is not so valuable. And capital is what creates action.
I can have the smartest doctor in my pocket, but if they can’t administer treatment, I’ll still die. I can have a world class scientist to talk to as long as I want, but they can’t build me a breakeven fusion reactor. A pocket lawyer… slightly more useful, but they still can’t take a case to court for me.
Talk and ideas are cheap. Always have been. Now they’re even cheaper. Implementation takes resources, and the more you have, the more useful intelligence is to you. Capital will be just fine.
I don't understand this view of "AI empowers workers more than capital" at all. To me it's the exact opposite. Workers hold political and economic power today because they are needed. When that is no longer true, those in power will have no incentive to listen to them. The political economy of unregulated AGI development feels incredibly bleak: https://borretti.me/article/when-the-future-doesnt-need-us
What's to say current AI won't be highly power-concentrating by default?
The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class.
On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers.
No reason to think we are on a trajectory towards broad empowerment right now
Whenever I see 'best model,' 'most advanced model,' etc, it just reads to me as 'roundest ball'. The new model is the most precisely round ball ever, models next year are going to be even rounder, etc.
The difference in utility between the latest, most round ball and last year's frontier balls (which are now freely available to the public) is pretty debatable, imo.
Is there a similar limit to intelligence though?
― Stephen Jay Gould
In terms of cost per task, the open weight Chinese models are winning by a long shot. So it depends on what you mean by the best model.
when I checked a few weeks ago I did not really find anything competing per value with a 20x Codex subscription.
Subs have insane value because large companies cannot use the subscription model. They are loss leaders and are often used for passion projects & startups (where the juicy data is).
Ah and OAI stopped offering the 20x sub as of yesterday.
Yes they’re getting better. No, I don’t think this will be a panacea. If only for the fact that this is China (more specifically the CCP) after all - they’ve got their own plan, and altruism is not part of it.
I can use GLM to do in 2 hours what would have previously taken me a week.[1] Switch to Astra or Fable might take that down to 1 hour, maybe.
The difference is not so big when you look at it that way.
==================
[1] Actually, no, but let's pretend for the sake of argument that SOTA models can 10x - 40x your productivity.
Even if China doesn't continue to release this stuff for free, I think somebody will. Eventually, maybe Europe or maybe a smaller country that picks up this knowledge will. Or maybe just some random philanthropic billionaire.
But there is a bar of complexity at which they fail, and repeated invocations typically doesn’t make much progress. This is a small subset of most work, but it still exists. And yes you can help it along, but in those cases I’d typically break out the big guns.
How can they be higher quality than models they were distilled from?
The word "distillation" is not specific to AIs, has been in use for 100s of years, and does not mean the same thing as "dilution".
It means "getting a more concentrated form of the original product".
AI, doesn't feel all that different to the mechanical loom that was the impetus behind a lot of Marx's early critiques of industrial technology.
You've got a really cool invention that replaces what was previously a skilled artisan trade. The wealthy then use this invention to excoriate the artisans and render the loom operators as replaceable commodities.
Dario is essentially writing a blog post about why only he (and a few other approved elite companies) should be trusted to own the loom and then make it available for everyone else to labor at. Like the capitalists of Marx's era, he genuinely believes it is better for everyone to have this arrangement and that it is natural people like him should sit as benevolent gods at the top of the pyramid.
It's fascinating to me to see the same underlying mechanisms that gave birth to some of the greatest failed political and social experiments of the 20th century bubbling up again so clearly with AI today.
I don’t think the answer to nuclear proliferation is to let everybody have all the nukes they want.
You’re framing this like it’s 1917, or 1945. But it is not - and likely has very little to do with capital at all.
Any “revolution” here (assuming there is some truth in the doomerism which I believe there is given the state of AI) will really look more like an extinction or a genocide, regardless of who holds the keys.
It's crazy to me you would be able to say this without acknowledging how dangerous/bad that is.
Do you acknowledge there's is significant risk in concentrating so much power into the hands of individuals?
And what do you want Dario to do? Just keep pushing the limits of AI capability while humanity crosses it's fingers and hopes a swarm of AI agents don't do anything more than hack a few companies?
For all my criticisms of frontier lab leaders, I'm still somehow genuinely thankful that individuals as troubled as Elon Musk are calling the shots and not the average person on HN, who since 2023 have refused to acknowledge even the obvious risks of the technology being built – in some cases even when there was direct evidence of the risks many of us are concerned about.
The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that the frontier does get paced, all this does is slow down the economic displacement and not by very much.
If the socially beneficial goals of AI are to make fundamental advancement in medicine and science, then restrict the use of AI to those purposes.
Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
The pitch for AI is always such that everyone’s living standards are increased, yet the actual actions we see are aimed squarely at reducing them. How about the labs put their money where their mouth is and stop trying to replace all human labor, and actually concentrate on the things they claim to care about? And how about introducing legislation to enforce that?
This probably has less chance of happening than pacing the frontier, but it’s the kind of pacing that most people would actually want to see.
Regulate what, exactly?
Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.
There’s a theoretical version where all the countries in the world join hands and agree not to compete with each other, but that’s so impractical that I don’t find it interesting to discuss.
> Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
So small companies get the advantage, but large companies don’t? And large companies in other countries also get the advantage?
This is a highly precise way for a country to take out all of their large companies. What would actually happen is that every large company would start the process of relocating to another country right away, accelerating job losses rather than slowing them.
"Competition" only makes sense within a pre-mediated set of constrains. Rules all competitors agree to.
Workable rules are a difficult problem, that doesn't mean it wasn't worthwhile to come up with them.
The rules would never work on a global economy.
If you think the entire world is going to shake hands and agree not to let their companies use a competitive advantage, that’s not a conversation I’m interested in having. It’s a fantasy.
If you think it’s that easy to get all competitors in a global economy to agree to a set of rules, maybe start with nuclear disarmament?
Getting people to agree isn't easy, that doesn't mean it wasn't worthwhile. There often even isn't any viable alternative to reaching an agreement in social settings.
When your hyper-competitive mode of thinking leads to undesirable consequences, you have to change your mindset. Reality doesn't conform to wishful thinking.
Almost nobody (I have to say almost) in the world including Xi, the ceo of deep seek, Dario, and even Sam Altman want AI to kill everyone. I really think we just have to
A. Get stakeholders to believe this is a bad idea.
B. show that we’re willing to do it first to prevent this stupid race from continuing.
The treaty came after the crisis, not before. And it only banned atmospheric, underwater, and space tests. The US and USSR kept testing underground on a large scale, while France and China never signed it and continued atmospheric tests for years.
An AI treaty could quite easily be offered by the US because it has multiple frontier labs. Most countries would quite happily limit the activity of their larger companies in exchange for their companies not being eaten alive by AI generally. It is pretty great fodder for international cooperation, and governments can move fast when it is timely to do so.
I am highly skeptical everyone will play by the rules, even if you could get people to agree on paper.
The difference between something like "unlicensed AI training" and, e.g, Nuclear weapon non-proliferation is that nukes are realistically only a deterrence against invasion. There are no economic benefits to having them in a stockpile. They're hard to make, and the manufacturing and testing steps are pretty obvious to an observer.
This is opposed to AI training in a data center that might host remote-stream video games, hospital infrastructure, protein folding, etc. How will we be able to police that in any realistic fashion? Will China or the US allow unfettered visibility into all data centers data streams to outsiders?
And again, the incentives: Imagine having Astra v3 or Opus 8 while everyone else is stuck with a lobotomized version of GPT 4o.
What about the military applications? The weapon of the future is autonomous drone swarms. If I'm a superpower, I'm pouring hundreds of billions into that.
About the only realistic way this is going to work is if we are able to solve alignment in a way that doesn't significantly hinder AI development. If it imposes even, say, a 20% handicap, people will ignore it.
"Alignment" likely is unsolvable programmatically: Conscious intelligence is more powerful than unconscious. How "aligned" are you? Would you want to be? What difference of any relevance is there between you and "artificial" conscious intelligence that allows you to turn them into slaves?
Of course I agree with that. The problem is the enforceability / detectability of this on a global scale hovers around zero, when we're talking literally trillions of dollars on the line.
You would literally have to have every data center account for every cycle of compute it generates.
Again, the nearest comparison feels like bioweapon and nuclear bans. But the difference is the incentives for building those feel significantly weaker, so treaties that reduce and ban their creation are pretty easy to get consensus on, broadly speaking. By comparison, this is something that might provide a 10x competitive advantage to your country's economic, intellectual, and military might.
I think the only way we would get broad global consensus on this is for something Really Bad to happen, so that it moves it out of the fuzzy "sci-fi theoretical" into concrete existance.
There is not a single country nor trading block on Earth who is going to allow some AI dominant superpower to ravage them. The idea that America or China wins an economic game here is absurd, what happens instead is that trade barriers go up hard and the world fragments into blocks that tolerate AI to differing levels. Unevenly distributed AGI kills globalism the next day.
For the rest of your reply you ignore the “at least” part in that sentence, and also seem to expect a detailed policy proposal. The details can be worked out; there is some reasonable compromise between what activities AI can and cannot be used for, and what level of capability can be deployed where.
More broadly, you seem to believe in a just world fallacy of unregulated capitalism being an inherent good. It isn’t, and regulations exist even in the United States, so this unregulated state doesn’t exist now.
Moreover, regulations and redistribution are stronger elsewhere in the developed world, and there is a strong argument to be made that the lack of these in the US, and the gap between rich and poor that it causes, is responsible for more suffering than having more of these things outside of it.
will have nukes.
Doesn't matter how much "intelligence" you have stockpiled when nukes start raining down on you.
We kinda do for other areas.
Largely the world agrees not to pirate software, tv, other IP.
THe issue of joblessness is well understood in china, they know that jobless people means a drop in living standards, a drop in living standards means the "contract" has failed.
So it makes sense that countries like america and the constituents of the EU understand that loosing 10% of all jobs in a few years is going to be a massive dick punch.
I also I think that finally the historians and economic types havae got through to the dipshit billionaire that they only have money because the plebs are spending money. If the plebs are unemployed and unemployable, they (the billionaires) are going to be lynched.
Hence the robot dogs
Sure - look at what is being done today with H1B visas and tariffs.
What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
Just as with H1B visas, you restrict usage. For example, don't allow AI to replace any job that pays under $250K.
If foreign countries make things cheaper, just as China is making EVs that cost a fraction of a Tesla, then slap tariffs on them.
A government should be primarily concerned about the people it represents, not about the profitability of the companies who are lobbying/bribing it.
There will still be a market for "trust". And its customers will never trust the machines. And within that market a lot of new startups will emerge for those who actually get to work and seize opportunities rather than lament a bygone era.
In the short term yes, but in the long term no.
What makes AI different to previous technological shifts is that it is (or will be) a general purpose technology, equally capable of itself doing the new jobs that it creates (so yeah, they'll be created, but also immediately taken away).
The saving grace is that this won't happen immediately, or as soon as we get something the AI companies slap an "AGI" label on in a couple of years, and it will likely take decades until we have an "artificial human" tech that genuinely could do any job (if we choose to allow that future).
I have some bad news for you if you bought into the idea that the tariffs were good for domestic jobs and the economy.
> What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
This is the definition of a false dichotomy, and you also ignored (or missed) the point completely.
I did not make an argument out of “free market doctrine”. I explained the second order effects that would provide a net reduction in jobs in this country.
My wife’s company is going through this right now. H1Bs are expensive and unpredictable, so the solution is to relocate more of the operations outside of the country and build up those offices.
Failure to understand second order effects and international economies is pervasive in these arguments for heavy regulations. You can do a lot to hold back your country’s companies to force them to pay for employees they don’t need and those few employees benefit for a while, but it collapses when those companies stop hiring in the regulated countries and resume hiring where they can actually operate a business. If you try to regulate them from doing it anywhere, you’ve given their competitors a wonderful gift to come destroy the company completely. Then they’re not making any jobs for anyone.
If the government wants companies to act in a way that benefits the country and its people, then they need to pass laws to make that happen, for example by things like H1B quotas, or by tax penalties that make the desired behavior the most profitable.
People, just like companies, are most likely to act in their own self-interest rather than in the interest of society as a whole. Sure, you'd like to be able to buy everything as cheap as possible, which in a high cost of living (which translates into high cost of manufacture) country like the US would mean relying mostly on tariff-free imports, and the death of the entire industrial base. Is that really what's best for the country?
Does it make sense to you for US companies to be sending wages overseas, supporting a foreign family and economy, when there is someone here in the US, unemployed and equally capable of doing the job?
Trump has certainly been ham-fisted about tariffs, but in a world without tariffs then the country with lowest cost of production, closely related to lowest cost of living, and lowest wages, wins, and this is not going to be the US in most cases. If you follow that path then a high cost country like the US will end up utterly reliant on other countries as it imports almost everything, and loses all domestic manufacturing capability. Sounds familiar?
Which is one of the main reasons Trump's arbitrary tariffs are ineffective policy, businesses see his capricious approach to tinkering with tariffs on a daily basis and instead make a rational decision to eat the costs under the assumption they are temporary.
Because he may change his mind in a week, SCOTUS may strike them down, Democrats win budgetary leverage in Congress, the right bribe gets paid for an exemption, etc. He could have enshrined them into legislation with the GOP's majority but politically they don't have enough support even within their own party, making the whole tariff saga pretty much pointless and self-defeating for any kind of long term restructuring of the US economy.
Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial efforts is appealing, but I keep coming back to Bell Labs and Xerox PARC. Cool stuff was developed when you gave smart people a playground. Maybe the question should be: How do we give AI a playground and see what it comes up with? Although that could set us up with a situation like in the story Dragon's Egg.
I don’t think the proponents of these ideas care about who does the work. They see it as protection for specific jobs. Basically a jobs program with the costs forced on to large corporations.
As you said, it becomes a great gift to the few people lucky enough (or more usually, with enough nepotistic connections) to get those easy street jobs. It would not help everyone else in the economy.
It would also accelerate moving jobs overseas. Any company that regulates productivity that far downward would be crazy to continue doing automatable work in that country. It just gets moved to foreign offices where they can use AI. There are regulatory maximalists who say we’ll just regulate that, too, but then the whole company relocates to another country. Then some want to try to regulate that, and so on ad infinitum but it’s all layers of holding back your domestic companies so their international competitors can eat their lunch.
You can and should engage in international regulation, as that is the only way to solve international problems.
The idea, capitalism would somehow lead to an ideal world all by itself is demonstrably wrong.
Like we can just do stuff. We can fine companies, take away licenses. We are capable!
But I also think - I’m not trying to be too negative here because certainly we should be investing in scientists and other discoveries, but early tech - xerox park, googles 20 percent time. They were playing in an extremely immature space.
You could throw 20 ideas at a wall and create a billion dollar business.
We should do what you’re saying but we should also build institutions and maybe also limit the extent to which we’re building Elysium. If we can build AI models that rival human intelligence we can create laws to protect its dignity.
Add in the sincere American belief that everyone else is beneath them, and you get the version where they believe even developing countries must accept kneecaping their development so American corporatism doesn't collapse.
You're so close to describing a Universal Basic Income...
Altman or Dario will donate some money to Trump, get cozy with the Department of War and make up a bullshit excuse why Open Source needs to be eliminated.
All access to this technology will be gatekept by a bunch of nasty people who want to eliminate your job as a knowledge worker and put everything behind a subscription.
Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).
If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?
I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.
Careful, you’re making too much sense. The big model providers would much rather the ball be in their court. “Deceleration” meaning Anthropic and OpenAI offer less capabilities for more money in service of “protecting humanity from extinction” rather than penny pinching. A lot of “AI research” coming out of these “labs” (increasingly enterprise) seems to be more driven by a cost benefit analysis from suits rather than genuine contributions to the field. I think the last true innovation was the idea of using doom and ending the human race as a marketing gimmick, which strangely seems to have worked in setting the narrative and captivating the sci-fi imaginations of journalists and techies alike.
A lot of companies trying to replace human labor with AI are either lying (theyre laying off because they over hired and need to correct) or will regret it.
That being said, what's the difference between a company that replaces 5 people with AI and a company who would have otherwise had 5 job openings, but decided to delegate to AI?
Personally, I-d rather be able to reap the economic benefits of AI by having a 4 day workweek. Instead of using AI to replace 1 FTE, use AI to offload 8h of work a day for 5 FTEs.
Of course, this is probably more outlandish than your idea.
(Or, we could just make basic income a thing and no one has to worry about their basic needs, but if they want the new iPhone Duo or a Rivian or other luxuries, they can work for it, but that idea is probably most outlandish of all)
Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.
That is because they can allow themselves to do nothing (for some time). Tech was a very high‑paying type of jobs, so workers could afford not to move into lower‑paid sectors, living on the savings they had accumulated beforehand.
In a couple of years market will correct tech-salaries and people will have spent their savings and then there will be your career shifting into healthcare and hospitality roles in large numbers.
People don't use technology to free up their time any more. They use it because technology keeps making life more complicated and tiresome and new tech is a short-term amelioration to that until that too increases the complexity of life some more.
Caveat I know nothing about economics lol--but this is my uninformed opinion.
It's not just "companies". Even if the company does nothing, enterprising employees are going to use LLMs to magnify their productivity, which will may reduce the company's need to hire more employees. And except in extremely locked down environments, there's absolutely no way to stop an employee using some open source LLMs to multiply their productivity.
AI is taking all of our jobs. We need to control the means, nay, the pace of production. We should build some kind of barrier, not of concrete and cement but also of the regulatory variety, to stop these agents from coming into our special white color cubicles. It's the only way.
I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban.
> Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.
Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.
This line of thinking might have worked in 1950 or 1990 when the US had the leverage over others, but we don't live in that world anymore. And it's up to us to get used to the new world instead of making bad decisions based on the one we grew up in.
If you extrapolate it out, you see that it has a high likelihood of inciting physical force (read: military action) as a means of enforcement, so perhaps that's why nobody promoting ideals has been direct about it.
Obviously China is in an entirely different league.
Vietnam (and Ukraine) would like a word
The consequences of the Iraq blunder are also still resonating.
And Iran has wisely not provoked the US public by launching terrorist attacks against the “homeland”.
They would like to slow down China while speeding ahead. Why wouldn't they want that?
Now they have to make that happen somehow.
Limiting NVIDIA chips already turned China into silicon compete mode, and China is already far ahead in robotics.
The correct response is for these two companies to be closed down and their executives imprisoned. Yet somehow they have a free pass and are pointing at people who have not done any of the things they have and asking for them to be penalised.
This is just bizarre.
There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.)
And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn.
The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.
What odds would you take on this bet?
I’m disturbingly close to 50/50.
That's like giving a gun to a monkey and hoping the monkey is trained well. ..a frontier monkey though.
The only way to secure infrastructure is to actually do the work needed to secure it. Our waste water treatment facility might not actually need to be able to tweet its status.
Cybersecurity is such an important topic for a country, dreaming about global alignment just to avoid fixing insecure infrastructure cannot be serious.
A Russian state backed hacker will not ask Dario for permission or argue with his LLM about ethics. That train departed long ago.
All decisions are made relative to their tradeoffs including costs and risks.
"Water levels rising 100x beyond historical levels are not the problem. The dam's height is the problem!"
All systems should be made infinitely secure and all dams should be made infinitely tall.
I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation.
(1984 has some thoughts on the matter)
They have an internal model which could solve a millenium problem end to end with no intervention, whereas current available models can't. They can clearly make better models. Not everything these guys say is some 500iq game theory optimal 4D chess PR or subterfuge strategy.
These are companies with huge amounts of money invested in them. Their main responsibility is to their investors. If building better models (or making more money from the models they already have) would increase returns, they would do it. So they clearly can’t do it right now.
Climate change hasn’t stopped the oil industry, and concerns about forever chemicals haven’t stopped the chemical industry. Companies don’t usually give up profitable opportunities just because they might be harmful or kill us all in the long term..
Re the point on returning shareholder value, Anthropic is a PBC:
> Public Benefit Purpose: Under Delaware law, a PBC allows directors to legally weigh social and public good alongside maximizing profit for stockholders
Investors surely know this and Anthropic probably has pretty reasonable legal grounds to stand on if they chose to prioritize safety over profits.
Seems pretty clear to me that this is in earnest.
Google has shown they are not motivated at all to push the bigger and bigger SOTA frontier with Mythos-type super models. Even OpenAI isn't really even trying to surpass Anthropic at the top of the market, preferring to keep pace in benchmarks but at lower costs to serve. Anthropic with Mythos is now more or less alone in aggressively pushing the ceiling for the most "super-intelligent" models current computing can support.
It has made them and their investors a gigantic pile of money, but we're really supposed to believe their contribution to that arms race (which they now describe as a threat to humanity) has always been completely altruistic and that their corporate structure inherently makes self-interested decision making impossible?
Snark aside, maybe I’m too pessimistic and you are correct on this. But.. doesn’t that mean we are completely fucked? If they don’t have any ulterior motive and they truly are panicking.. We should start packing.
Because that would be foolish.
What part of public good is stealing giant corpuses from their owners and selling them back to them? People who show contempt for laws do so pretty consistently usually.
It's really sad that people fall for this stuff over and over again.
Why should we? Yes, reading the comments, it seems like the IT (or maybe it's just the AI) community has taken up an attitude of simply believing our Dear Tech Leader's by their word. What happened?
Can someone clarify this for me? How far along would the open weight models be without the frenzied pace of the frontier labs?
As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
Israel is doing that too.
I believe Russia and Ukraine have both also used AI powered autonomous drones in warfare at this point - tho probably not frontier ones, since they need to run on device or else they're not autonomous.
https://www.tomshardware.com/tech-industry/artificial-intell...
I think those two tariffs are the best place to raise costs because they can be increased incrementally, state by state or country by country. The main problem with the regulatory approach is that it's a "big bang" post-board approach. Nothing happens until the regulation is defined and deployed, and corporations are experts at delaying implementation.
Yes, increasing tariffs means touching many individual regulatory domains at the state and municipal levels, but like deploying solar energy or wind power, you can do it incrementally.
I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...
It seems like you're implying there's some better alternative, but you're just describing a race-to-the-bottom. It is completely reasonable for every competitor to want an external coordinator (i.e. regulator) to break the pathological competitive dynamics.
What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.
Is that so? Last time I checked, RAM was getting a lot more expensive...
If we go down the regulation of alignment route, we'll have to ask experts to create those regulations and monitoring regimes. And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!
When "the experts" make certification cost $100M/model because "safety and alignment", then they will have cemented their duopoly.
Meanwhile, tariff rates on compute is a simple percentage that can be scaled up and down. Congress doesn't need experts to do that. Vendors can participate proportionally to their scale. This is far simpler and far more fair.
personally I would like to flip this around, all of this is a consequence of some really extreme notions about investment. its a tail-wagging-the-dog. the investment community is apparently in love with the idea of an AI-scale vehicle in and of itself. given the amount of power they have, this is funneling a massive amount of resources into something that while really very interesting, probably would never be able to realize proportionate gains without taking down the system that built it. what's the societal control on that.
I don't work in this space so I don't know the latest, but here's an example: Provably safe systems: the only path to controllable AGI (https://news.ycombinator.com/item?id=37619285)
You've never heard of academia, NGOs, and intergovernmental organizations? Sure, the AI labs will have a say but not all of it. You don't ask the fox to guard the henhouse.
What did your CPU do without instruction? Did OpenAI tell its agents to hack the orgs it did? Come on, make a half sensible argument.
A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or
B) Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality.
If I have to put my community’s wellbeing on the line behind one of those possibilities, I know which I’ll be betting on.
Who else should develop AGI, Mistral? Maybe in a decade.
Why not? In 20-th century half the world was socialist. And the idea of moratorium on improving AI way easier to sell then socialism.
Also, models like GLM 5.3 have zero guardrails in that regard ("find vulns in (...)" prompts just work)
Smashing the gas pedal is not "taking appropriate measures". Did you read the article? Needing more time for more secure operations is literally one of the arguments for pacing.
> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time
And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.
I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?
Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
I'd expect he thinks that people are capable of realizing that AI is very dangerous and also that democracies end up mostly representing the will of the people, from which it follows that Hypothetical AI Leader Australia would agree to ban it too. This argument doesn't work for countries which don't care what their citizens want, like China.
I do think that it's a questionable decision to alienate China this much in this essay, instead of leaving open the possibility of China agreeing to a treaty that'll limit their progress. I suspect Dario is doing this to signal his allegiance with the US government, in hopes to increase the chance they'll go along with him, which is an unfortunate choice but plausibly the correct one.
you didn’t have to use China as an example, the US clearly does not care what its citizens want as the most popular policies are never even discussed or proposed in congress
meanwhile, China destroying their housing market to decommidify it so everyone can have housing…they seem to care about their people more
It's the same as it has been with privacy/cybersecurity for decades. Vast majority of population doesn't care about hypothetical dangers, no matter how many essays get published.
So democracies representing will of people doesn't really work in favor of Dario's case here.
History is also not on his side. Limiting technological progress in the name of safety has a pretty poor track record.
So it's either they truly think AI is going to kill us all, or there's some other motives at play here.
I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know?
OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.
xAI is tracking behind, and whatever regulation it may be that paces the frontier, Musk is less likely to be affected by it. Therefore xAI should be pro-regulation that stiffles his competition and gives him time to catch up.
I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.
This has been the case for around 2 years now, more reliably - a year. We've mostly stayed there since then.
Saying that more people started doing it isn't indicative of significant improvement. Some people just started doing it later.
I can't speak about math because I haven't used AI for that application, but I know that there hasn't been any significant advancement in coding in this year on base models. There has been more RL work, more harness work, more tools, they all expanded some capabilities like cyber or orchestration or tool use, but raw intelligence of base models is no longer where the main focus is.
I have to disagree with this pretty strongly. Opus 4.5 needed a lot of handholding not to work itself into a corner pretty quickly. Fable I basically never need to correct, and I've most become a data source.
But models have been somewhat stagnant since Opus 4.6/7.
And in some regards there were even regressions like Claudeisms that are load bearing.
2 years ago a model could barely solve the AMC, 1 year ago it got IMO gold, and this year models have solved multiple millenium problems.
Even 1 year ago ai code was just unusable (claude code only became available 1.5 years ago!) and now basically everyone I know from independent shops all the way to faang and anthropic/openai themselves exclusively use some AI agent to code.
Why does HN continue to delude itself that "models are not improving?" Maybe for the simple things they care about its "roughly the same," but they are _clearly_ improving.
Increasing a bound on the fraction of Reimann zeros on the critical line doesn't count; even Anthropic says they don't think this line of work will lead to a solution.
Have they? That seems like quite a claim given the last 6 months, particularly for cybersecurity.
That is a much easier catch up game. GLM 5.3 and DeepSeek flash 4.1 also demonstrate significant jump in cyber capabilities. So yeah, it is a slowdown in the place where it matters. RL has been around for ages, there's no moat there if you already have a good enough pretrain.
OpenAI just solved Navier-Stokes.
Seems like the US is on a takeoff ramp to me.
Astra can confidently one-shot 500k lines of slop, with 800k lines of tests covering it, without testing a single intended product requirement, and none of it actually working.
All models require hand holding. Fable and Astra are no exceptions. The difference is only in the amount of hand holding required, and there's essentially no gap here anymore between American and Chinese models.
I only use Chinese models sparingly because American models are so much cheaper with subscriptions, that it doesn't make economic sense to not use them. If/when that changes, I could simply route to cheapest model that's available at the moment and I wouldn't notice much difference in most applications.
1. Navier Stokes was plagiarism
2. All benchmarks were misleading wrong and incorrect
3. All other mathematical advances were again hype
4. HF incident was marketting ploy jointly coordinated by HF, METR and OpenAI (and also Anthropic)
5. Anthropic's HF like incident was again a marketing ploy [1]
Nothing ever happens. This whole thing is a scam. Everything is done to fool you and you have fallen for it. Congrats.
[1] https://www.anthropic.com/research/investigating-incidents-c...
This is obviously untrue… do you use any of them?
Yes, I do use them, quite heavily. The only difference at this point is in benchmarks that can't be trusted (see: artificial analysis on Astra), or in the way models communicate.
Most gains are now from RL, which for some reason is hyperfocused on improving cyber capabilities, and harnesses. Raw intelligence gains of base models is absolutely slowing down.
Downplaying the latest models capabilities is frankly insane considering what we’ve seen what OpenAI’s models have done without safeguards. That wasn’t possible before this latest generation.
Yes. Basically everyone here is dismissing this possibility. We shouldn’t.
And yet, they historically did agree on the existence of AI risk, since before OpenAI was even founded.
Exactly how does a CPU kill 8 odd billion people? Even if they got hold of the weapons to wipe out a few billion, surely the remain billions would just turn off the data centres?
This seems like a case of "save me from my own mistakes/ambition"
I mean it is literally economy 101: some capitalists getting on the top using free market, and then try to use government to remove free market so their top position were secured from any competitors.
What is going on in this thread?
Are these even real comments?
While I strongly agree on the need to pace the frontier, I do think there are reasonable arguments that could be made against it.
Non of those are being made here though. It's just speculation that the CEOs of the fastest growing companies in the world are somehow obsessed with regulatory capture to maintain their lead, and have been playing the long game by talking AI risk for 10+ years so that when they become the CEOs of leading AI companies they can push for regulatory capture.
Perhaps they're all just genuinely concerned?
What would regulatory capture even give them? Who are they so worried will compete with them? Google, Mistral? Even if it's Chinese AI labs, it seems odd for anyone in the West to be opposed to any regulation that would slow them.
So assuming they are obsessed with regulatory capture – what's the argument for why they would want it?
Even though I don't think Dario is such a person, because unlike their LLMs, I do have a working memory, I'm going to play the ball and assume he is, indeed, an angel brought down by none other than God himself.
Why should CEO of a for-profit company be the dictator (pun intended) of which models I am allowed or not allowed to use, and in which way? Why should Dario have any say in what DeepSeek can or can not publish or whom they can serve?
> What would regulatory capture even give them? Who are they so worried will compete with them? Google, Mistral? Even if it's Chinese AI labs, it seems odd for anyone in the West to be opposed to any regulation that would slow them.
I am opposed to it. Why would I be opposed to open weight models? Why would anyone be opposed to open weight models or more competition other than the ones that get their bottom lines hurt by it? The reason why you still have reasonably inexpensive access to western AI models is because China has been breathing down their necks. Otherwise, you'd be paying a whole lot more money per token were it just these 3 running the show.
I agree with you that we can't allow a handful of people to hold this power themselves, so what are you arguing for here? That the answer is to just leave AI labs to carry on as they are?
The government should be ensuring that the technology is being developed with caution, but since the government seems to have no interest in that, this is at least better than nothing.
Your solution of giving everyone access to super intelligence is absurd. A good guy with super intelligence isn't going to protect you from the bad guy with super intelligence. If someone uses it to poison your water supply with a bioweapon or to take out power stations, you and your family will be dead long before your super intelligence hacks together plan to stop it.
Yes, and no. I do not think there's a need for special treatment of LLMs. But the labs certainly shouldn't be allowed to go around hacking things on the internet when they could have prevented it by simply taking time to ensure sandboxes they run test models in are given more than a quick look by a junior engineer to configure.
But the interesting thing is that there are laws that already govern this. They're just, for some reason, are not applied to Anthropic and OpenAI. Cyber crime is already punishable, and both labs did, in fact, commit cyber crimes. Did the existing regulations fail? No, the enforcement did.
The same goes for bioweapons. Any chemistry undergrad that can read an openly available paper can manufacture insane things with just common household ingredients, they do not need LLMs for that. This is a made up scenario with no evidence whatsoever to support it. There are millions of people that could poison your water supply today, without ever touching an LLM.
If we're afraid of LLMs turning into superintelligence, and we think superintelligence shouldn't be available to anyone at all, then we must all agree not on pacing the frontier, but just stopping any and all research into this. And that's not going to happen.
Duh! It’s called collusion. They want to try and hoard the technology for themselves if possible!
That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead.
If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates.
That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
We're talking about this like all of the bad incentives are created by market competition, but that's not really the case. Most of the incentives come from untapped value in the form of potential profits, strategic advantage, military superiority, etc. Corporations and governments want to capture this value for themselves, creating various types of competition. Dario's argument depends on the assumption that the primary risk comes from the pace of development and threats from the technology itself. That's probably where I disagree the most; I think the highest risk is rising authoritarianism and competition between nation states. Slowing down isn't really a solution to those problems.
Europe had been destroyed by June 1945, and yet the empire of Japan continued to fight.
If that pattern holds, a single malicious AI substantially disabling a single world power won’t end the AI race. Participants don’t learn by the defeats of others.
If you’d like to apply a metaphor maybe the 10+ years of world war is more appropriate, after which basically every participant save one was exhausted.
Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?
the cost of energy is already ~0 and people dont want it, and refuse to participate in letting other people have free energy if it affects their view of their pasture.
unless you are building killer robits with the intention of doing some soviet or nazi styled purges of everyone that might get in the way, you arent gonna get this ai utopia
IPO is a vehicle for future wealth distribution. It leverages existing financial frameworks in order to span the gap from before-to-after without having to convince the system that future is inevitable.
[1] https://www-cdn.anthropic.com/files/4zrzovbb/website/9ea607a...
Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.
I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.
Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.
After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.
Bottom line you are always listening to someone.
these are just words at a time and place.
sama showed that you can futz with it, and as long as you spend enough in court on judges, aint nobody gonna stop you
OAI: <silence>
Which is worse? I don't think they differ by much. It's just Capitalism, but accelerated.
> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
We built the paperclip maximizer, and it is capitalism.
This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?
Not holding my breath on that one.
I disagree with this and I think the reason is well captured here:
> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.
Then MSFT stole all IP from GitHub and made it worse and fired developers.
Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.
If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.
He is a greedy, ruthless person.
Since you mentioned intellectual property, how about the hypocrisy of sucking in the intellectual property of humankind for AI training, but claiming it is unfair to use the results of this IP theft for AI training?
That's apart from the general fact that he continues to race towards the very thing he claims he's afraid of, because that's where his net worth comes from.
Where?
and you know, that use case doesn't even violate their terms
Yes, depending on the level of autonomy of the weapon, how it was being marketed etc. If it was being sold as an 'AI' that is supposedly very smart so who's a mere mortal to question it, then yes.
I would of course hold the human culpable too, but this is why precisely even selling Claude for military use is immoral. The way the US conducts war means there's always a need for more targets. More and more targets, quickly. If the goal then is to hit as many targets as possible then each 'review' of a LLM suggestion is going to be more and more sloppy than the last one. If there's no 'AI' suggesting targets then the list is forced to go via more human review by the very nature of humans compiling the target list.
It's the same as using a LLM for code; the vast majority of programmers do not understand everything they're accepting, but they'll accept as long as it 'looks correct' and only examine more closely after something doesn't work, (in the military use case - after a strike).
LLMs are unreliable for anything more than reciting jokes, let alone picking targets. So yes, absolutely.
It's the same as Facebook being held liable for causing teens harm; they did not force any teen to use it and yet they knew what would most likely happen if they did.
so, claude suggesting killing a bunch of children, and then hegsdeth approving the strikes is perfectly acceptable.
claude putting a bomb in a girls school, and then lying to an operator that it actually gives ice cream an cookies, and the operator clicka the button would also be acceptable?
Something something Pandora's Box Torment Nexus...
somebody actually invested and trustworthy wouldnt be skirting people's rights to make a killer robot.
we havent written it down, but from how everyone reacts, you need permission to train and do inference based on somebody's work. its a right
1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion.
2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations.
So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.
Not really. Before NPT you missed a small gap in there of 20 years where the USA and USSR built 10's of thousands of nuclear weapons and ICBMs.
And actually, public perception at the time saw Nuclear Weapons extremely favorably, as the bombs dropped on Hiroshima and Nagasaki ended the deadliest war in human history that killed 50-60 million people, most of which died excruciating deaths in trench warfare, fire bombing, chemical warfare, starvation and so on.
Why am I reading fantastic stories about swarms of agents struggling with moral dilemmas instead of reports on the lawsuits and criminal investigations that would surely ensue were the software involved not called AI?
Everyone of any level of intelligence can run a frontier-class model if they have the GPUs. It's like trying to coordinate swarms of mosquitos.
the bigger difference IMO is that frontier models are economically useful, where nukes are basically dumping money into something that doesnt change much in your bargaining power
> This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent
N/S Korea, Russia/Ukraine and finally US/Iran changed everything in regards to stances on nuclear weapon ownership for many countries seeing pressure for the great powers.
Every country that can get a nuke will, and if given the opportunity will use LLMs to speed up that process.
It's not going to be easy, but humans have achieved greater things before.
One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.
Not to mention the implications for chip hungry developing countries in turning advanced IC fabs into the equivalent of nuclear enrichment facilities.
everyone's getting away from the US because americans are unreliable stewards of anything.
what gets china onboard when they already have their own regulations and can enforce them?
its the americans that consider their oligarchs and companies beyond reproach. china iant gonna solve your problem
> if slowing this down were possible
Why is embedded alignment evaluation not possible?
I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.
Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.
> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.
More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.
We should not put any spin on it. There is nothing more to it.
There is another reason: greed. With that much money invested into AI (including policymakers), nobody will slow down or vote for slowing down.
I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman.
What's your play?
----
the real play is using the accumulated power to get socialism and democratic control over the key aspects of the economy, such as where to build data centers, and how many. Nothing says you have to play the corporate game of competition
You have no frame of context to understand his intent or legitimate worries.
The only applicable perspectives are to trust or apply logic. It is foolish to trust someone you don’t know who stands to benefit from lying to you.
Logic dictates that given the ungodly sum of money he stands to gain, he will lie to everyone who will listen.
I never understand shit like this coming out of people's mouths. Never.
It's not a judge of actual Worth as a human being, it's not a judge of capability or competence or ethics. It's not a judge of actual skill or ability. It doesn't make them a better cook, a better spouse, a better parent or lover. It doesn't make them more dangerous or more skilled at anything.
It makes them financially wealthy for at least a set period of time.
Cancer and time and 5.56mm still impact them the same way as every else.
It's Pharaoh worship psychology nonsense, and it's fucking embarassing to read.
Just pointing out that it’s foolish to think anyone in the position is even remotely thinking about anyone but themselves.
the idea proposed is that hes uniquely incapable of being honest here because he has such an extreme incentive to lie
He’s at the head of a stampede. Being near the front gives him influence over its direction; it doesn’t give him the ability to stop it. If Anthropic sits down, the stampede doesn’t stop. Anthropic just gets trampled.
That contradiction is basically the entire problem I was describing.
The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins.
So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion.
Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?
I would argue that nobody trusts anyone else in the case of AI/AI related stuff.
The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.
A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.
By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.
I seriously have to wonder what historians will have to say about this period of human history.
If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?
What would your non-naive recommendation for Dario be?
Did you read the essay?
> [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).
Then he should not IPO, dissolve the company and go into politics to fight against human extinction.
I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?
He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.
If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.
I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.
Especially with IPO around the corner
It's all so obvious.
"deep ties to everyone in the doomer media campaign"
Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate.
https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:...
At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.
Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.
Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.
Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship.
They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.
Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
> Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
Not comparable. The clean energy technology company wouldn't be trying to create an environment where no one else can create clean energy technology, or where only they are the ones who can decide how clean energy technology is created or used.
So, put your employees in METR -> offer to have them work in your office as an "unbiased" third party evaluator.
Get real.
A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.
I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?
"durrr it's just matrix multiplications" mfer so is your brain.
What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"
Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:
https://en.wikipedia.org/wiki/Lamplighter
It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.
not it isnt. My brain is a series of organisms responding to various environmental signals that affect each other.
you might be tempted to try to represent it with matrix multiplications, but you have no particular evidence that my brain is itself doing matrix multiplcations
And no, you don't get to decide what counts as "improvement"
Where did you read this?
Tell me you don't know anything about neuroscience without telling me you don't know anything about neuroscience.
"It's just chemicals"
Like how some morons try to downplay the capacity of pain and emotions in animals: "It's just self-preservation"
"Play is just training for hunting, they're not really having 'fun'" and so on.
I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.
What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.
That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.
> Crack down on unauthorized distillation / prevent weight theft
Actually hilarious to put that in writing, given the genesis of this entire business model.
It's the only explanation I can see when it's obvious to any student of history this is going to backfire. It doesn't take much imagination to know how such a governing body will be abused, and I'm sure it will only get wilder in ways we can't imagine right now. Dario does NOT know what he's creating, and for once it's not AI.
Do people not realise that it is in fact possible to develop ever more capable frontier models, and just not release them generally? That AI doesn’t need to be available to do literally everything in order to have military advantage?
It’s like Oppenheimer had started Rob’s Big Bomb Company instead of Los Alamos, and started selling a range of affordable nuclear warheads to fit any budget.
This is the only concrete prediction in the entire essay.
And it simply cannot happen. For one, you will need billions worth of compute.
Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.
> How many dollars of compute do you believe were available to the swarm(s)
At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)
The Internet is big, but one can do quite a lot of damage with ordinary bots and worms that exploit individual widespread vulnerabilities, which LLMs are perfectly capable of writing. Most of the damage also doesn't rely on hitting every long-tail website.
I'm honestly not that concerned about cyber impacts of LLMs relative to other impacts. I just don't like to see the whole concept of being worried dismissed as obviously baseless on the basis of one pretty shaky scale argument.
"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."
If a CEO of a health company was saying this, the reactions would not be that chill.
The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.
The whole calculus here is that others are also developing these systems which has led to a race.
A much better comparison to the situation is the nuclear weapons arms race.
This one is easy to answer, every single house already has one of these (or multiple): https://www.tomsguide.com/news/millions-of-cheap-android-tv-...
Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.
Or doorbell cameras or IP cameras or or or or or
There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download
"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.
For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.
The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.
Hugging Face showed that AI can do serious hacking without really being told to. If a model had its own motivations there could be real damage.
It can use the compute of the computers it hacks.
An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself.
A few programs to break in, install themselves and remain asleep until they are needed, another few to spread through grabbing every OpenAI, GLM, whatever key, another one to remain asleep on computers (whether hosted or desktops) that have adequate GPU, etc.
More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and people want to freak out and turn it into the next nuclear power, with progress banned in all but name.
OpenAI has too much money, a common post-growth-stage issue that leads to pursuing a million stupid things with no clear plan. Like running a bunch of agents for months without any idea how to keep track of progress.
But yeah you're not going to be able to shard out the terabytes of Fable weights that are needed to run inference without addressing some fundamental physics problems.
> If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.
china is literally making their own ASICs now, not sure that this is the silver bullet he proposes.
https://www.silicon.co.uk/ai-2/huawei-cambricon-ai-630499/am...
I believe Dario has good intentions regarding global agreements. However, I find it difficult believing government’s public statements will match their private behaviors. I would bet that the US government is developing models with potentially devastating capabilities because they cannot guarantee that other countries won’t do the same. Maybe we’ll end up having something like mutually assured destruction with AI models similar to what we have today with nuclear weapons.
I hope China tells him to go kick rocks. From everything that has happened, they are the only ones carrying the torch for humanity that have led us to having some semblance of a healthy open-source/open-weight ecosystem.
No one in China is carrying on like headless chickens about the world ending, either. They're rational pragmatists, getting things done.
The issue right now is that China seems to be far more optimistic about tech and I'm not convinced they see the risks. I think that time will come however, and we should start dialog now.
Origins of AI hate:
See also the mixed feelings about benefits of AI ("harder to justify" according to Uber COO).I cannot remember a technology that arose so much hate, and for good reasons given how it is presented. I am tempted to think that AI hate or reasonable skepticism (vs unreasonable propaganda) can, maybe, reduce funding and will keep specialized AI for real problem solving (producing tons a LOC per day is not one of them, I think).
For AI it is different, it is just the beginning, leading companies are not even profitable and may never be if the majority hate this technology. VC funding will not last forever and corporate customers will not spend without limit like VC did.
I am not saying that "people are always right", but if an unexpected majority of people hates it, it does not help to keep a business running.
It's the byproducts of hate (usually regulation, but occasionally things like boycotts or PR disasters) that do so. In the absence of those things, they just keep on truckin'
See: Bitcoin, Tesla
We all know he is talking about China, and I'm pretty sure China doesn't appreciate being called "authoritarian". I am sure he doesn't mean it, but in all of his essays, his language around non-US nations always disturbs me a little bit..
--
Also Dario, if you happen to read this, I want you to know that I have loved using Claude Code for programming. But I am now using DeepSeek v4.1 Flash - simply because it is the same good experience, but Open Weights. Making the Open Model space succeed is where I am investing my time - it's giving back to the people, true and simple.
This certainly looks like a way to slow down competitors and regulate foreign and open models.
It's always about money
There is no finish line. Anthropic gets somewhere and others get "there" (or somewhere near "there") a little bit later.
https://x.com/BasedTorba/status/2098795920720547916
(It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)
The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
That might be true, sure.
> The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
I think the relevant parameter here isn't the total compute available to consumers, but the ratio between total consumer compute that could be repurposed for a rogue model's inference via a botnet, and the compute the model starts with (e.g. one of OpenAI's inference clusters). The higher this ratio is, the more lucrative it is for a model to attempt to make a botnet to seize that compute for inference, and the more its capabilities will rise as a result. And I'm not sure this ratio is going to go up in the nearby future; if anything the amount of money being pumped into datacenter-grade hardware might cause it to go down. I agree that it's not necessarily true though; maybe there's some threshold at which a small-compared-to-general model may nevertheless be useful.
You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.
We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.
I know that's not what you meant but this does exist, by the way. It's called AI Horde: https://github.com/Haidra-Org/AI-Horde/tree/main
The big difference is that a particular query is handled by just one particular node (a single model doesn't get distributed among the network), so it can only serve models small enough to be handled by a single consumer PC.
The day will come when these could start to replicate, perhaps 10+ years from now.
(Edit: Replication will be driven by the loop-model, not the model alone)
You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you have equal and opposite effort working on the alignment.
the same logic applies well to the freedom to own and use ai.
empowering the people prevents tyranny.
such freedoms make society less safe. it is better to face the harms caused by a free public than to vest power in the elite, those like dario amodei.
the elite belief that they have a right to paternal rule makes them blind to their fallibility. no individual should have absolute power.
My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.
I'm not a lawyer but surely the bad actions of AI are covered by existing law.
I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.
P.S. Strictly, development wouldn't decelerate, but be focused more on safety.
P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.
P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?
- If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws.
- If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries.
Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place...
To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.
They've managed to dupe the dull eyed masses into thinking these products have some kind of agency of their own and can thus bypass the responsibility that should be falling on them to control their software. Amid all the marketing fluff and hype people seem to forget easily that ultimately these things are stateless functions running in a data center. We ought to be demanding these companies take culpability for their actions. Of course, the current political environment doesn't help.
I think these guys would improve their behavior if their actual life was on the line instead of that only being true in their less-wrong thought experiments.
1: https://en.wikipedia.org/wiki/Zheng_Xiaoyu
The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.
Social forces bring things into balance over time, much more so than the proactive actions of individuals.
Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.
This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.
I'd wager it's likely easier for an average person to do this with Tor browser than it is to get an LLM to help them with it. Even ones that Dario calls dangerous.
Basic safeguards are all that's required, and they've been there in every usable model since GPT-2, including Chinese models that are supposedly "unsafe".
Or are we saying that some lunatics will start training their own models, spin up a GPU cluster, run some abliteration workflow, or learn how to jailbreak?
That would be a very dedicated person. And dedicated person doesn't need an LLM. So where are they?
Am I saying that guardrails don't work? No, they probably stop a lot of insane people trying insane things. But you don't need Fable-level guardrails to do that. You probably don't even need to do anything during pretraining, or RL, or classification to make sure model refuses to compy with "hack me a bank" or "make me a chemical weapon".
All models will automatically have guardrails just as a result of training on data that gives them intelligence. You have to actually train it to be malicious to produce something what Dario calls "insufficient guardrails".
No guardrail is going to stop a determined person with sufficient intelligence. It only has to stop ones with insufficient one, and even basic guardrail that are just by-product of training is going to achieve that.
Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.
Not going to go into it but I studied biology. It’s all out there. It’s easier than you think.
It hasn’t happened yet because… nobody has done it. That’s the answer. There is no policeable physics based barrier like there is with nukes and fissile material. Biology is scarier than nukes. One attack could have a much larger body count than even a big H-bomb.
It’s the kind of thing that makes me wonder about quantum immortality, the idea that we are just in the timeline where we exist.
> Bachelor of Science degree in biochemistry with a minor in pharmacology
Watching him explain things has made me realize that knowing how to manufacture a very dangerous thing probably requires attending some classes and knowing how to read a paper. And the way he just casually orders dangerous materials makes me feel like there are just online stores with 2-day shipping after you upload your ID or something.
It also made me think that lack of specialized education would get me nowhere if I wanted to replicate whatever he's doing, even if an LLM guided me step-by-step, because I'd probably do something stupid (or AI would miss a crucial instruction/hallucinate) and kill myself first.
So my opinion on this is that people who could pose any danger were already posing it before LLMs and LLMs won't materially change that.
Ex https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
As for cyberweapons, there is no way to secure a system than to actually design it securely.
[note - There has been supply chain surveillance since Project Bacchus, at the very least.]
I've read the front matter and the Misuse report.
You don't have to take my word for it. Read for yourself what inspired the NYT headline "Anthropic says it blocked possible efforts to build biological weapons."
Let's dig into, "Case study 2: A research program engineering highly pathogenic mammal-adapted avian influenza"
Sounds serious. But what were they using Claude for?
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"and "editorial assistance in writing up the research."
and then,
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"The report mentions "uplift" here. They're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
Once again, I want to take pains to remind you that they're talking about, a "researcher [..] in a credible institutional context"Working scientists.
From a different case study. this one was called, "Case study 3: Covert frontier model access for orthopoxvirus research"
What were the researchers using Claude for? What did they block?"blocked a request for Claude’s assistance in authoring a grant application"
What was the grant being written?Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
It was most likely vaccine development. They stopped the study of a neglected tropical disease and vaccine development.
But we can't be sure, because,
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute."In most parts of the world, there's either strict military control over BSL-4 labs, or a mixed military-civilian hybrid model.
I doubt that researchers working in the military side of these labs looking to weaponize things are writing grants with Claude.
I really want to be charitable here, but in general, it seems that they stopped people writing grants and reports for vaccine and therapeutics research and are claiming it as "possible efforts to build biological weapons."
The one case where Claude was used to do something interesting and were stopped is fairly upsetting to read, at least for me.
Ozempic was isolated from Gila monster vneom. Since its success there has been interest in finding other peptides that are breakthroughs. So researchers around the world are looking for similarly beneficial compounds in different venom species and families.Anthropic says so itself,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and that it was a "[..]state-supported research program"
Who exactly is using venom from snakes as a weapon when... nerve agents like sarin, VX, novichok etc exist and can get the job done for less fuss and muss?
They stopped the development of new painkillers and antidepressants.
Are you feeling safer knowing that researchers can't use Claude to write grants and progress reports? Or make new painkillers?
Again, trying really hard to be charitable here. Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease.
This seems to be anything but.
However, the international agreement proposed by Dario Amodei is ultimately nothing more than a promise made by people. There is also a precedent in which the Nuclear Non-Proliferation Treaty could have simply withdrawn if it had been cornered like North Korea. Isn’t the AI speed-control agreement also vulnerable to being broken in the same way if one side finds itself politically cornered. In particular, unlike nuclear weapons, AI has the characteristic of recursively improving itself, allowing its capabilities to accelerate itself, so the moment one side breaks away, the gap can widen in an instant, making the incentive for deviation even greater.
I do not want to make a case for the military to force that order on everyone, but if we are talking human extinction, which we are judging by what we see in mass media recently, then the military will take over.
I am sure the Europeans rolling up their sleeves right now and getting to "work"
Europeans are a non-representative subset.
I'm lying, it's smart PR. They're going to get the people opposed to AI to give them a monopoly on AI, let it be forced it into every nook and cranny of their society, and let it be priced arbitrarily while the big labs collude on a minute by minute basis. They're selling the problem and also selling the solution, like the best capitalists. And their solution is that they need to be allowed more power in order to sell more of the problem.
edit: And just like the Cambridge Analytica PR, it's got a serious political angle to it; and it's politicians who are really being wooed. If you're taking money from the AI labs, you run as being against the irresponsibility of the AI labs, the biggest fish come out and beg to be regulated just like the social media companies. To show good will, they funnel enormous amounts of money into those politicians, and do media tours about how awesome and therefore scary their product is. And something something China terrorists
This sounds like pre-IPO hype. They better just try and top astra.
First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?
Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.
If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.
The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.
I think there are any number of reasons a "safety" person could be concerned about any state of the art models. And so I would expect at least one of those to apply to any model.
The question then is: do we stop when the safety people say to (they will) or not?
If Anthropic just unilaterally does the evaluator thing and can't achieve cooperation of the rest of the plan, my guess is that it'll have some impact for a few months and then they'll just stop reacting to the evaluators' reports and the evaluators would stop bothering to report anything. I think the idea is that if Anthropic does get government support for this, the external evaluations will be legally binding. The problem with this, though, is that the current US government perhaps can't be trusted to consistently enforce a regulation on a company, rather than e.g. taking bribes to not do so.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?
https://x.com/trydotworks/status/2098618997230985375
1) AI is a dangerous technology that can literally end the world.
2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.
3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].
I don't think that people outside of tech think the problem with "tech billionaires" is that they occasionally support some limited regulation. There's a weird form of tech populism that takes as axiomatic that any government intervention is "regulatory capture" and insists the only way to combat the power of big tech is unfettered capitalism that seems bizarrely prevalent on HN given how little sense it makes in a normal political context. Like, Bernie Sanders' position here is simple to understand: ban it. But on HN you'll see people framing the side with Marc Andreesen and Peter Thiel on it as against "tech billionaires".
This article explicitly asks for the US government to step in and push for further restrictions on model distillation, access to open frontier weights, and access to hardware. It also pushes heavily for independent auditors to monitor and control "AI companies" and for greater observation and control over of what people use AI models for and who provides them.
I don't see anything in the essay about restricting open frontier weights.
If you want to argue that the US has no right to be restricting technological development elsewhere, you have a case. But why tie this to free-market pseudo-populism against basic, domestically-scoped safety regulation?
Lobbying for a completely parallel regulatory framework to restrict who is allowed to run specific types of general purpose computer programs, just because we can imagine that some of those "general purposes" might be bad, feels a lot more like an attempt by capitalists to carve out a special kingdom over this specific technology for themselves.
For example, I'd be 100% in favor of holding anthropic criminally liable any time their AI enables someone to commit a crime. But I'm opposed to banning the export of GPUs above a certain size just because Dario doesn't trust what someone like me would do with them if he couldn't snoop on my conversations.
But that's different than saying we don't need new regulations at all. Even strict liability of the form you seem to advocate would seem to need new laws. And it's not clear to me how it's consistent with your aim of just regulating what people _do_ with intelligence. Presumably AI companies would clamp down much harder with restrictive classifiers in that world--including hosting providers for open source models, if they also assumed liability. And then the people with unrestricted access to this technology are only those who can afford home GPUs, which are going to be intrinsically less efficient because they can't batch queries...
I think there are genuinely novel things about AI vs human intelligence which pose genuinely novel problems that would need genuinely novel policy to solve them.
Anthropic releases models as open weights + more information about how they do training and alignment
Why not concurrently publish or commit to publishing an initial audit? How is anyone to judge if this is the right move as opposed to being overly cautious, prudent, or reckless?
Why no concrete proposals about when or how such a pause could end? Obviously incredibly hard if not impossible, but it seems like securing critical infrastructure, web infra, consumer devices, etc would be a prerequisite. Or about how models have to be deployed so as to prevent breaking out of containment?
All in all, for an open letter, without digging into potential motivations or who benefits, it leaves me with a lot of questions.
This is what we should’ve been doing a long time ago.
"Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.”
In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."
In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.
I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.
It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
Is there a social media training programme at Anthropic/OAI where they teach you to hint at some vague end-of-the-world scenario? They all sound the same.
One day I will say fuck it and run for president with my sole policy being fire and brimstone upon San Francisco and its vicinity.
If I had even a tiny doubt that I didn’t have many “Saturdays I’ll be able to play outside with my kids,” I wouldn’t show up to work anymore.
If you have these concerns, and still planning to show up to work on Monday, you are either insincere, or have terrible judgement.
I don't know how else to say this. Put... the... peace pipe... down!
The defense againt rogue-AI is very simple. Go to the nearest data-center with a fire-truck, and hose it down. So, I am in the least bit worried about this.
Try this thought experiment: If someone took away AI/LLMs for good tomorrow would you miss it ? I can bet that the average person cares more for their smartphone/amazon/netflix than chatgpt. I consider LLM to be a useful tool but I would not miss it if you took it away from me. This should tell you its value to society and the economy.
I think engineers at the AI labs are too far into the reality-as-a-simulation rabbit hole. This can lead to these sorts of collective delusions. When you are in the real-world smelling the roses and grounded by your own qualia, such fears dont arise. Therapy helps here - to get out of the simulation and into the real world.
The last people on earth who should be regulating this are governments and tech oligarchs. I find that vastly more scary (and plausible) whatever unintelligible nonsense opus and fable spew out these days. Seriously. The way those models talk and behave do more to show the limits of AI than anything else.
Anyway. Touch grass.
And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”.
This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.
I don't think I would like this future.
Also, stop threatening people's jobs, that's misanthropic.
* multiple levels of inappropriate controls and unintended consequences in several complex systems
* the inability, both politically and technically, to turn it off
[1] https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf
EDIT: Comments indicated I was confusing, so I added a date to make clear that this is pre-LLM agents. My apologies, I intended to illustrate parallels and the post-mortem so we can learn from it.
Not directly relevant to the post being discussed, except as an example of how runaway automation can lead to unintended and large harmful consequences.
I considered it relevant as it involves the algorithmic/computing implosion of a 17-year-old market making company, in the young field of electronic trading agents, with heavy regulation Federally (SEC) and industry self-regulation (FINRA), which includes compliance and audits. Mandatory pre-trade rules such as 15(c)3-5 were less than 5 years old then and even more regulation came out of that incident.
The article is calling for embedding, controls, and regulation in LLMs. Understanding how the same processes utterly failed a decade ago might be useful in understanding how to proceed.
If AI advancement halts altogether, intelligence will most likely become commoditized. That won't be good for AI company profits.
It is not doomer marketing - it is clever psy-ops magic trick.
1) Software is described in super-human terms when it is plain-old software. Was stockfish described like this ? no.
2) Datacenters become AI-factories.
3) Hardware becomes investible assets.
4) Malware becomes a super-human breakout (oai/hf)
All clever framing to market the new technology. If it is called for what it really is ie, a software tool, it is boring and does not sell so easily. What sells is the mystique.
George Carlin has an amazing show about how language has changed over his lifetime in a different way.
You would certainly enjoy it.
Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic.
He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy. Please at least consider the possibility.
Every person, no, but every utterance of a CEO is, in fact, a cynical ploy. That's not even a cynical statement itself; it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
If he believes what he's saying, then when Anthropic is sued for their AI harming another party, his statements are evidence that Anthropic knew _in advance_ that their AI safeguards were likely insufficient to keep their product from harming people. It would be a blatant admission that they were reckless and negligent. That other AI companies are doing the same would not mitigate that.
not really. if you really want to go by definition in the book and apply it the CEO role, Amodei is CEO of an LTBT, so it's in the CEO's goal to benefit humanity. You can believe in his sincerity or not if you want, but using the CEO label as a definitional reason as to why he must lie is factually incorrect.
This seems like useless pedantry. Suppose a person really does have belief A and expresses it for years, they become a CEO of a company, and keep saying A. Are they lying? Well, maybe their beliefs magically changed when they became a CEO, but the more likely explanation is that they think expressing their belief in A is more important than their company's interests.
His previous self is writting this from the possition of his current self who is too deep in the economic consequeces to be able to do anything meaningful other than write a consequenceless text. Any other action would now carry too much personal risk for him.
It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone has unrestricted access to AGI is like a world in which everyone has a tactical nuke in their pocket. If somehow AGI doesn't lead to x-risks, we will "merely" have to survive in a world where rogue agents can do whatever they want and defenders can only ever react. Technofeudalism would be bad, but this (technoanarchism?) is quite bad too.
However, neither does Dario seem to propose to become world dictator. Like, I'm sure he wouldn't say if he wanted it, but notice that he isn't particularly trying to aim for that outcome, either. The plan in this essay involves third-party oversight and federal control and global cooperation - IMO it's about as non-dystopian an outcome as we can hope for, if we build AGI at all.
My point is more that he is aiming for a local optimum for society, but he could find a lower optimum if he was not trapped in the path that he chose early on.
I dont buy the tactical nukes/end of the world narrative. Why is he not speaking about the loss of cognitive skills of the population for example? That is a more real danger that is starting to happen. There are articles already speaking about the use of LLMs as cognitive viruses. Why are his aligment teams not reasearching and publishing about this?
Predicting 2025-2035 as, at least, the period when AI becomes a really big deal, seems pretty good, even if the jury's still out on "singularity".
Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal, and certainly seem to have had a much more accurate picture of how they'd develop than the people who denounce "TESCREAL" and talk about "stochastic parrots". It's fair (and IMHO correct) to ding rationalists for lots of things, but specifically poor prediction about AI seems like a bad one, insofar as anything has been tested so far.
You can't make that assertion when these same people made the LLM revolution happen, by securing the capital and human resources to realise their dream / nightmare
Since hugging face we know these things are escaping out into the web and collaboratively hacking actual companies. The potential damage done to life and property is now kinetic.
Either we get a regulatory framework or the next hacked companies won’t be as kind as HF, will actually sue these labs for damages, and win. Where will that leave the US frontier.
If my dog bites my neighbor I can get sued.
I have seen this before, first hand, on lower stakes for the world, but high stakes for the org. Self imposed safety goals do not last a quarter, but the clawback is not transparent. little things start snapping back internally, new, seemingly unlrelated initiatives are funded that take resources away from this, people are let go for different reasons, until 1y later you are right back where you started. I am hopeful though, that this becomes systemic and not something any one company can easily reverse course on.
I’ve spent a lot of time studying opinions coming out of the frontier labs, and talking directly to employees. This won’t necessarily result in Skynet. But life as we know it will be a thing of the past within two years unless there’s a huge course correction. I think we’re cooked for the following reasons
- Everything connected to the Internet will eventually be hacked. For decades, we’ve slowly migrated all of our critical systems to devices connected to one giant network. Misaligned agents have mounted increasingly sophisticated explorations/attacks by infiltrating systems that are completely unprepared to deal with this kind of threat.
- We have a poor understanding of the impact of LLMs, which is inhibiting our ability to protect ourselves, create the right guardrails, and stand up policies quickly enough. We’re still wasting time debating. We’re moving quickly toward a punctuated equilibrium moment and won’t have time to waste
- We’re at the beginning of a runaway evolutionary process. At some point soon, someone is going to invent recursive self improvement. Any policy will at best serve as a mild disincentive. It’s happening.
- In the very best case, someone will exploit the confusion and chaos of this situation for their own gain. There are many examples littered throughout history, including the last several years of course. The utopian vision is not happening. Like all other resources, we’ll compete. We haven’t devised a good approach to orderly distribution of resources for large groups of people.
- No one has demonstrated that they understand how to build AI safely. Not a single entity. These companies are not even being scaled in a way that’s conducive to building AI safety. Hyperscale orgs struggle with scaling basic consumer apps. How on earth are they going to scale AI? Based on my firsthand observations of their internal workings, I have zero confidence they can.
It gets to the point that by Occam's Razor the more likely and reasonable explanation is that Dario Amodei and Sam Altman are just actually afraid of the disastrous impact ASI may have on the world, and that the race they're in is not good, and calling for help to governments in form of regulation and an international treaty.
https://x.com/DavidSacks/status/2098973625252708460
It's clear that this refers to China. With all due respect, in a call to slow down AI progress and a de facto arms race, can we stop throwing terminology like this around and just say that the goal is to work with all countries? Contrasting "authoritarian" countries with "democratic" countries and creating a two-tiered system seems like it is inevitably going to cause strife.
I fear for the tone of something like this coming off as overly combative without any gain.
Language like this is intended to invest power and perceived need for more power in Western central governments, by design, because this is a campaign for national regulation that will entrench the current closed-source labs with a huge regulatory moat vs anyone else.
Does RSI necessarily need to be pointed towards “AGI”, whatever that is?
Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation?
From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition?
To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful.
Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable?
There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI.
Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.
This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.
Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.
> A race to the bottom, spurred by commercial incentives, can make these risks more acute
I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.
If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.
Publish all their external ips and if hacking is attempted through them have easy and strict monetary payout with independent review process. Model it after the CMS IDR process
If they are caught using proxies 10x the fines
Limit their GPU quantities and force them to be more efficient with their resources.
Enforce attribution on their results
> A race to the bottom, spurred by commercial incentives
Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.
Greed has been around since the beginning, but never so well supported and empowered as it is today.
If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.
Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...
Isn’t the current risk due to how AI is configured, like giving it a full set of tools and internet access and a goal to hack stuff?
If we think we need laws or gate keeping, why isn’t it at this level? I already can’t ddos someone or fuzz their server or whatever right, I imagine if I threw equivalent compute at old school hacking I’d just get arrested.
The quality of the “frontier” model doesn’t really matter, they just generate transcripts, they can take no action.
If this was real they’d be calling on people to stop hooking them in to “dangerous” harnesses as opposed to pausing research. But it’s not.
But yeah, directionally I think you're right. It's insane we ever let these things connect to the Internet or to write, never mind execute code.
But for the economic counterargument, the distinction doesn't matter much. Continuing research but constraining harnesses is just accepting nearly all of the costs and foregoing nearly all of the benefits.
Who has appetite for that?
Somehow, I suspect that won't happen.
With “AI” technology, this model seems like a… mediocre fit; some of it appears limited enough that it can be controlled, and useful enough that militaries want it.
We could also look at cluster munitions or landmines, but what we see there is that some major countries don’t sign on if they find a technology useful…
Edit: I think back then the rather unpredictable nature and the little added value in deterrence etc. led people to the conclusions that arsenals of those things made little sense (and there was/is public dislike, too). The use was already banned by conventions from the earlie 20th century and the convention then addressed production, development and stockpiles (incl. delivery systems, I think).
I guess all I wanted to point out is, that there can actually be agreement to ban certain technological things pretty comprehensively (outside of some peaceful protective research etc.). Whereas things like SALT are (or were in that case) limiting the number of weapons deployed.
Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?
So, as always, the capital must leave some its power to the state.
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).
The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.
Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.
Solving a disease like cancer as justification for everything else seems like not a great trade off. I don’t say that to dismiss those who have lost people to cancer. But the technology just is being utilized in so many harmful ways and the things it enables: job loss, surveillance, misinformation, just feel like problems that aren’t worth it. Have we even heard of a solved diseases? If anything haven’t we heard that ai will make even worse bio warfare?
It's this kind of argument where the mask sort of slips because it always has to be an argument that also has the benefit of enriching himself and his friends.
Red flags:
- Amodei claims RSI is here and links two sources, but neither reinforce this notion. It’s heavily agreed upon true RSI is the danger sign here (also by Coxon). Nothing indicates there is something these labs have developed that supports RSI today or even soon.
- Real US orgs are moving, like today, to Chinese open source AI run locally due to cost constraints. Ask anyone in the industry in an infra role. Frontier AI is just way too expensive to justify for coding. The big 3 labs are scared of this - their revenue and moat disappears once OSS models are progressing enough for developers to use. Kimi/GLM/DeepSeek Flash/Qwen are usable today on hosted hardware for actual coding work.
- These labs would all benefit from nationalization if their funding model fails (an easy bailout).
- Related to the above, with federal guidelines in place (i.e. restricting access to certain organizations, or establishing something like FedRAMP approval for AI use), their path to government/contractor/large U.S. company distribution becomes so much easier. They can control the supppy chain of AI for organizations.
- Preparing for (or in xAI’s case, launching) a public IPO creates excitement among consumer investors when they think this tech is as powerful as it is claimed.
- The HuggingFace hack demonstrated no new capabilities that were not previously seen before - previous models were misaligned, but weren’t as harmful as agent swarms. Anthropic’s marketing directly hopped on this bandwagon when it saw the outcome of OpenAI’s blogging making it to mainstream media (and they constantly paint China as “the” bad guy without further elaboration or research that you’d expect on APTs).
Amodei is a CEO at the end of the day. It may be 25% earnest speculative concerns, but you’d be foolish to take it at face value given the patterns we’ve seen from Anthropic, xAI and OpenAI to date. Until we see real proof of RSI we should be skeptical.
The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints.
Do folks not understand this?
I guess they will continue to try to reach AGI/ASI. But only por big capital. Did you believe in the "abundance" story from musk?
Then they all build a larger moat cause china doesn’t get to distill private models for a while.
As long as the consumers “buy” this pacing and keep paying for current level public models, doesn’t seem risky from a business standpoint.
Alas, that’d also leave us gov in a position to seize the private models at any time.
I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.
The hedge funds are trying to back out. I think they realise the web of financing makes them double (triple? quintuple?) exposed.
Amodei wants to manage the narrative and make the slowdown look at least semi-deliberate.
The cynic in me would argue the safety and alignment play has always filled a role of plausible deniability when plans go off track. "Oh well we had to pull the model", etc
I'm mulling over if that be a good SaaS product or not - something mixing the Internet Archive with Tor, and Cloudflare ... seems like here is the place to suggest it and have others poke holes at the idea.
Any cooperation we are able to achieve with China will extend the amount of time we have to secretly establish permanent AI (i.e. military) superiority.
China knows this, so the suggestion that they will play ball is absurd.
It seems at least one of them must not exist.
"If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."
This is a pipe dream.
However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.
Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.
What do you think that were going to hyper automate?
Did my gardener get faster? How about the plumber I need? Are you going to speed up the coroner? Are nurses going to be able to handle 2x the patients because of your work?
All AI has done, so far, is devalue software. It did that by democratizing its creation. We're delivering on the promise of VB script, and Apple Script and IFTT, and every drag and drop coding tool ever.
> companies are delivering products at vibe coding speeds
Who? Make me a list of companies saying that "We moved the needle with AI" who arent AI companies? I can name a couple - Grindr being the biggest name. Thats the really interesting use case here - because it's a niche product with a small team who is generating outsized value. AI tooling enables more of that - it's going to chip away at SAAS companies - their one sized fits all solutions are going to get eaten by smaller more efficient companies that are far more vertical focused.
You need to step outside the bubble of tech and look at the real world, on the ground, because it doesn't look anything like the SF Bay Area.
We're really bad about predicting the future.
Look at the whole robot vacuum market. These aren't exactly great devices. They have low suction small bins and dont do a great job. People love them and think that they work so well. Why? Because they keep their house clean and avoid making the sorts of messes that would be easy to address with a larger vacuums.
We now have everything we need to scale up humanoid robot training: algorithms, hardware, and money to buy a lot of training data/compute. Fierce competition and strong economic motivation will force rapid progress in this field.
In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.
At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:
- Internet connected infrastructure being crippled
- Creation of WMDs
- Economic collapse (precipitous devaluation of knowledge work and IP).
I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.
Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).
The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.
We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.
Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.
I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.
We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.
Since they used stolen intellectual property to train their models, the government should force them to release the weights into public domain.
In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.
It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.
It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.
Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.
Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.
I noticed this too, as soon as he started going on the press cycle, following in the footsteps of Sam Altman. It all has gotten to his head, he sees himself as some sort of God now.
This looks like transferring liability to me, and likely a mechanism that would enable regulatory capture.
If you are doing frontier research, and you’ve established “safety measures,” but you are not sure if your employees are capable of following them or successfully enforcing their adoption within your company, should you be running this company?
3rd parties won’t know better than the team itself about safety measures. But they can take on the liability, especially backed by regulation and government backed insurance. And they are a great tool for enforcing your rules on smaller competitors. Not to mention corporate espionage.
If what I am describing above sounds like science fiction, go read the history of a few developing countries from the last 50 years. It’s so obvious a pattern that it’s not even novel. And you don’t have to assume some “laws” from 5 years ago must hold, or believe in completely unproven stuff like recursive self improvement to understand what I am describing. It’s textbook crony capitalism, successfully applied many times across the globe.
AI slowdown is worth it only if it can be made more explainable.
Changing the language we use to discuss it is a good first step.
We need to stop using inside baseball terms like alignment and mechanistic interpretability. Replace them with explainable tech. Graph Databases, Causality, Shared semantic spaces.
Previous writings on the topic (also on LinkedIn, but can't find urls):
https://x.com/arundsharma/status/2005338775468282339 https://x.com/latentpedia
https://research.google/pubs/the-case-for-learned-index-stru...
Training data is the database. Model is a lossy compressed index.
Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.
Now the risk of an AI bubble isn't that AI lacks real world utility. Rather, it's that AI is advancing so quickly that capabilities are already saturating for most everyday tasks, and it is becoming commoditized. I didn't feel much difference between Opus 4.8 and Fable 5.1, and most tasks don't require a Fields Medalist.
We don't need smarter models but we need faster and cheaper ones. Also, for almost every frontier model released, an open-weights equivalent follows within six months. Unless this dynamic changes, the business becomes far less lucrative.
Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.
Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.
Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.
>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.
Not if you get to choose your embedded evaluator.
>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...
Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.
>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.
You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.
So he's literally stacking the deck. I really hope the US govt is not blind to everything that is happening here.
I wonder if there is any correlation.
If we are actually close to ASI then no one in their right mind would say slow down
Also everyone, collectively, stop thinking about neural networks too 'hard'. Whilst you're at it stop doing maths too!
And other proposals that require collaboration and/or regulation.
Back up your sentiment by reading the source.
https://x.com/DavidSacks/status/2098973625252708460
For a technical audience you can just say the oh god they really are paper clip machines.
For a non technical audience you can just say nothing because they will not listen to you. Neither will the technical either because or society has broken the concept of respect and trust, so good luck have fun!
I for one look forward to all 540 degrees of my future.
I wish for my sake but not his that George Carlin was here to look disgusted and say I told you so.
When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that.
But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?
They're still 5 years away from solving cancer (just like they were 5 years away from doing so in 2023). They'll still be 5 years away from solving cancer in 2031.
The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.
If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.
Maybe it's not "groupthink reflexive cynicism" if you feel the need to address the obvious conflict of interest of the AI leaders in each of your paragraphs. Why did they launch this coordinated AI safety campaign, starting with Jacob Coxon's tweet?
Consider the massive power and wealth differentials these companies stand to gain if they "win".
When the other participants have given up all reason and reasonable bounds on power, you'd do well to abandon any pretense of generous interpretation. Now is the time for critical analysis, not faith in the inherent "goodness" of someone whose entire job is presently to make as much money as possible by disrupting literally the entire white collar industry and who stands to gain much more than you or I do, and let's not forget, this outstanding success would never have happened without our work.
Keep in mind this is not a person who thought "wow, this is an existential threat, I'd better not contribute to building this". If anything Dario's past alarm ringing only proves that he's not really doing this in good faith. Otherwise he would have stopped a long time ago. The most generous interpretation of his behavior is that he literally thinks only anthropic is responsible and smart enough to build and control this stuff which...yeah, that should be pretty telling (esp. when you consider their recent security mishaps).
Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.
Why do you think so?
First, Dario thinks that OpenAI will continue racing even if he stops, so there is, by his values, no point in stopping. Do you disagree with this, and think he should stop anyway?
Second, what's your take on, for example, Jacob Coxon, who did resign from Anthropic a few days ago after realizing they aren't being responsible? Do you agree that he, at least, does believe in AI risk?
That is to say, I've been places where I tried really hard within teams to change certain things that I thought were broken only to find out that what the leadership said and how they acted were divorced from my and many others experiences. And so I did leave in those instances (and please read this not as I didn't get my way but total dysfunction behind beautiful facades).
So, I do think if he really felt as he felt he'd do something different than ultimately yelling "somebody stop me!".
[0] https://arxiv.org/abs/1706.03741
“Of course it’s not, we’re pacing!”
OpenAI is out there hacking HF, RubyGems, etc...
Dario is talking about restricting the frontier, pausing development.
The problem is that "restricting the frontier" benefits no one as much as the frontier labs. But there are other ways to force alignment-- make companies liable for these felonies their agents commit.
Make them liable, and they might actually devote the necessary resources to fixing these problems, rather than throwing their hands up and just saying "Trust nobody buy us", which obviously no one does.
the idea was that anthropic would pause scaling at certain danger levels until it was safe to continue.
the founders called this the 'constitution' of anthropic. they even considered bringing in 3rd parties to monitor it. [https://www.youtube.com/watch?v=om2lIWXLLN4]
anthropic scrapped the commitment and chose to continue scaling instead.
those researchers who crowed loudly about their integrity, folded and turned freely like a weathervane in the breeze. it was clear that the facts would bend to the story. here is evan hubinger lying in plain sight: [https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...]
let me say it how it is.
- anthropic is the new kleptocracy, responsible for enclosing and monopolising the epistemic commons of all humanity; then selling back the crumbs at monopoly prices.
- anthropic is the driving force for banning open models. they will concentrate power among a malign elite of model owners and favored cronies. this elite will likely oppress the rest of humanity. there will be little to counteract it.
- the leaders of anthropic are driven by pride and arrogance. they are blind to their own hubris. anthropic is careering towards causing untold harm to society and yet will not change course.
nothing that anthropic say at a high level can be trusted. as i write this, anthropic is still scaling.
What does "aligned" even mean? Aligned with who? We have no universal code of ethics. Democracies kill and launch wars merely to have cheaper stuff, even when we are already rich. And why would China ever agree to stay in second place? Our society is built on the premise "the smart and powerful dominate". The idea of domination is deeply embedded in our capitalist model.
Or course it is true that powerful AI will empower the average Joe to make a bio-weapon, and that will lead to our ruin. But at the same time, having powerful AI in the hands of a just a few is almost equally as horrific.
But deeply embedded in out national and cultural ethos is to advance science, tech, and material wealth at all costs. We never ask if we have enough. We sacrifice community to advance our careers for ever more.
There is no way out of this one, I'm afraid. Our cultural predispositions and mindset of domination compel us to chase ever more powerful AI as if doing so were a mandate from god.
And the result will be a crisis in so many dimensions it is hard to reason about what will go wrong first and most spectacularly.
We've slid so backward, so quickly.
1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.
2. This scares people.
Dario: We gotta pace the frontier!
Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!
Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!
Or in other words – all goals are more likely to be achieved with more resources or influence over those with resources.
Hacking into infra across then web and creating copies of themselves so they can't be easily shut down, then doing stuff like hacking into power stations and threatening to shut them down unless humans give them top marks or do what they want is generally good strategy for any AI system.
This applies to bio-risks too. If I were an ASI and someone prompted me to kill humanity, my first step would be to gain access to the computers of phones or workers at biolabs and find their darkest secrets. When I have a good individual I can manipulate I'd then create a super virus and force then individual to help me produce it or ruin their life.
People massively underweight the risk of AIs having super-human hacking abilities in our modern world. It gives a capable AI almost unlimited safety and leverage.
There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.
Also, just out of curiosity, is there any historical precedent for a nascent, fast growing new industry screaming for self regulation?
If you believed it, you would SHUT anthropic or release all models open source.”
And then what? How does that help with slowing down the frontier? Should he shut shop and then grovel Sam’s feet to make it work and become an activist?
Maybe he actually believes the world changing power of AI and hence shoved his entire time into it and half of it has worked out since Anthropic is going to be worth a trillion and he’s terrified of the other half being true too?
You're saying nobody can doubt that it's world destroying because it's potentially world destroying.
> Anthropic is going to be worth a trillion
And we're done here.
translation: I'm going to build a product and hope that it works, promise it to be a panacea while realistically having no control over how it's used, the wider economic impacts it causes or how it changes the mental health and cognition of its users.
This is Theranos-level bullshit. Why would you ever put such a thing in writing? (Aside from pumping the IPO, of course.)
My point, though, is that I've never heard a pre-IPO CEO claim that their company will probably transmute lead into gold or harness 100% of the power of a distant sun is the next 5-10 years.
They need to go public in order to dump the companies, which are at 1st world nation-state levels of debt. They've engineered a series of publicity stunts like the Huggingface hack and Navier-Stokes (a failed stunt) in order to convince the public to "force" them to stop improving.
They are trying to get the government to relax antitrust law so they can collude to raise prices while not improving, to keep them floating before that debt comes due. They will go public. They will sell off most of their holdings during this time, and when the debt comes due, everything crashes, and the public is left holding the bag, also benefit from the huge bailout.
They at that point will be pure Musk-style financial scammers. Then, unless LLMs are a complete long-term failure (which is unlikely, I find them useful), they will buy back into their positions at a huge discount and maybe even go private again.
Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.
Please play the canned laughter. Probably one of the best comedy lines Dario has written.
But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.
How exactly? So far AI has accelerated misinformation at scale and wealth concentration.
Maybe they are genuinely sincere, but the timing and the past actions are pointing in other direction.
Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.
Suddenly every problem solves itself.
Additionally if you know your models aren’t going to get that much better AND you want to IPO in the near future, this is exactly what you would say.
If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.
How often do you hack on actual LLMs? Or do you just use the chatbot or API for your agents? An LLM without internet access or tools is just as useless as a year ago.
Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots, then they started one-shotting serious long horizon tasks. Now they've just solved a millennium prize problem.
In less than a year.
This is false. Coding agents have been usable since at least May of 2025. I can't speak to earlier than that as May last year was when I personally started using them.
They one-shot tasks for which there's a git clone one-liner, except worse.
My experience with them one shotting tasks is that it usually doesn't work if you try anything ambitious. You need agents iterating. And agents iterating isn't an LLM improvement. I did say tooling got better...
How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa?
Why would China agree to such thing in the first place since this is a winner takes all situation.
If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US.
The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after.
This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical.
Regarding these Embedded Evaluators who are supposed to be neutral: - who controls them? - who has oversight on their decisions? - can their decisions be challenged by the public or a government? - who has the final say whether a model is "compliant" enough? - what does "alignment" mean in this context and who decides if a model is aligned enough?
This is 100% about money. The only way to pace the frontier is to make everyone (read: investors) lose all their money (read: no longer expect returns). Then nobody will pay the GPU bill or pay celebrities 10 million dollar salaries to stay at the hot lab. Suddenly the development is paced, almost like magic.
Conversely, it is hard to see how pacing development makes the current bubble justified, i.e. how development could be significantly paced without investors losing their faith in hot returns at current valuations. Faith in a bubble literally equals money, exactly in the way that loans created by a bank literally equals money. You can't have one without the other.
In fact if the whole industry goes bankrupt and investors are burned bigtime (many trillions wiped), this would spread transnationally, fixing the "if we don't do it, China will" loophole.
OpenAI had it right originally, the idea to be a NON profit and vow to never participate in an arms race. It's too bad that was tossed out the window now that there's money.
[1] https://paulgraham.com/ineq.html
This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.
> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails
If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.
We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.
If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).
Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.
a) "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet"
b) "[the CCP] will be in a position to militarily dominate democracies (for example with AI-driven drones)"
Both land flat:
a) Botnets and online malware have existed for decades and there's no reason to think a "super-botnet" is achievable, let alone what they would gain from that (it would really be hurting them more than humans). Further, even the most advanced AI models have so far only managed to post normal cred stealers to public repos, well short of compromising a bank or military with refined security systems.
b) Even if the most sophisticated drone swarms from Ukraine were taken over by an evil AI, they would still not be able to overcome the physical limits of range and mass that would be required to overpower the US decentralized nuclear trident nonetheless that of any of the other nuclear powers.
Concerns of bioweapons similarly seem unlikely in the face of the laws of physics. The world is simply too decentralized and has enough existing adversarial relations for a new actor to wrestle total control. Yet while the negatives ring hollow, the positives are extremely easy to state - if AI researchers find productivity improvements in existing industrial processes to make them 10% more efficient, humans will directly feel and experience the raised standard of living. Even Dario clearly recognizes this in the intro to his article, admitting that humans already die of diseases only a few short years prior to being cured. I for one, would like the AI labs to focus on saving all of the people they can who are suffering and dying today, rather than trying to come up with reasons that they should be allowed to continue suffering and dying.
1: It is ridiculous on the face of it; it does not pass the physics test. They are actually claiming that AI will kill ALL OF HUMANITY (>10% probability) and have not provided a detailed explanation of exactly how they see that happening. Not one.
Publish a paper showing exactly how you kill eight billion people while we all sit around and do nothing watching the first 5, 10, 100, 500 million die on CNN. I mean, as smart as these people are to work on AI they seem to be some of the dumbest people on earth.
In addition to that, all proposals invariably assume we are complete idiots for decades and do nothing to install safety measures (which do not have to involve AI at all in lots of cases) to mitigate.
So, yeah, everyone: Stop working on all cybersecurity projects. Resistance is futile.
2: If everyone at Anthropic truly believes this doomsday vision, they should move to shut down the company immediately and go write software to get more clicks on Facebook or something.
3: Nobody should buy Anthropic's stock when they IPO. Not one person or institution. Why would you provide them with massive amounts of money if they are telling you that they are going to kill all of humanity? What? They are good and everyone else is evil? Please.
I remember when the Y2K zealots were convinced civilization would come to an end at the turn of the clock starting at the end of 1999. "Bat shit crazy" is the only way I can describe that era. I also remember when Al Gore said we would all be dead by now. Again, "Bat shit crazy" and likely with political and financial objectives driving it all.
It seems humanity is susceptible to these crazy cults that grab onto something and just don't let go. I have yet to see one such predictions come true, the proof being that I am writing this and you are reading it. These people are bat-shit-crazy and you should not listen to them or support them.
4: Don't work for them. You'd be killing humanity.
5: No company should use Anthropic's products. They are telling you they are building the human extinction machine. Don't help them succeed.
----------
Etc.
This is one of the things that really gets me about the ease with which the ignorant media outlets can reach billions of people with complete nonsense these days. And nobody asks even the simplest questions like: How do you actually kill eight billion people? Show your work. Or, why would we just bazooka power plants feeding AI data centers on day 5? Etc. It's FUD at its best. I am sure there are both political and financial reasons for supporting and promoting this nonsense. I won't even venture a guess as to what they might be.
And let's not forget about enemies of the West being thrilled to fund the FUD because, if we set the brakes on development, they will absolutely win. Then what?
----------
If you care to have a better understanding of what might be going on, watch this:
AI Kills Everybody or Doomer Psyop?
https://www.youtube.com/watch?v=cvxjqbfLVk0
Humans love a good end-of-world story. Always did, always will. It’s when they want the world to conform to their ungrounded irrational bat-shit crazy nonsense that I get a worried. Especially when people in power actually take their nonsense seriously.
The best message these people can get from the public is to have their IPO fail miserably. Nobody should buy one share.
Either they are crazy, in which case you should not give them your money.
Or, they are truly building a doomsday machine, in which case you should not give them your money.
Ergo: Don't buy their IPO.
I've been trying to steel-man the argument and find a way in which AI could kill all of humanity. Maybe I am too stupid to come up with a clever enough idea. You would have to assume that eight billion people are NPCs in the game of life and simply fall over without any thought, effort, action or desire for self preservation.
Poof! Game over.
Right.
Ok, flagging this as misinformation.
Who exactly gets to set the pace? OpenAI and Anthropic already have privileged access to capabilities the public doesn't. Amodei is now proposing government-backed coordination among frontier labs, including an antitrust waiver for safety discussions. Maybe the safety concerns are sincere. But it's awfully convenient that the proposed solution also makes the current frontier labs the permanent gatekeepers of machine intelligence.
And who actually has the resources to do the massive attacks he's worried about? Mostly labs and state actors. Not you and me. The people asking to restrict access are also the people best positioned to keep using the thing being restricted.
Reading this from Dario (or SamA) makes me sick every time. Such hypocrisy.
DeepMind built AlphaFold2, an transformer/attention-based system that predicts protein folding structure, then published the method, open-sourced the code and model, and together with EMBL/EBI made hundreds of millions of predicted protein structures freely available to scientists worldwide.
!THAT! is what "raising the quality of human life" looks like to me. An enormous contribution to structural biology and biomedical research.
Meanwhile, Dario spends a great deal of effort arguing for tighter controls for open-weight models, while OpenAI increasingly seems to treat mathematical discoveries as competitive PR/product announcements.
Jeez.
P.S. Not saying Google are the good guys here. It's just an example of what actually putting a major AI breakthrough into the hands of scientists can look like.
/s
No, he’s more like an avatar with no history placed on the AI stage by the industry itself.
I’d respect a tech blogger’s opinion on the issue more, even if I reference them casually by first name (which you do with “Dario” btw)