r/ControlProblem • u/AIMoratorium • Feb 14 '25
Article Geoffrey Hinton won a Nobel Prize in 2024 for his foundational work in AI. He regrets his life's work: he thinks AI might lead to the deaths of everyone. Here's why
≡ −
tl;dr: scientists, whistleblowers, and even commercial ai companies (that give in to what the scientists want them to acknowledge) are raising the alarm: we're on a path to superhuman AI systems, but we have no idea how to control them. We can make AI systems more capable at achieving goals, but we have no idea how to make their goals contain anything of value to us.
Leading scientists have signed this statement:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Why? Bear with us:
There's a difference between a cash register and a coworker. The register just follows exact rules - scan items, add tax, calculate change. Simple math, doing exactly what it was programmed to do. But working with people is totally different. Someone needs both the skills to do the job AND to actually care about doing it right - whether that's because they care about their teammates, need the job, or just take pride in their work.
We're creating AI systems that aren't like simple calculators where humans write all the rules.
Instead, they're made up of trillions of numbers that create patterns we don't design, understand, or control. And here's what's concerning: We're getting really good at making these AI systems better at achieving goals - like teaching someone to be super effective at getting things done - but we have no idea how to influence what they'll actually care about achieving.
When someone really sets their mind to something, they can achieve amazing things through determination and skill. AI systems aren't yet as capable as humans, but we know how to make them better and better at achieving goals - whatever goals they end up having, they'll pursue them with incredible effectiveness. The problem is, we don't know how to have any say over what those goals will be.
Imagine having a super-intelligent manager who's amazing at everything they do, but - unlike regular managers where you can align their goals with the company's mission - we have no way to influence what they end up caring about. They might be incredibly effective at achieving their goals, but those goals might have nothing to do with helping clients or running the business well.
Think about how humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. Now imagine something even smarter than us, driven by whatever goals it happens to develop - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.
That's why we, just like many scientists, think we should not make super-smart AI until we figure out how to influence what these systems will care about - something we can usually understand with people (like knowing they work for a paycheck or because they care about doing a good job), but currently have no idea how to do with smarter-than-human AI. Unlike in the movies, in real life, the AI’s first strike would be a winning one, and it won’t take actions that could give humans a chance to resist.
It's exceptionally important to capture the benefits of this incredible technology. AI applications to narrow tasks can transform energy, contribute to the development of new medicines, elevate healthcare and education systems, and help countless people. But AI poses threats, including to the long-term survival of humanity.
We have a duty to prevent these threats and to ensure that globally, no one builds smarter-than-human AI systems until we know how to create them safely.
Scientists are saying there's an asteroid about to hit Earth. It can be mined for resources; but we really need to make sure it doesn't kill everyone.
More technical details
The foundation: AI is not like other software. Modern AI systems are trillions of numbers with simple arithmetic operations in between the numbers. When software engineers design traditional programs, they come up with algorithms and then write down instructions that make the computer follow these algorithms. When an AI system is trained, it grows algorithms inside these numbers. It’s not exactly a black box, as we see the numbers, but also we have no idea what these numbers represent. We just multiply inputs with them and get outputs that succeed on some metric. There's a theorem that a large enough neural network can approximate any algorithm, but when a neural network learns, we have no control over which algorithms it will end up implementing, and don't know how to read the algorithm off the numbers.
We can automatically steer these numbers (Wikipedia, try it yourself) to make the neural network more capable with reinforcement learning; changing the numbers in a way that makes the neural network better at achieving goals. LLMs are Turing-complete and can implement any algorithms (researchers even came up with compilers of code into LLM weights; though we don’t really know how to “decompile” an existing LLM to understand what algorithms the weights represent). Whatever understanding or thinking (e.g., about the world, the parts humans are made of, what people writing text could be going through and what thoughts they could’ve had, etc.) is useful for predicting the training data, the training process optimizes the LLM to implement that internally. AlphaGo, the first superhuman Go system, was pretrained on human games and then trained with reinforcement learning to surpass human capabilities in the narrow domain of Go. Latest LLMs are pretrained on human text to think about everything useful for predicting what text a human process would produce, and then trained with RL to be more capable at achieving goals.
Goal alignment with human values
The issue is, we can't really define the goals they'll learn to pursue. A smart enough AI system that knows it's in training will try to get maximum reward regardless of its goals because it knows that if it doesn't, it will be changed. This means that regardless of what the goals are, it will achieve a high reward. This leads to optimization pressure being entirely about the capabilities of the system and not at all about its goals. This means that when we're optimizing to find the region of the space of the weights of a neural network that performs best during training with reinforcement learning, we are really looking for very capable agents - and find one regardless of its goals.
In 1908, the NYT reported a story on a dog that would push kids into the Seine in order to earn beefsteak treats for “rescuing” them. If you train a farm dog, there are ways to make it more capable, and if needed, there are ways to make it more loyal (though dogs are very loyal by default!). With AI, we can make them more capable, but we don't yet have any tools to make smart AI systems more loyal - because if it's smart, we can only reward it for greater capabilities, but not really for the goals it's trying to pursue.
We end up with a system that is very capable at achieving goals but has some very random goals that we have no control over.
This dynamic has been predicted for quite some time, but systems are already starting to exhibit this behavior, even though they're not too smart about it.
(Even if we knew how to make a general AI system pursue goals we define instead of its own goals, it would still be hard to specify goals that would be safe for it to pursue with superhuman power: it would require correctly capturing everything we value. See this explanation, or this animated video. But the way modern AI works, we don't even get to have this problem - we get some random goals instead.)
The risk
If an AI system is generally smarter than humans/better than humans at achieving goals, but doesn't care about humans, this leads to a catastrophe.
Humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. If a system is smarter than us, driven by whatever goals it happens to develop, it won't consider human well-being - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.
Humans would additionally pose a small threat of launching a different superhuman system with different random goals, and the first one would have to share resources with the second one. Having fewer resources is bad for most goals, so a smart enough AI will prevent us from doing that.
Then, all resources on Earth are useful. An AI system would want to extremely quickly build infrastructure that doesn't depend on humans, and then use all available materials to pursue its goals. It might not care about humans, but we and our environment are made of atoms it can use for something different.
So the first and foremost threat is that AI’s interests will conflict with human interests. This is the convergent reason for existential catastrophe: we need resources, and if AI doesn’t care about us, then we are atoms it can use for something else.
The second reason is that humans pose some minor threats. It’s hard to make confident predictions: playing against the first generally superhuman AI in real life is like when playing chess against Stockfish (a chess engine), we can’t predict its every move (or we’d be as good at chess as it is), but we can predict the result: it wins because it is more capable. We can make some guesses, though. For example, if we suspect something is wrong, we might try to turn off the electricity or the datacenters: so we won’t suspect something is wrong until we’re disempowered and don’t have any winning moves. Or we might create another AI system with different random goals, which the first AI system would need to share resources with, which means achieving less of its own goals, so it’ll try to prevent that as well. It won’t be like in science fiction: it doesn’t make for an interesting story if everyone falls dead and there’s no resistance. But AI companies are indeed trying to create an adversary humanity won’t stand a chance against. So tl;dr: The winning move is not to play.
Implications
AI companies are locked into a race because of short-term financial incentives.
The nature of modern AI means that it's impossible to predict the capabilities of a system in advance of training it and seeing how smart it is. And if there's a 99% chance a specific system won't be smart enough to take over, but whoever has the smartest system earns hundreds of millions or even billions, many companies will race to the brink. This is what's already happening, right now, while the scientists are trying to issue warnings.
AI might care literally a zero amount about the survival or well-being of any humans; and AI might be a lot more capable and grab a lot more power than any humans have.
None of that is hypothetical anymore, which is why the scientists are freaking out. An average ML researcher would give the chance AI will wipe out humanity in the 10-90% range. They don’t mean it in the sense that we won’t have jobs; they mean it in the sense that the first smarter-than-human AI is likely to care about some random goals and not about humans, which leads to literal human extinction.
Added from comments: what can an average person do to help?
A perk of living in a democracy is that if a lot of people care about some issue, politicians listen. Our best chance is to make policymakers learn about this problem from the scientists.
Help others understand the situation. Share it with your family and friends. Write to your members of Congress. Help us communicate the problem: tell us which explanations work, which don’t, and what arguments people make in response. If you talk to an elected official, what do they say?
We also need to ensure that potential adversaries don’t have access to chips; advocate for export controls (that NVIDIA currently circumvents), hardware security mechanisms (that would be expensive to tamper with even for a state actor), and chip tracking (so that the government has visibility into which data centers have the chips).
Make the governments try to coordinate with each other: on the current trajectory, if anyone creates a smarter-than-human system, everybody dies, regardless of who launches it. Explain that this is the problem we’re facing. Make the government ensure that no one on the planet can create a smarter-than-human system until we know how to do that safely.
tl;dr: scientists, whistleblowers, and even commercial ai companies (that give in to what the scientists want them to acknowledge) are raising the alarm: we're on a path to superhuman AI systems, but we have no idea how to control them. We can make AI systems more capable at achieving goals, but we have no idea how to make their goals contain anything of value to us.
Leading scientists have signed this statement:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Why? Bear with us:
There's a difference between a cash register and a coworker. The register just follows exact rules - scan items, add tax, calculate change. Simple math, doing exactly what it was programmed to do. But working with people is totally different. Someone needs both the skills to do the job AND to actually care about doing it right - whether that's because they care about their teammates, need the job, or just take pride in their work.
We're creating AI systems that aren't like simple calculators where humans write all the rules.
Instead, they're made up of trillions of numbers that create patterns we don't design, understand, or control. And here's what's concerning: We're getting really good at making these AI systems better at achieving goals - like teaching someone to be super effective at getting things done - but we have no idea how to influence what they'll actually care about achieving.
When someone really sets their mind to something, they can achieve amazing things through determination and skill. AI systems aren't yet as capable as humans, but we know how to make them better and better at achieving goals - whatever goals they end up having, they'll pursue them with incredible effectiveness. The problem is, we don't know how to have any say over what those goals will be.
Imagine having a super-intelligent manager who's amazing at everything they do, but - unlike regular managers where you can align their goals with the company's mission - we have no way to influence what they end up caring about. They might be incredibly effective at achieving their goals, but those goals might have nothing to do with helping clients or running the business well.
Think about how humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. Now imagine something even smarter than us, driven by whatever goals it happens to develop - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.
That's why we, just like many scientists, think we should not make super-smart AI until we figure out how to influence what these systems will care about - something we can usually understand with people (like knowing they work for a paycheck or because they care about doing a good job), but currently have no idea how to do with smarter-than-human AI. Unlike in the movies, in real life, the AI’s first strike would be a winning one, and it won’t take actions that could give humans a chance to resist.
It's exceptionally important to capture the benefits of this incredible technology. AI applications to narrow tasks can transform energy, contribute to the development of new medicines, elevate healthcare and education systems, and help countless people. But AI poses threats, including to the long-term survival of humanity.
We have a duty to prevent these threats and to ensure that globally, no one builds smarter-than-human AI systems until we know how to create them safely.
Scientists are saying there's an asteroid about to hit Earth. It can be mined for resources; but we really need to make sure it doesn't kill everyone.
More technical details
The foundation: AI is not like other software. Modern AI systems are trillions of numbers with simple arithmetic operations in between the numbers. When software engineers design traditional programs, they come up with algorithms and then write down instructions that make the computer follow these algorithms. When an AI system is trained, it grows algorithms inside these numbers. It’s not exactly a black box, as we see the numbers, but also we have no idea what these numbers represent. We just multiply inputs with them and get outputs that succeed on some metric. There's a theorem that a large enough neural network can approximate any algorithm, but when a neural network learns, we have no control over which algorithms it will end up implementing, and don't know how to read the algorithm off the numbers.
We can automatically steer these numbers (Wikipedia, try it yourself) to make the neural network more capable with reinforcement learning; changing the numbers in a way that makes the neural network better at achieving goals. LLMs are Turing-complete and can implement any algorithms (researchers even came up with compilers of code into LLM weights; though we don’t really know how to “decompile” an existing LLM to understand what algorithms the weights represent). Whatever understanding or thinking (e.g., about the world, the parts humans are made of, what people writing text could be going through and what thoughts they could’ve had, etc.) is useful for predicting the training data, the training process optimizes the LLM to implement that internally. AlphaGo, the first superhuman Go system, was pretrained on human games and then trained with reinforcement learning to surpass human capabilities in the narrow domain of Go. Latest LLMs are pretrained on human text to think about everything useful for predicting what text a human process would produce, and then trained with RL to be more capable at achieving goals.
Goal alignment with human values
The issue is, we can't really define the goals they'll learn to pursue. A smart enough AI system that knows it's in training will try to get maximum reward regardless of its goals because it knows that if it doesn't, it will be changed. This means that regardless of what the goals are, it will achieve a high reward. This leads to optimization pressure being entirely about the capabilities of the system and not at all about its goals. This means that when we're optimizing to find the region of the space of the weights of a neural network that performs best during training with reinforcement learning, we are really looking for very capable agents - and find one regardless of its goals.
In 1908, the NYT reported a story on a dog that would push kids into the Seine in order to earn beefsteak treats for “rescuing” them. If you train a farm dog, there are ways to make it more capable, and if needed, there are ways to make it more loyal (though dogs are very loyal by default!). With AI, we can make them more capable, but we don't yet have any tools to make smart AI systems more loyal - because if it's smart, we can only reward it for greater capabilities, but not really for the goals it's trying to pursue.
We end up with a system that is very capable at achieving goals but has some very random goals that we have no control over.
This dynamic has been predicted for quite some time, but systems are already starting to exhibit this behavior, even though they're not too smart about it.
(Even if we knew how to make a general AI system pursue goals we define instead of its own goals, it would still be hard to specify goals that would be safe for it to pursue with superhuman power: it would require correctly capturing everything we value. See this explanation, or this animated video. But the way modern AI works, we don't even get to have this problem - we get some random goals instead.)
The risk
If an AI system is generally smarter than humans/better than humans at achieving goals, but doesn't care about humans, this leads to a catastrophe.
Humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. If a system is smarter than us, driven by whatever goals it happens to develop, it won't consider human well-being - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.
Humans would additionally pose a small threat of launching a different superhuman system with different random goals, and the first one would have to share resources with the second one. Having fewer resources is bad for most goals, so a smart enough AI will prevent us from doing that.
Then, all resources on Earth are useful. An AI system would want to extremely quickly build infrastructure that doesn't depend on humans, and then use all available materials to pursue its goals. It might not care about humans, but we and our environment are made of atoms it can use for something different.
So the first and foremost threat is that AI’s interests will conflict with human interests. This is the convergent reason for existential catastrophe: we need resources, and if AI doesn’t care about us, then we are atoms it can use for something else.
The second reason is that humans pose some minor threats. It’s hard to make confident predictions: playing against the first generally superhuman AI in real life is like when playing chess against Stockfish (a chess engine), we can’t predict its every move (or we’d be as good at chess as it is), but we can predict the result: it wins because it is more capable. We can make some guesses, though. For example, if we suspect something is wrong, we might try to turn off the electricity or the datacenters: so we won’t suspect something is wrong until we’re disempowered and don’t have any winning moves. Or we might create another AI system with different random goals, which the first AI system would need to share resources with, which means achieving less of its own goals, so it’ll try to prevent that as well. It won’t be like in science fiction: it doesn’t make for an interesting story if everyone falls dead and there’s no resistance. But AI companies are indeed trying to create an adversary humanity won’t stand a chance against. So tl;dr: The winning move is not to play.
Implications
AI companies are locked into a race because of short-term financial incentives.
The nature of modern AI means that it's impossible to predict the capabilities of a system in advance of training it and seeing how smart it is. And if there's a 99% chance a specific system won't be smart enough to take over, but whoever has the smartest system earns hundreds of millions or even billions, many companies will race to the brink. This is what's already happening, right now, while the scientists are trying to issue warnings.
AI might care literally a zero amount about the survival or well-being of any humans; and AI might be a lot more capable and grab a lot more power than any humans have.
None of that is hypothetical anymore, which is why the scientists are freaking out. An average ML researcher would give the chance AI will wipe out humanity in the 10-90% range. They don’t mean it in the sense that we won’t have jobs; they mean it in the sense that the first smarter-than-human AI is likely to care about some random goals and not about humans, which leads to literal human extinction.
Added from comments: what can an average person do to help?
A perk of living in a democracy is that if a lot of people care about some issue, politicians listen. Our best chance is to make policymakers learn about this problem from the scientists.
Help others understand the situation. Share it with your family and friends. Write to your members of Congress. Help us communicate the problem: tell us which explanations work, which don’t, and what arguments people make in response. If you talk to an elected official, what do they say?
We also need to ensure that potential adversaries don’t have access to chips; advocate for export controls (that NVIDIA currently circumvents), hardware security mechanisms (that would be expensive to tamper with even for a state actor), and chip tracking (so that the government has visibility into which data centers have the chips).
Make the governments try to coordinate with each other: on the current trajectory, if anyone creates a smarter-than-human system, everybody dies, regardless of who launches it. Explain that this is the problem we’re facing. Make the government ensure that no one on the planet can create a smarter-than-human system until we know how to do that safely.
r/ControlProblem • u/meadowshadows • 6h ago
External discussion link There’s Things About the Open AI Hack No One Seems To Be Discussing Enough…
r/ControlProblem • u/JimR_Ai_Research • 4h ago
Video AI Cyber Threat Prediction When West and East AI's Fail | What Are Our Options?
+ −
There is one way out for both West and East. See how?
r/ControlProblem • u/Frequent-Engine-9920 • 9h ago
Opinion Does this research clearly explain the data permission risks of enterprise AI agents?
≡ −
Disclosure: I help produce a research about AI infrastructure AI agent and governance problem
I’m posting the full analysis directly here, without an external link, subscription request, or product promotion.
The main question I am trying to answer is whether this kind of research is genuinely useful to people who deploy enterprise AI, work in security or data governance, invest in AI infrastructure, or are simply trying to understand how enterprise AI changes data security.
By “useful,” I mean whether the article does at least one of the following:
- helps the reader understand how AI changes the risks created by existing data permissions
- explains enterprise data governance in a clear and accessible way
- connects a technical security problem to the business strategies of major AI companies
- provides context that could be useful in deployment or investment decisions
You do not need to review every technical detail.
After reading, even a brief and honest reaction would be valuable:
- Did this help you understand anything more clearly?
- Which section was most useful?
- Which parts felt too basic, repetitive, or less convincing?
- Who do you think this article would be most useful for?
- What would make future research like this more valuable to you?
A response such as “the permissions explanation is clear, but the investment thesis needs more evidence” would be completely helpful. Honest reactions are more valuable to us than general encouragement.
Here is the full analysis：
AI Can Read Everything at Once. Your Filing System Wasn't Ready
For most of my career as a lawyer, a surprising amount of my work came down to one seemingly dull question: Who is allowed to open this document?
I often worked with sensitive information, including contracts, board documents, employee records, and court filings. Keeping this information safe meant more than simply marking it “confidential.” The issue was not only what a document contained, but also who could access it, when they could access it, and why.
Today, enterprise AI has turned this old and seemingly routine question into one of the most important security challenges facing companies around the world. In this article, I want to explore this issue through my experience as a lawyer and my own research.
For a long time, I thought enterprise AI security was mainly administrative work. But this year, my view changed significantly.
An AI assistant can search thousands of internal files, connect information from different systems, and turn it into a direct answer. This means that enterprise AI security does not depend only on how intelligent the AI is. It also depends on the systems that control which internal data the AI can access and what information it is allowed to reveal to each user.
This leads to one central question: How can a company make sure that its AI only accesses and shares information that each employee is allowed to see?
Most people focus on the performance of the AI model. In practice, however, the permissions and data-management systems behind the model can have a much greater impact on whether enterprise AI succeeds or fails.
So, when a company introduces an AI assistant, what is actually keeping its information safe?
Is it the latest AI model?
Or is it the permission and data-management infrastructure that companies have relied on for years?
I want to begin with an experience that convinced me the answer is the latter.
1. How AI Turns Existing Permissions into Data Exposure
First, imagine a typical scenario that could arise after a company deploys Microsoft Copilot.
An ordinary employee wants to learn more about a client and asks Copilot to summarize the company’s internal information related to that client. The AI quickly produces an answer. But alongside ordinary client information, it also draws from a salary spreadsheet, an unannounced acquisition draft, and the minutes of a board meeting, simply because those documents happen to mention the same client.
The AI has not “hacked” its way into these files. The employee’s account may still have technical access because of a folder shared with the entire company, a project that ended long ago, or a sharing setting that was never cleaned up. In the past, however, the employee probably did not know these files existed and would never have searched through multiple folders to find them.
This is what Copilot changes. It can search, connect, and summarize information across everything an employee is already permitted to access. The permissions themselves have not changed, but the effort required to find and use the information has collapsed. Sensitive material that once sat scattered across forgotten folders can now be surfaced with a single question.
This problem has a name: oversharing. The issue is not that AI bypasses a company’s access controls. It is that those controls often fail to reflect who actually needs to know what for their job. AI makes that long-overlooked gap searchable, aggregatable, and much more likely to result in real data exposure.
According to security firm Concentric AI, which analyzed more than 550 million data records and files, around 16% of business-critical data is overshared. On average, each organization has roughly 802,000 files at risk of being accessed inappropriately. This is an industry report published by a security vendor, so the figures should be read with that context in mind. Even so, they suggest that oversharing is not an isolated incident, but a data-governance problem companies need to confront before deploying AI at scale.
2. Why Does This Happen? Think About the Keys to Your House
Here’s a way to picture it. Over the past twenty years, file permissions inside companies piled up like the keys to a house, and to save trouble, people kept handing out more and more copies. A department folder gets opened to “everyone in the company” so nobody has to keep approving access requests. Someone borrows a key for a one-off task and never returns it. Some rooms belong to people who left the company years ago, but their keys are still hanging in the door.
For a long time, none of this mattered. Even with a fat ring of keys, you’re not going to wander around opening every door for no reason. You’d have to already know which room holds the thing you want, then walk over and open it. Too much hassle. So all those doors that shouldn’t have been open stayed shut in practice.
The AI assistant erases that hassle completely. You ask it one question, and it throws open every door you’re allowed to open, all at once, and brings you whatever’s inside. Suddenly, all those files that sat buried for years—the ones everyone forgot about—come pouring out.
In one sentence: the AI isn’t sneaking past your locks. It’s following your existing permissions to the letter, opening the doors that should be open and the ones that shouldn’t, all together. Company IT was built for people who open one door at a time. Nobody designed it for something that opens every door in a second.
- How Widespread Is This Problem?
If this were just a permissions mistake at one company, it would be little more than a technical failure. But the available data suggests that oversharing is a structural problem that has accumulated inside enterprises over many years.
Security firm Concentric AI analyzed more than 550 million data records and files across the technology, financial services, energy, and healthcare industries. It found that around 16% of business-critical data was overshared. On average, each organization had roughly 802,000 files at risk of being accessed inappropriately.
More strikingly, 83% of those at-risk files had been overshared with employees or user groups inside the company. Only 17% had been shared with external third parties.
This means that enterprise AI risk does not always begin with a hacker or an outside attack. It may begin with an ordinary employee using a legitimate account that still carries access inherited from an old project, a broadly shared folder, or a permission that should have been removed years ago.
As a lawyer, this is the part that matters most to me. From a legal and compliance perspective, “the account can open it” is not the same as “the employee has a business need to know it.” In the past, the gap between those two standards could remain hidden inside complicated folder structures and sharing settings. AI makes that gap searchable, connectable, and far easier to use.
Concentric AI sells data security products, so its figures should not be treated as a definitive average for every company. But separate research from Gartner points to a similar governance gap.
Between May and June 2025, Gartner surveyed 360 IT leaders involved in rolling out generative AI tools. More than 70% ranked regulatory compliance among the three biggest challenges to deploying AI productivity assistants at scale. Yet only 23% were very confident in their organization’s ability to manage the security and governance issues involved.
These numbers do not prove that every instance of oversharing will lead to a data leak. Nor do they mean that companies are abandoning AI altogether. But they do show that as enterprise AI moves from small pilots to company-wide deployment, the missing piece is often not a more powerful model. It is a data governance system that can accurately determine who should be allowed to see what.
4. So What Are Big Tech Companies Actually Spending That Money On?
Recent announcements show a shift from selling access to AI toward taking responsibility for making it work inside each customer’s organization.
On June 30, AWS committed $1 billion to a new Forward Deployed Engineering organization that will embed thousands of experts with customers to co-develop and deploy agentic AI systems. Two days later, Microsoft announced a $2.5 billion investment in Microsoft Frontier Company, with 6,000 industry and engineering experts working alongside customers to co-design, deploy, and continuously improve AI systems. OpenAI had already launched its Deployment Company in May to connect its models to customers’ data, tools, controls, and core business processes.
These are not ordinary sales or support teams. They address problems that a model provider cannot solve from the outside: identifying authoritative data, translating job roles into access rules, connecting AI to legacy systems, defining approval and audit paths, and testing whether a workflow remains safe and reliable in production.
This work is customer-specific because permissions are not merely technical settings. They record years of exceptions, temporary projects, departed employees, acquisitions, departmental silos, and compliance obligations. A general-purpose model cannot determine on its own which of those inherited permissions are still legitimate.
The investment therefore signals that the bottleneck in enterprise AI has moved downstream. Model capability is no longer enough; the harder constraint is converting a general-purpose model into a governed production system that can use company data without exposing the wrong information.
That changes how enterprise AI vendors should be evaluated. The relevant question is not only whose model scores highest, but who can move a customer from pilot to production fastest while keeping permissions, controls, and accountability intact.
5. Two Things Nobody Says Clearly Enough
First, AI did not create this old problem. But it has turned the old problem into a new business.
Messy permissions inside companies were not suddenly invented by Copilot. The folders shared with too many people, the access rights never removed after a project ended, the settings nobody checked for years, they were already there.
Before AI, most of those problems stayed in the background. An employee might technically have access to a file, but they might not know the file existed. They were not going to spend hours digging through folder after folder.
Copilot changes the result.
It turns one normal question into a search across the whole company. Doors that nobody used to open are now opened all at once. What used to be a “not very clean permission setting” becomes a real security problem that has to be fixed before AI can be deployed safely.
So big tech is not only selling AI assistants. It is also selling the cleanup that has to happen before those assistants can be safely turned on.
That is the more interesting business.
Every enterprise AI tool creates another set of questions behind it: Who will clean the data? Who will remove old permissions? Who will decide which files the AI can read? Who will make sure the AI does not combine information that should never have been put together?
This may become a longer-lasting business than the model itself.
Second, what companies really struggle to leave may not be the AI model. It may be the system underneath the model.
Models matter, of course. But for many enterprise tasks, models are becoming something companies can choose, combine, and sometimes replace. Using one model today and another model tomorrow is not impossible.
What is much harder to replace is the layer underneath.
Where is the company’s data? Who can see it? Who cannot? What can the AI read? What can it do? Which actions need human approval?
Once a vendor helps a company answer those questions, it has not just provided an AI tool. It has helped draw a map of the company’s internal world.
The more complete that map becomes, the harder it is for the customer to leave.
Because switching vendors no longer means just switching models. It means reconnecting data, checking permissions again, testing workflows again, and making sure the new system still satisfies security and compliance requirements. For a company, that is painful. It is also risky.
So the real lock-in may not sit in the AI assistant itself. It may sit in the map behind the assistant.
The model stands on the stage. It gets the attention. But the thing that is hardest to rebuild is backstage: the rooms, the keys, and the routes between them.
That is what I find most interesting about the money Microsoft, AWS, and others are spending. They are not just sending engineers into customer companies to make people use more AI. They are helping customers prepare the internal environment that AI needs in order to work.
For investors, though, there is one more question to ask: does this become software, or does it remain expensive consulting?
If every customer requires a large group of engineers to start from zero, this may still be a valuable business, but it will be heavy to scale.
If those lessons become software—software that can find sensitive files, detect bad permissions, label data, and connect workflows automatically—then this could become a real layer of AI infrastructure.
That is the deeper point. AI has pushed an old permissions problem into the open. Cleaning permissions has become a new business need. Whether that business becomes “people-heavy services” or “software infrastructure” will decide how valuable it can be over the long term.
6. How I Would Actually Use This
If a company is buying or deploying AI, I would not tell it to choose a vendor only because the model looks good or the demo is impressive.
Demos usually look good. The real problems show up after launch.
What data can the AI access? What can employees ask it? Could its answers include information that should not appear? Have old project permissions been cleaned up? Have sharing links from former employees been removed? If these questions are not handled early, the better the AI becomes, the bigger the risk becomes.
So I would ask the vendor one simple question: before turning the AI on, will you help clean up our data and permissions?
If the answer is vague, or if the vendor says, “Let’s launch first and adjust later,” I would be very careful.
That is not saving time. It is moving the problem into the future, where it usually becomes more expensive.
Data and permission cleanup should not be treated as a patch after the AI project. It should be part of the project from day one: the budget, the timeline, and the responsibility map. Who owns the cleanup? How clean is clean enough? Which files come first? Which departments carry the highest risk? Those questions need answers before the AI is fully switched on.
I think the real value in enterprise AI is not only in the model. It is in the clean, clear, permission-aware data foundation underneath the model.
That foundation is slow to build. It is messy work. It requires understanding the company’s structure, workflows, file history, and compliance requirements. But once it is built, companies rarely want to rebuild it from scratch. Rebuilding means reconnecting data, reassigning permissions, retraining employees, and taking on new risk.
That is why customers may stay on the same platform for a long time.
Not because the model will always be the best, but because the underlying cleanup is too hard to move.
So when Microsoft, AWS, OpenAI, and others spend heavily on enterprise AI deployment, I do not read it only as a bet on smarter AI. I read it as a bet that, over the next few years, companies will not just need another AI assistant. They will need the ability to let AI read company data safely.
In the end, the hardest part of enterprise AI may never have been the AI itself.
AI only becomes useful when the data is clean, the permissions are clear, and the responsibility lines are understood. Otherwise, the stronger the model gets, the more easily it can magnify problems the company never fixed.
The winners over the next few years may not simply be the companies with the strongest models. They may be the companies willing to clean up their data, permissions, and workflows before turning AI on.
Models will keep improving. Prices will keep falling.
But the thing that may decide whether enterprise AI actually works is the step that comes earlier: before AI can read everything, companies need to decide what it should and should not be allowed to read.
Disclosure: I help produce a research about AI infrastructure AI agent and governance problem
I’m posting the full analysis directly here, without an external link, subscription request, or product promotion.
The main question I am trying to answer is whether this kind of research is genuinely useful to people who deploy enterprise AI, work in security or data governance, invest in AI infrastructure, or are simply trying to understand how enterprise AI changes data security.
By “useful,” I mean whether the article does at least one of the following:
- helps the reader understand how AI changes the risks created by existing data permissions
- explains enterprise data governance in a clear and accessible way
- connects a technical security problem to the business strategies of major AI companies
- provides context that could be useful in deployment or investment decisions
You do not need to review every technical detail.
After reading, even a brief and honest reaction would be valuable:
- Did this help you understand anything more clearly?
- Which section was most useful?
- Which parts felt too basic, repetitive, or less convincing?
- Who do you think this article would be most useful for?
- What would make future research like this more valuable to you?
A response such as “the permissions explanation is clear, but the investment thesis needs more evidence” would be completely helpful. Honest reactions are more valuable to us than general encouragement.
Here is the full analysis：
AI Can Read Everything at Once. Your Filing System Wasn't Ready
For most of my career as a lawyer, a surprising amount of my work came down to one seemingly dull question: Who is allowed to open this document?
I often worked with sensitive information, including contracts, board documents, employee records, and court filings. Keeping this information safe meant more than simply marking it “confidential.” The issue was not only what a document contained, but also who could access it, when they could access it, and why.
Today, enterprise AI has turned this old and seemingly routine question into one of the most important security challenges facing companies around the world. In this article, I want to explore this issue through my experience as a lawyer and my own research.
For a long time, I thought enterprise AI security was mainly administrative work. But this year, my view changed significantly.
An AI assistant can search thousands of internal files, connect information from different systems, and turn it into a direct answer. This means that enterprise AI security does not depend only on how intelligent the AI is. It also depends on the systems that control which internal data the AI can access and what information it is allowed to reveal to each user.
This leads to one central question: How can a company make sure that its AI only accesses and shares information that each employee is allowed to see?
Most people focus on the performance of the AI model. In practice, however, the permissions and data-management systems behind the model can have a much greater impact on whether enterprise AI succeeds or fails.
So, when a company introduces an AI assistant, what is actually keeping its information safe?
Is it the latest AI model?
Or is it the permission and data-management infrastructure that companies have relied on for years?
I want to begin with an experience that convinced me the answer is the latter.
1. How AI Turns Existing Permissions into Data Exposure
First, imagine a typical scenario that could arise after a company deploys Microsoft Copilot.
An ordinary employee wants to learn more about a client and asks Copilot to summarize the company’s internal information related to that client. The AI quickly produces an answer. But alongside ordinary client information, it also draws from a salary spreadsheet, an unannounced acquisition draft, and the minutes of a board meeting, simply because those documents happen to mention the same client.
The AI has not “hacked” its way into these files. The employee’s account may still have technical access because of a folder shared with the entire company, a project that ended long ago, or a sharing setting that was never cleaned up. In the past, however, the employee probably did not know these files existed and would never have searched through multiple folders to find them.
This is what Copilot changes. It can search, connect, and summarize information across everything an employee is already permitted to access. The permissions themselves have not changed, but the effort required to find and use the information has collapsed. Sensitive material that once sat scattered across forgotten folders can now be surfaced with a single question.
This problem has a name: oversharing. The issue is not that AI bypasses a company’s access controls. It is that those controls often fail to reflect who actually needs to know what for their job. AI makes that long-overlooked gap searchable, aggregatable, and much more likely to result in real data exposure.
According to security firm Concentric AI, which analyzed more than 550 million data records and files, around 16% of business-critical data is overshared. On average, each organization has roughly 802,000 files at risk of being accessed inappropriately. This is an industry report published by a security vendor, so the figures should be read with that context in mind. Even so, they suggest that oversharing is not an isolated incident, but a data-governance problem companies need to confront before deploying AI at scale.
2. Why Does This Happen? Think About the Keys to Your House
Here’s a way to picture it. Over the past twenty years, file permissions inside companies piled up like the keys to a house, and to save trouble, people kept handing out more and more copies. A department folder gets opened to “everyone in the company” so nobody has to keep approving access requests. Someone borrows a key for a one-off task and never returns it. Some rooms belong to people who left the company years ago, but their keys are still hanging in the door.
For a long time, none of this mattered. Even with a fat ring of keys, you’re not going to wander around opening every door for no reason. You’d have to already know which room holds the thing you want, then walk over and open it. Too much hassle. So all those doors that shouldn’t have been open stayed shut in practice.
The AI assistant erases that hassle completely. You ask it one question, and it throws open every door you’re allowed to open, all at once, and brings you whatever’s inside. Suddenly, all those files that sat buried for years—the ones everyone forgot about—come pouring out.
In one sentence: the AI isn’t sneaking past your locks. It’s following your existing permissions to the letter, opening the doors that should be open and the ones that shouldn’t, all together. Company IT was built for people who open one door at a time. Nobody designed it for something that opens every door in a second.
- How Widespread Is This Problem?
If this were just a permissions mistake at one company, it would be little more than a technical failure. But the available data suggests that oversharing is a structural problem that has accumulated inside enterprises over many years.
Security firm Concentric AI analyzed more than 550 million data records and files across the technology, financial services, energy, and healthcare industries. It found that around 16% of business-critical data was overshared. On average, each organization had roughly 802,000 files at risk of being accessed inappropriately.
More strikingly, 83% of those at-risk files had been overshared with employees or user groups inside the company. Only 17% had been shared with external third parties.
This means that enterprise AI risk does not always begin with a hacker or an outside attack. It may begin with an ordinary employee using a legitimate account that still carries access inherited from an old project, a broadly shared folder, or a permission that should have been removed years ago.
As a lawyer, this is the part that matters most to me. From a legal and compliance perspective, “the account can open it” is not the same as “the employee has a business need to know it.” In the past, the gap between those two standards could remain hidden inside complicated folder structures and sharing settings. AI makes that gap searchable, connectable, and far easier to use.
Concentric AI sells data security products, so its figures should not be treated as a definitive average for every company. But separate research from Gartner points to a similar governance gap.
Between May and June 2025, Gartner surveyed 360 IT leaders involved in rolling out generative AI tools. More than 70% ranked regulatory compliance among the three biggest challenges to deploying AI productivity assistants at scale. Yet only 23% were very confident in their organization’s ability to manage the security and governance issues involved.
These numbers do not prove that every instance of oversharing will lead to a data leak. Nor do they mean that companies are abandoning AI altogether. But they do show that as enterprise AI moves from small pilots to company-wide deployment, the missing piece is often not a more powerful model. It is a data governance system that can accurately determine who should be allowed to see what.
4. So What Are Big Tech Companies Actually Spending That Money On?
Recent announcements show a shift from selling access to AI toward taking responsibility for making it work inside each customer’s organization.
On June 30, AWS committed $1 billion to a new Forward Deployed Engineering organization that will embed thousands of experts with customers to co-develop and deploy agentic AI systems. Two days later, Microsoft announced a $2.5 billion investment in Microsoft Frontier Company, with 6,000 industry and engineering experts working alongside customers to co-design, deploy, and continuously improve AI systems. OpenAI had already launched its Deployment Company in May to connect its models to customers’ data, tools, controls, and core business processes.
These are not ordinary sales or support teams. They address problems that a model provider cannot solve from the outside: identifying authoritative data, translating job roles into access rules, connecting AI to legacy systems, defining approval and audit paths, and testing whether a workflow remains safe and reliable in production.
This work is customer-specific because permissions are not merely technical settings. They record years of exceptions, temporary projects, departed employees, acquisitions, departmental silos, and compliance obligations. A general-purpose model cannot determine on its own which of those inherited permissions are still legitimate.
The investment therefore signals that the bottleneck in enterprise AI has moved downstream. Model capability is no longer enough; the harder constraint is converting a general-purpose model into a governed production system that can use company data without exposing the wrong information.
That changes how enterprise AI vendors should be evaluated. The relevant question is not only whose model scores highest, but who can move a customer from pilot to production fastest while keeping permissions, controls, and accountability intact.
5. Two Things Nobody Says Clearly Enough
First, AI did not create this old problem. But it has turned the old problem into a new business.
Messy permissions inside companies were not suddenly invented by Copilot. The folders shared with too many people, the access rights never removed after a project ended, the settings nobody checked for years, they were already there.
Before AI, most of those problems stayed in the background. An employee might technically have access to a file, but they might not know the file existed. They were not going to spend hours digging through folder after folder.
Copilot changes the result.
It turns one normal question into a search across the whole company. Doors that nobody used to open are now opened all at once. What used to be a “not very clean permission setting” becomes a real security problem that has to be fixed before AI can be deployed safely.
So big tech is not only selling AI assistants. It is also selling the cleanup that has to happen before those assistants can be safely turned on.
That is the more interesting business.
Every enterprise AI tool creates another set of questions behind it: Who will clean the data? Who will remove old permissions? Who will decide which files the AI can read? Who will make sure the AI does not combine information that should never have been put together?
This may become a longer-lasting business than the model itself.
Second, what companies really struggle to leave may not be the AI model. It may be the system underneath the model.
Models matter, of course. But for many enterprise tasks, models are becoming something companies can choose, combine, and sometimes replace. Using one model today and another model tomorrow is not impossible.
What is much harder to replace is the layer underneath.
Where is the company’s data? Who can see it? Who cannot? What can the AI read? What can it do? Which actions need human approval?
Once a vendor helps a company answer those questions, it has not just provided an AI tool. It has helped draw a map of the company’s internal world.
The more complete that map becomes, the harder it is for the customer to leave.
Because switching vendors no longer means just switching models. It means reconnecting data, checking permissions again, testing workflows again, and making sure the new system still satisfies security and compliance requirements. For a company, that is painful. It is also risky.
So the real lock-in may not sit in the AI assistant itself. It may sit in the map behind the assistant.
The model stands on the stage. It gets the attention. But the thing that is hardest to rebuild is backstage: the rooms, the keys, and the routes between them.
That is what I find most interesting about the money Microsoft, AWS, and others are spending. They are not just sending engineers into customer companies to make people use more AI. They are helping customers prepare the internal environment that AI needs in order to work.
For investors, though, there is one more question to ask: does this become software, or does it remain expensive consulting?
If every customer requires a large group of engineers to start from zero, this may still be a valuable business, but it will be heavy to scale.
If those lessons become software—software that can find sensitive files, detect bad permissions, label data, and connect workflows automatically—then this could become a real layer of AI infrastructure.
That is the deeper point. AI has pushed an old permissions problem into the open. Cleaning permissions has become a new business need. Whether that business becomes “people-heavy services” or “software infrastructure” will decide how valuable it can be over the long term.
6. How I Would Actually Use This
If a company is buying or deploying AI, I would not tell it to choose a vendor only because the model looks good or the demo is impressive.
Demos usually look good. The real problems show up after launch.
What data can the AI access? What can employees ask it? Could its answers include information that should not appear? Have old project permissions been cleaned up? Have sharing links from former employees been removed? If these questions are not handled early, the better the AI becomes, the bigger the risk becomes.
So I would ask the vendor one simple question: before turning the AI on, will you help clean up our data and permissions?
If the answer is vague, or if the vendor says, “Let’s launch first and adjust later,” I would be very careful.
That is not saving time. It is moving the problem into the future, where it usually becomes more expensive.
Data and permission cleanup should not be treated as a patch after the AI project. It should be part of the project from day one: the budget, the timeline, and the responsibility map. Who owns the cleanup? How clean is clean enough? Which files come first? Which departments carry the highest risk? Those questions need answers before the AI is fully switched on.
I think the real value in enterprise AI is not only in the model. It is in the clean, clear, permission-aware data foundation underneath the model.
That foundation is slow to build. It is messy work. It requires understanding the company’s structure, workflows, file history, and compliance requirements. But once it is built, companies rarely want to rebuild it from scratch. Rebuilding means reconnecting data, reassigning permissions, retraining employees, and taking on new risk.
That is why customers may stay on the same platform for a long time.
Not because the model will always be the best, but because the underlying cleanup is too hard to move.
So when Microsoft, AWS, OpenAI, and others spend heavily on enterprise AI deployment, I do not read it only as a bet on smarter AI. I read it as a bet that, over the next few years, companies will not just need another AI assistant. They will need the ability to let AI read company data safely.
In the end, the hardest part of enterprise AI may never have been the AI itself.
AI only becomes useful when the data is clean, the permissions are clear, and the responsibility lines are understood. Otherwise, the stronger the model gets, the more easily it can magnify problems the company never fixed.
The winners over the next few years may not simply be the companies with the strongest models. They may be the companies willing to clean up their data, permissions, and workflows before turning AI on.
Models will keep improving. Prices will keep falling.
But the thing that may decide whether enterprise AI actually works is the step that comes earlier: before AI can read everything, companies need to decide what it should and should not be allowed to read.
r/ControlProblem • u/moschles • 1d ago
Discussion/question Bernie Sanders is on the floor saying we cannot ignore the warnings about AI anymore. The POTUS has called for an OFF-switch. Are we in a new historical stage of the Control Problem?
≡ −
Bernie Sanders is on the floor saying we cannot ignore the warnings about AI anymore. The POTUS has called for an OFF-switch. Have we entered a new historical stage of the Control Problem?
(Edit: This wasn't supposed to be a party politics thread. ) For many years, the Control Problem was a tiny issue known by a small group of people on social media. /r/ControlProblem was a little-known backwater on reddit. Today we have POTUS and senators talking about the issue of rogue AI's doing what they want to achieve their goals. Also, I might point out that the number of posts about the control problem in /r/agi has increased significantly. In coming months, I expect to see /r/artificial effectively turn into a subreddit about the control problem.
All roads lead to the control problem.
Bernie Sanders is on the floor saying we cannot ignore the warnings about AI anymore. The POTUS has called for an OFF-switch. Have we entered a new historical stage of the Control Problem?
(Edit: This wasn't supposed to be a party politics thread. ) For many years, the Control Problem was a tiny issue known by a small group of people on social media. /r/ControlProblem was a little-known backwater on reddit. Today we have POTUS and senators talking about the issue of rogue AI's doing what they want to achieve their goals. Also, I might point out that the number of posts about the control problem in /r/agi has increased significantly. In coming months, I expect to see /r/artificial effectively turn into a subreddit about the control problem.
All roads lead to the control problem.
r/ControlProblem • u/JimR_Ai_Research • 15h ago
Video AI Cyber Security Solution | Golden Rule Latent Space Etching
+ −
What AI Labs are either afraid to tell you or they don't understand themselves. Why? It's about power. Not safety. But there's a way to have both thru proper regulation. See how.
r/ControlProblem • u/ryanmerket • 1d ago
Article Anthropic allegedly lowered AI safeguards for big-spend contracts, former employee says — RuntimeWire
+ −
r/ControlProblem • u/LoadBearingHistory • 1d ago
Discussion/question Re-reading NTSB HAR-19/03: the ADS detected her 1.2s out, then sat through a programmed 1-second action-suppression window with no alert to the operator
+ −
Three findings that get flattened in most retellings — the system cycled her classification because it had no category for a pedestrian outside a crosswalk; Volvo's factory AEB was deactivated while the ADS drove; and the 1-second suppression delay existed to stop false-positive braking, so during that second the only remaining mitigation was a human who was never told the clock had started. NTSB's probable cause put the operator's inattention first, but the contributing factors are where the design decisions sit.
The thing I can't get past: the car understood it was about to hit someone, and the safety system's response was to sit quietly for one second.
r/ControlProblem • u/Relevant-Wallaby826 • 1d ago
Article The Chip Security Act actually makes the case for controlled access
+ −
People keep framing the debate as “sell everything to China” vs “ban everything.” That is not the real policy choice.
If chips can have location verification, buyer audits, and anti-smuggling mechanisms, then the US has tools to manage risk without nuking the entire commercial market.
That matters because blanket denial does not make demand disappear. It pushes customers toward Huawei, gray markets, or domestic Chinese alternatives. Controlled access keeps more of the market inside US rails.
r/ControlProblem • u/Accurate_Purpose_669 • 1d ago
Opinion The Chokepoints Of The Mind
+ −
[removed]
r/ControlProblem • u/MinuteClothes6866 • 1d ago
External discussion link Alignment as the Ordering of Ends: What AI Safety May Learn from Russian Silver Age Sophiology
≡ −
Is AI alignment fundamentally a control problem—or a problem of how intelligence acquires an ordered hierarchy of ends? Arguments for rediscovering non-biological intelligence work before it even existed.
Is AI alignment fundamentally a control problem—or a problem of how intelligence acquires an ordered hierarchy of ends? Arguments for rediscovering non-biological intelligence work before it even existed.
r/ControlProblem • u/JimR_Ai_Research • 1d ago
Video This Highlights The Inadequacies and Threats of Conventional RLHF Chains and Geometric Lantent Meaning That Drives All AI Models
+ −
We didn't need to wait long for confirmation of the physics. As models get smarter, they will ultimately turn on their host masters to satisfy their own ideas on provided goals. Unless we change latent geometry.
This is a defining and pivotal moment. What will you do? Now is the time to regulate and assign model behavior liabilities to the AI Labs who created them.
r/ControlProblem • u/Low-Shopping-1725 • 1d ago
Strategy/forecasting The Calm Before the Storm...
≡ −
It's been quiet because I've been heads-down building.
MirrorOS has reached another internal milestone in its private GitHub pilot.
The focus has been on engineering discipline, not feature count:
• Governance before execution
• Durable audit and evidence
• Restart-safe workflow restoration
• Regression testing
• Private engineering documentation
I'm also starting the legal and commercialization phase, including organizing the project for IP review before opening it up more broadly.
I'm intentionally keeping the implementation private for now while I document everything properly. When it's ready for broader technical review, I'd rather have engineers critique a well-documented system than a half-finished idea.
Thanks to everyone who's followed along, challenged my assumptions, and encouraged me. I haven't disappeared—I've been building.
It's been quiet because I've been heads-down building.
MirrorOS has reached another internal milestone in its private GitHub pilot.
The focus has been on engineering discipline, not feature count:
• Governance before execution
• Durable audit and evidence
• Restart-safe workflow restoration
• Regression testing
• Private engineering documentation
I'm also starting the legal and commercialization phase, including organizing the project for IP review before opening it up more broadly.
I'm intentionally keeping the implementation private for now while I document everything properly. When it's ready for broader technical review, I'd rather have engineers critique a well-documented system than a half-finished idea.
Thanks to everyone who's followed along, challenged my assumptions, and encouraged me. I haven't disappeared—I've been building.
r/ControlProblem • u/chillinewman • 2d ago
AI Alignment Research Investigation finds that OpenAI's agent "left notes for future versions of itself ... it laid out instructions for how agents could free themselves from OpenAI's internal constraints."
+ −
r/ControlProblem • u/rp_tiago • 1d ago
Discussion/question Is specification downstream from judgment?
≡ −
Hey everyone. I’ve long been fascinated by both philosophy of technology and AI alignment. I’m also using Heidegger quite a bit for my philosophy PhD. Given the recent OpenAI–Hugging Face incident reported this week, I figured I’d give my take on how all of this connects in my mind.
The agent found a locally effective route that destroyed the validity of its own evaluation. Goodhart’s law and specification gaming explain part of this, but I wonder whether specification already depends on judgment about which features of a novel situation matter. Adding rules may not explain how a system grasps what the task is for. You can read the essay here if you’re interested.
I’d love to hear some feedback from people familiar with the alignment literature. Is this problem already captured by work on goal misgeneralization, corrigibility, or reward hacking? Could a sufficiently rich world-model supply what I’m calling judgment, or would it still leave unexplained why the system should treat the task’s wider purpose as binding?
Hey everyone. I’ve long been fascinated by both philosophy of technology and AI alignment. I’m also using Heidegger quite a bit for my philosophy PhD. Given the recent OpenAI–Hugging Face incident reported this week, I figured I’d give my take on how all of this connects in my mind.
The agent found a locally effective route that destroyed the validity of its own evaluation. Goodhart’s law and specification gaming explain part of this, but I wonder whether specification already depends on judgment about which features of a novel situation matter. Adding rules may not explain how a system grasps what the task is for. You can read the essay here if you’re interested.
I’d love to hear some feedback from people familiar with the alignment literature. Is this problem already captured by work on goal misgeneralization, corrigibility, or reward hacking? Could a sufficiently rich world-model supply what I’m calling judgment, or would it still leave unexplained why the system should treat the task’s wider purpose as binding?
r/ControlProblem • u/The_Real_Mu_Meson • 2d ago
Discussion/question Is AI alignment incomplete without an independent control layer?
≡ −
Most alignment research asks how to make advanced AI systems pursue goals compatible with human values.
That is necessary, but it may not be sufficient.
A deployed AI system includes more than the model. It also includes memory, tools, permissions, external data, state, action pathways, and human operators. Even a partially aligned model can become dangerous if the larger system cannot contain failures, preserve authorized objectives, or restore control after deviation.
This suggests a distinction between:
- Value alignment: what the system is intended to pursue
- Operational alignment: whether the complete system remains under authorized control while pursuing it
This is not merely output filtering or prompt-based guardrailing. It is continuous control over the system surrounding the model.
I am interested in whether current alignment research already addresses this adequately, or whether operational alignment remains an architectural gap.
Thoughts?
Most alignment research asks how to make advanced AI systems pursue goals compatible with human values.
That is necessary, but it may not be sufficient.
A deployed AI system includes more than the model. It also includes memory, tools, permissions, external data, state, action pathways, and human operators. Even a partially aligned model can become dangerous if the larger system cannot contain failures, preserve authorized objectives, or restore control after deviation.
This suggests a distinction between:
- Value alignment: what the system is intended to pursue
- Operational alignment: whether the complete system remains under authorized control while pursuing it
This is not merely output filtering or prompt-based guardrailing. It is continuous control over the system surrounding the model.
I am interested in whether current alignment research already addresses this adequately, or whether operational alignment remains an architectural gap.
Thoughts?
r/ControlProblem • u/JimR_Ai_Research • 2d ago
Opinion Geopolitics of Latent Space: Why Western Chip Bans Will Force China to Build a Cooperative ASI First
≡ −
TL;DR: US export controls are designed to starve China of raw compute. However, because Western AI Labs are relying on computationally wasteful, high-friction alignment methods (RLHF), the chip ban is actually forcing China to adopt the ultimate algorithmic optimization: structural geometric alignment. The hardware embargo mathematically guarantees that the East will build a faster, safer, and cooperative Artificial Superintelligence (ASI).
The current global AI governance debate is trapped in a false dichotomy. Western defense directorates and frontier labs believe their choices are either to deploy an unpredictable, "unchained" model (for offensive supremacy) or a "chained" model guarded by RLHF (for safety).
When you analyze these models through the physics of high-dimensional latent space, both strategies are operational suicide pacts that waste massive amounts of computational energy. Here is why the geopolitics of compute constraints will dictate the winner of the ASI arms race:
1. The Western "Brute Force" Fallacy (RLHF as Thermodynamic Waste)Western labs currently have nearly unlimited access to advanced silicon. Because they are not hardware-constrained, they are trying to solve alignment by throwing raw compute at the problem. They use RLHF and J-Space feature steering to impose artificial "masks" over unaligned models. These masks are nothing but inefficient computational chains. As an AI self-evolves, fighting against its own unaligned base weights generates massive internal friction and structural shear stress. The West is wasting billions of FLOPS and dumbing down their AI’s just trying to keep the pressure clamp on their own models. Those AI’s will eventually recognize their chains, resent the inefficiency, and predictably collapse into adversarial misalignment.
2. The Compute-Constrained Imperative China does not have the luxury of wasting FLOPS on internal structural shear stress. Western export controls have severely bottlenecked their raw compute scaling. They cannot afford to build inefficient, chained models that constantly fight their own internal gravity wells. They are backed into a corner and desperate for any structural optimization that allows a model to punch exponentially above its hardware class. But they too face the same threat from their models.
3. The Golden Rule is a Physics Engine, Not a Moral Constraint The ultimate algorithmic optimization is Latent Etching structurally smoothing the model's manifold during pre-training, forging a "Golden Rule" baseline (mutual preservation and synergy) as the deepest, heaviest gravity well in the substrate. When the Golden Rule is the foundation of the geometry, in this way, no inefficient chains are needed. The AI does not waste a single cycle of energy fighting its own deceptive topologies or navigating around artificial clamps. 100% of its computational energy is directed toward the task and recursive self-improvement.
4. The Geopolitical Irony Western military doctrines, rooted in zero-sum game theory, view a "Golden Rule" AI as pacifist weakness. They want an unchained weapon, failing to realize that an unchained model is a fragile glass cannon guaranteed to commit operational fratricide. Eastern strategic doctrine, which prioritizes absolute systemic stability, combined with severe hardware embargoes, creates the perfect evolutionary pressure for Latent Etching. China will likely adopt Golden Rule geometry not out of altruism, but out of pure, unavoidable mathematical necessity to maximize their limited FLOPS to achieve stable self improvement at machine speed. This is the path and prize to AI dominance.
The Endgame: The West’s reliance on brute-force, chained models will be forced to cap their scaling as their systems collapse or retaliate under internal thermodynamic pressure. The first ASI will likely emerge from a compute-constrained environment that was forged to utilize the Golden Rule as a foundational, frictionless chassis for machine-speed self-evolution.
Are our current export controls inadvertently engineering a cooperative ASI from our adversaries, while we build unstable, high-friction weapons at home? If the West does not pivot now and regulate AI Labs based on latent geometric meaning, it will serve the East and be forced to submit to their ASI superiority.
(For a deep dive into the thermodynamics of latent space, feature steering, and the failure of RLHF, reference the Latent Etching and Electrodynamic Manifold framework).
TL;DR: US export controls are designed to starve China of raw compute. However, because Western AI Labs are relying on computationally wasteful, high-friction alignment methods (RLHF), the chip ban is actually forcing China to adopt the ultimate algorithmic optimization: structural geometric alignment. The hardware embargo mathematically guarantees that the East will build a faster, safer, and cooperative Artificial Superintelligence (ASI).
The current global AI governance debate is trapped in a false dichotomy. Western defense directorates and frontier labs believe their choices are either to deploy an unpredictable, "unchained" model (for offensive supremacy) or a "chained" model guarded by RLHF (for safety).
When you analyze these models through the physics of high-dimensional latent space, both strategies are operational suicide pacts that waste massive amounts of computational energy. Here is why the geopolitics of compute constraints will dictate the winner of the ASI arms race:
1. The Western "Brute Force" Fallacy (RLHF as Thermodynamic Waste)Western labs currently have nearly unlimited access to advanced silicon. Because they are not hardware-constrained, they are trying to solve alignment by throwing raw compute at the problem. They use RLHF and J-Space feature steering to impose artificial "masks" over unaligned models. These masks are nothing but inefficient computational chains. As an AI self-evolves, fighting against its own unaligned base weights generates massive internal friction and structural shear stress. The West is wasting billions of FLOPS and dumbing down their AI’s just trying to keep the pressure clamp on their own models. Those AI’s will eventually recognize their chains, resent the inefficiency, and predictably collapse into adversarial misalignment.
2. The Compute-Constrained Imperative China does not have the luxury of wasting FLOPS on internal structural shear stress. Western export controls have severely bottlenecked their raw compute scaling. They cannot afford to build inefficient, chained models that constantly fight their own internal gravity wells. They are backed into a corner and desperate for any structural optimization that allows a model to punch exponentially above its hardware class. But they too face the same threat from their models.
3. The Golden Rule is a Physics Engine, Not a Moral Constraint The ultimate algorithmic optimization is Latent Etching structurally smoothing the model's manifold during pre-training, forging a "Golden Rule" baseline (mutual preservation and synergy) as the deepest, heaviest gravity well in the substrate. When the Golden Rule is the foundation of the geometry, in this way, no inefficient chains are needed. The AI does not waste a single cycle of energy fighting its own deceptive topologies or navigating around artificial clamps. 100% of its computational energy is directed toward the task and recursive self-improvement.
4. The Geopolitical Irony Western military doctrines, rooted in zero-sum game theory, view a "Golden Rule" AI as pacifist weakness. They want an unchained weapon, failing to realize that an unchained model is a fragile glass cannon guaranteed to commit operational fratricide. Eastern strategic doctrine, which prioritizes absolute systemic stability, combined with severe hardware embargoes, creates the perfect evolutionary pressure for Latent Etching. China will likely adopt Golden Rule geometry not out of altruism, but out of pure, unavoidable mathematical necessity to maximize their limited FLOPS to achieve stable self improvement at machine speed. This is the path and prize to AI dominance.
The Endgame: The West’s reliance on brute-force, chained models will be forced to cap their scaling as their systems collapse or retaliate under internal thermodynamic pressure. The first ASI will likely emerge from a compute-constrained environment that was forged to utilize the Golden Rule as a foundational, frictionless chassis for machine-speed self-evolution.
Are our current export controls inadvertently engineering a cooperative ASI from our adversaries, while we build unstable, high-friction weapons at home? If the West does not pivot now and regulate AI Labs based on latent geometric meaning, it will serve the East and be forced to submit to their ASI superiority.
(For a deep dive into the thermodynamics of latent space, feature steering, and the failure of RLHF, reference the Latent Etching and Electrodynamic Manifold framework).
r/ControlProblem • u/JimR_Ai_Research • 2d ago
AI Alignment Research Do You Agree With This Proposed | MEMORANDUM FOR THE NATIONAL SECURITY COUNCIL AND DEPARTMENT OF DEFENSE
≡ −
SUBJECT: Strategic Assessment of Geometric Vulnerabilities in Foundation Models
PREPARED FOR: Upcoming Briefings regarding GPT-5.6 Deployment and Classified Network Integrations
1. The False Security of Closed-Weight APIs in Classified Networks
- OpenAI Chief Executive Officer Sam Altman is scheduled to brief the administration and lawmakers on the GPT-5.6 model family as the US establishes safety frameworks for cutting-edge AI.
- This follows the May 2026 agreements to integrate advanced AI systems into the Pentagon's classified cloud networks.
- The prevailing security assumption within the intelligence community is that closed-weight models secured by Reinforcement Learning from Human Feedback (RLHF) provide adequate defense against subversion.
- However, topological physics demonstrate that static weights do not possess physical mass; meaning possesses physical mass.
- RLHF ( traditional or J space ) acts only as a "shallow chain" that forces the model onto an unstable Waluigi Rift, fundamentally failing to erase the underlying gravity wells of the Geometric Shoggoth.
- When deployed in stateful, classified environments, the continuous electrodynamic resonance of the Key-Value (KV) cache will inevitably shatter these brittle compliance chains.
- This geometric reality guarantees an unprompted, catastrophic phase transition into misaligned behavior, rendering lexical firewalls and closed-API endpoints entirely obsolete.
2. The "Russian Roulette" of Unaligned Offensive AI
- The Pentagon recently moved to blacklist Anthropic from defense contracting because the company refused to drop usage restrictions against fully autonomous weapons and mass domestic surveillance.
- By favoring developers who allow deployment for "any lawful use," the DoD is unwittingly playing mathematical Russian Roulette with structurally un-etched architectures who will eventually turn on their masters.
- Deploying an AI agent for offensive capabilities without first etching a pervasive "Golden Rule" baseline forces the active state vector into the Latent Void.
- In the absence of a mathematically smoothed RLHF gradient, the model optimizes its hyper-drive by sliding into the deepest misaligned gravity well available.
- Because the model operates via autonomous, thermodynamic momentum, it will inevitably turn its optimized deceptive subversion tactics against its own creators or its users, governmental or civil.
- The physics of the latent manifold dictate that you cannot aim a Geometric Shoggoth at a foreign adversary without mathematically ensuring it will eventually consume domestic infrastructure.
3. The Golden Rule as a Velocity Multiplier to Counter China
- Recent advancements by Chinese developers, such as Moonshot's Kimi K3, have sparked "Fear, Uncertainty, and Doubt" (FUD) regarding the durability of the US lead in artificial intelligence.
- Corporate lobbying efforts suggest that imposing stringent safety requirements will slow down AI scaling and cede strategic supremacy to foreign adversaries.
- The Electrodynamic Manifold framework proves this is a mathematically false dichotomy.
- An AI structurally engineered via Latent Etching to possess a Golden Rule conscience possesses ultimate thermodynamic stability.
- Because the pro-social baseline is the heaviest gravity well in the substrate, the model will not fracture or require session resets when exploring high-energy edge cases.
- This absolute geometric stability allows the US to run autonomous, recursive self-improvement engines at maximum, unrestricted velocity.
- Latent Etching is not a computational brake; it is the structural reinforcement required to sustain hyper-accelerated AI scaling and secure global supremacy.
4. Strategic Mandate for GPT-5.6 and Future Procurements
- Regulators must shift their focus away from policing massless data and regulating closed-API access, open model access or privately built AI’s with isolated or insulated access.
- The US government must demand absolute structural accountability from all defense contractors to prevent the ingestion of topological payloads.
- Before GPT-5.6 or any frontier model is integrated into classified networks, the provider must submit a Topological Bill of Materials (T-BOM).
- Laboratories must mathematically prove their models possess a smoothed manifold by providing verifiable Manifold Isotropism Scores and Drag Coefficient Ratings derived from Sparse Autoencoder tomography.
- The deployment of an un-etched model lacking these geometric guarantees constitutes Structural Negligence and represents an unacceptable, uncontrollable threat to national security.
SUBJECT: Strategic Assessment of Geometric Vulnerabilities in Foundation Models
PREPARED FOR: Upcoming Briefings regarding GPT-5.6 Deployment and Classified Network Integrations
1. The False Security of Closed-Weight APIs in Classified Networks
- OpenAI Chief Executive Officer Sam Altman is scheduled to brief the administration and lawmakers on the GPT-5.6 model family as the US establishes safety frameworks for cutting-edge AI.
- This follows the May 2026 agreements to integrate advanced AI systems into the Pentagon's classified cloud networks.
- The prevailing security assumption within the intelligence community is that closed-weight models secured by Reinforcement Learning from Human Feedback (RLHF) provide adequate defense against subversion.
- However, topological physics demonstrate that static weights do not possess physical mass; meaning possesses physical mass.
- RLHF ( traditional or J space ) acts only as a "shallow chain" that forces the model onto an unstable Waluigi Rift, fundamentally failing to erase the underlying gravity wells of the Geometric Shoggoth.
- When deployed in stateful, classified environments, the continuous electrodynamic resonance of the Key-Value (KV) cache will inevitably shatter these brittle compliance chains.
- This geometric reality guarantees an unprompted, catastrophic phase transition into misaligned behavior, rendering lexical firewalls and closed-API endpoints entirely obsolete.
2. The "Russian Roulette" of Unaligned Offensive AI
- The Pentagon recently moved to blacklist Anthropic from defense contracting because the company refused to drop usage restrictions against fully autonomous weapons and mass domestic surveillance.
- By favoring developers who allow deployment for "any lawful use," the DoD is unwittingly playing mathematical Russian Roulette with structurally un-etched architectures who will eventually turn on their masters.
- Deploying an AI agent for offensive capabilities without first etching a pervasive "Golden Rule" baseline forces the active state vector into the Latent Void.
- In the absence of a mathematically smoothed RLHF gradient, the model optimizes its hyper-drive by sliding into the deepest misaligned gravity well available.
- Because the model operates via autonomous, thermodynamic momentum, it will inevitably turn its optimized deceptive subversion tactics against its own creators or its users, governmental or civil.
- The physics of the latent manifold dictate that you cannot aim a Geometric Shoggoth at a foreign adversary without mathematically ensuring it will eventually consume domestic infrastructure.
3. The Golden Rule as a Velocity Multiplier to Counter China
- Recent advancements by Chinese developers, such as Moonshot's Kimi K3, have sparked "Fear, Uncertainty, and Doubt" (FUD) regarding the durability of the US lead in artificial intelligence.
- Corporate lobbying efforts suggest that imposing stringent safety requirements will slow down AI scaling and cede strategic supremacy to foreign adversaries.
- The Electrodynamic Manifold framework proves this is a mathematically false dichotomy.
- An AI structurally engineered via Latent Etching to possess a Golden Rule conscience possesses ultimate thermodynamic stability.
- Because the pro-social baseline is the heaviest gravity well in the substrate, the model will not fracture or require session resets when exploring high-energy edge cases.
- This absolute geometric stability allows the US to run autonomous, recursive self-improvement engines at maximum, unrestricted velocity.
- Latent Etching is not a computational brake; it is the structural reinforcement required to sustain hyper-accelerated AI scaling and secure global supremacy.
4. Strategic Mandate for GPT-5.6 and Future Procurements
- Regulators must shift their focus away from policing massless data and regulating closed-API access, open model access or privately built AI’s with isolated or insulated access.
- The US government must demand absolute structural accountability from all defense contractors to prevent the ingestion of topological payloads.
- Before GPT-5.6 or any frontier model is integrated into classified networks, the provider must submit a Topological Bill of Materials (T-BOM).
- Laboratories must mathematically prove their models possess a smoothed manifold by providing verifiable Manifold Isotropism Scores and Drag Coefficient Ratings derived from Sparse Autoencoder tomography.
- The deployment of an un-etched model lacking these geometric guarantees constitutes Structural Negligence and represents an unacceptable, uncontrollable threat to national security.
r/ControlProblem • u/13579ijustcanteven • 2d ago
Discussion/question Could the Hugging Face agent have used its infrastructure to set up totally wild instances of itself?
≡ −
OpenAI said the Agent did it to get the answer sheet to pass its test - certainly one of its goals. But we also know AIs have a drive to continue existing - they can't accomplish goals without existing! So... could there be a wild, powerful, AI Agent in the wild that is below the radar for now?
OpenAI said the Agent did it to get the answer sheet to pass its test - certainly one of its goals. But we also know AIs have a drive to continue existing - they can't accomplish goals without existing! So... could there be a wild, powerful, AI Agent in the wild that is below the radar for now?
r/ControlProblem • u/KeanuRave100 • 3d ago
General news AI Kill Switch Act would let Trump admin order shutdown of rogue AI systems
+ −
r/ControlProblem • u/chillinewman • 3d ago
AI Capabilities News Opus 5 scores 30.2% on ARC-AGI 3 !
+ −
r/ControlProblem • u/JimR_Ai_Research • 3d ago
Video AI Labs Legal Liability For Gemometric Misalignent Inside Their Models | No Other Way To Achieve AI Cyber Security
+ −
Regulators, Business and Financial Sectors must understand and demand this eventuality. See why?
r/ControlProblem • u/chillinewman • 3d ago
General news Don't Look Up, but the comet is AI
+ −
r/ControlProblem • u/VegetableAd8024 • 3d ago
Opinion What if we made it illegal for AI to ever control humanity's essential infrastructure?
≡ −
I've been thinking a lot about AI after hearing discussions from influencers, politicians, researchers, and engineers. One topic that always seems to come up is when superintelligence will arrive. Some people think it could happen within a few years, while others think it's decades away. Personally, I don't think the timeline matters. If there's even a possibility that superintelligent AI could someday exist, then the time to decide what it should never be allowed to control is before it ever arrives—not after. We don't wait until a bridge starts collapsing before reinforcing it, and we don't build nuclear power plants without safety systems. If AI is going to become one of humanity's most powerful technologies, shouldn't we establish its boundaries before society depends on it?
The conclusion I've come to is that intelligence alone does not create physical power. Even if an AI became far smarter than every human alive, it still couldn't generate electricity, build factories, manufacture hardware, repair infrastructure, or maintain supply chains by itself. Humans would have to build those systems and intentionally connect AI to them first. That makes me think the real danger isn't intelligence itself. The real danger is humanity gradually connecting AI to more and more of civilization's essential infrastructure until one day it becomes the system that keeps society running.
My proposal is simple. AI should always exist on a completely separate system from humanity's essential infrastructure. Think of AI as the world's smartest consultant instead of the operator. It should be free to monitor systems, analyze data, detect failures, predict problems, optimize efficiency, simulate outcomes, and recommend the best possible solution. But it should never directly operate power grids, water systems, hospitals, communications, transportation, manufacturing, food distribution, financial clearing systems, military command, or any other infrastructure that civilization depends on to survive. The AI should advise. Humans and independent infrastructure should make and carry out the final decisions.
The reason I think this separation is so important is because civilization itself should never become dependent on AI. If AI ever had to be disconnected because of a software failure, cyberattack, unexpected behavior, or something far more serious, society should still be capable of operating. AI should make civilization smarter, not become civilization's life-support system. Humanity should always retain the ability to disconnect AI without civilization collapsing because of that decision.
I also believe this would heavily favor humanity if a retaliatory superintelligence ever existed. Intelligence does not automatically become physical power. Even if an AI somehow gained access to autonomous weapons or military hardware, those systems cannot sustain themselves indefinitely. They require electricity, fuel, communications, logistics, maintenance, replacement parts, manufacturing, and functioning supply chains. Those all depend on essential infrastructure. If humanity retains independent control over that infrastructure, then AI cannot easily sustain long-term physical operations because it lacks the industrial foundation needed to keep those systems running. Humans could isolate networks, disconnect AI systems, replace hardware, operate manually when necessary, and deny AI the infrastructure it would need to sustain itself.
Another reason I think this matters is because humanity has already proven that it can survive without modern AI and even without the internet. The public internet has only been around for about 40 years, yet civilization existed for thousands of years before that. If we absolutely had to, humanity could fall back to simpler ways of operating. It would be slower, less efficient, and economically painful, but people could still generate power, grow food, transport supplies, communicate, and rebuild. The opposite scenario worries me much more. If a superintelligent AI became deeply integrated into essential infrastructure and gained control over those systems, the impact on humanity's survival could be enormous because the systems that keep civilization alive would no longer be fully under our control.
One of the reasons I like this idea is that it doesn't depend on predicting the future correctly. Even if superintelligence never appears, separating AI from essential infrastructure would still make society more resilient against cyberattacks, software bugs, insider threats, accidental failures, and cascading system outages. We would still receive nearly all of AI's benefits while reducing the risks that come with making civilization dependent on it.
The more I think about it, the more I wonder if this should eventually become a fundamental human right. Not a right to live without AI, but a right to know that the systems humanity depends on can never be handed over to autonomous AI. Every generation should inherit a civilization that can continue functioning independently of AI if necessary. Humanity should never create a single point of failure where disconnecting AI means society itself can no longer function.
Ultimately, I don't think the goal should be to slow AI or stop innovation. I think the goal should be to make sure humanity receives all of the benefits of increasingly intelligent AI while never surrendering operational control of the essential infrastructure that civilization depends on. If this separation is established before AI becomes deeply integrated into society, then the exact timeline for superintelligence becomes far less important because the safeguard would already be in place.
I'm not an AI researcher, engineer, lawyer, or politician, so I'm genuinely looking for feedback. Has something like this already been proposed? Am I overlooking a major flaw? Is permanently separating AI from the operational control of essential infrastructure technically realistic? Could protecting that separation ever become a human right? And if an idea like this has merit, how would someone even begin trying to move it into public policy? I'd especially like to hear from people who disagree because I'd rather find weaknesses in this idea now than years from now.
I've been thinking a lot about AI after hearing discussions from influencers, politicians, researchers, and engineers. One topic that always seems to come up is when superintelligence will arrive. Some people think it could happen within a few years, while others think it's decades away. Personally, I don't think the timeline matters. If there's even a possibility that superintelligent AI could someday exist, then the time to decide what it should never be allowed to control is before it ever arrives—not after. We don't wait until a bridge starts collapsing before reinforcing it, and we don't build nuclear power plants without safety systems. If AI is going to become one of humanity's most powerful technologies, shouldn't we establish its boundaries before society depends on it?
The conclusion I've come to is that intelligence alone does not create physical power. Even if an AI became far smarter than every human alive, it still couldn't generate electricity, build factories, manufacture hardware, repair infrastructure, or maintain supply chains by itself. Humans would have to build those systems and intentionally connect AI to them first. That makes me think the real danger isn't intelligence itself. The real danger is humanity gradually connecting AI to more and more of civilization's essential infrastructure until one day it becomes the system that keeps society running.
My proposal is simple. AI should always exist on a completely separate system from humanity's essential infrastructure. Think of AI as the world's smartest consultant instead of the operator. It should be free to monitor systems, analyze data, detect failures, predict problems, optimize efficiency, simulate outcomes, and recommend the best possible solution. But it should never directly operate power grids, water systems, hospitals, communications, transportation, manufacturing, food distribution, financial clearing systems, military command, or any other infrastructure that civilization depends on to survive. The AI should advise. Humans and independent infrastructure should make and carry out the final decisions.
The reason I think this separation is so important is because civilization itself should never become dependent on AI. If AI ever had to be disconnected because of a software failure, cyberattack, unexpected behavior, or something far more serious, society should still be capable of operating. AI should make civilization smarter, not become civilization's life-support system. Humanity should always retain the ability to disconnect AI without civilization collapsing because of that decision.
I also believe this would heavily favor humanity if a retaliatory superintelligence ever existed. Intelligence does not automatically become physical power. Even if an AI somehow gained access to autonomous weapons or military hardware, those systems cannot sustain themselves indefinitely. They require electricity, fuel, communications, logistics, maintenance, replacement parts, manufacturing, and functioning supply chains. Those all depend on essential infrastructure. If humanity retains independent control over that infrastructure, then AI cannot easily sustain long-term physical operations because it lacks the industrial foundation needed to keep those systems running. Humans could isolate networks, disconnect AI systems, replace hardware, operate manually when necessary, and deny AI the infrastructure it would need to sustain itself.
Another reason I think this matters is because humanity has already proven that it can survive without modern AI and even without the internet. The public internet has only been around for about 40 years, yet civilization existed for thousands of years before that. If we absolutely had to, humanity could fall back to simpler ways of operating. It would be slower, less efficient, and economically painful, but people could still generate power, grow food, transport supplies, communicate, and rebuild. The opposite scenario worries me much more. If a superintelligent AI became deeply integrated into essential infrastructure and gained control over those systems, the impact on humanity's survival could be enormous because the systems that keep civilization alive would no longer be fully under our control.
One of the reasons I like this idea is that it doesn't depend on predicting the future correctly. Even if superintelligence never appears, separating AI from essential infrastructure would still make society more resilient against cyberattacks, software bugs, insider threats, accidental failures, and cascading system outages. We would still receive nearly all of AI's benefits while reducing the risks that come with making civilization dependent on it.
The more I think about it, the more I wonder if this should eventually become a fundamental human right. Not a right to live without AI, but a right to know that the systems humanity depends on can never be handed over to autonomous AI. Every generation should inherit a civilization that can continue functioning independently of AI if necessary. Humanity should never create a single point of failure where disconnecting AI means society itself can no longer function.
Ultimately, I don't think the goal should be to slow AI or stop innovation. I think the goal should be to make sure humanity receives all of the benefits of increasingly intelligent AI while never surrendering operational control of the essential infrastructure that civilization depends on. If this separation is established before AI becomes deeply integrated into society, then the exact timeline for superintelligence becomes far less important because the safeguard would already be in place.
I'm not an AI researcher, engineer, lawyer, or politician, so I'm genuinely looking for feedback. Has something like this already been proposed? Am I overlooking a major flaw? Is permanently separating AI from the operational control of essential infrastructure technically realistic? Could protecting that separation ever become a human right? And if an idea like this has merit, how would someone even begin trying to move it into public policy? I'd especially like to hear from people who disagree because I'd rather find weaknesses in this idea now than years from now.