I keep feeling like the AGI conversation is somewhat… off.
Not because the technology isn’t real. It very much is. I use it daily and rank among the top 0.01% of ChatGPT users and am a consistent user of Gemini, Claude, Grok and other AI powered tools for writing, synthesis, code generation, ideation, and sure, blogpost like this as well.
And it’s certainly not because the stakes aren’t real. They are real and without a doubt massive for human kind.
I say the AGI conversation feels “off” because we keep having this meta-discussion about AGI without fully admitting that we’re having a meta-discussion.
We talk about “AGI” like it’s one thing. We argue in circles. Everyone leaves feeling like everyone else is either coping or hype-posting.
And I think the reason is pretty simple.
We’ve turned AGI into a mirror.
The Mirror
The way we talk about “how close are we to AGI?” has quietly become shorthand for: how well can this thing do my job?
Within the niche I occupy as the CEO of an AI company and living in San Francisco, it’s about how well can it do laptop-class knowledge work.
Can it click the buttons I click? Type into the boxes I type into? Generate the reports I generate?
Move the tickets along. Write the emails. Summarize the docs.
Produce the decks. Do the workflows that make the modern world go round.
And look, I get it. That stuff matters. It’s economically massive. It’s going to reshape companies, whole industries, probably entire career paths.
But I also think this framing is an ego trap for us knowledge work humans.
Because if your definition of “general intelligence” is “can it replicate the day-to-day work that I do,” then what you’re really saying is: the peak of intelligence is… managing my inbox.
And, to me, as “flattering” as that is, it’s just not necessarily the right frame.
It feels like a very misguided definition.
Imitation vs Expansion
Here’s the split that makes everything clearer:
AGI as imitation vs AGI as expansion.
Imitation is: can it look like us? Sound like us? Do our tasks? Slot into our workflows? Replace the roles we recognize?
Expansion is: does it push the frontier of what is possible for civilization and, to go beyond that, does it push the frontier of what is in fact imaginable at all?
Now here’s where I have to be careful. Because my instinct is to dismiss imitation and elevate expansion. But that’s too clean.
The truth is they’re connected. Causally connected.
If AI can do knowledge work at scale, it can do scientific research at scale. It can do coordination at scale. It can synthesize fragmented information, navigate regulatory complexity, allocate resources more intelligently.
A lot of the bottlenecks to curing diseases and building better infrastructure aren’t physics problems. They’re coordination problems. Information synthesis problems. Bureaucracy problems.
Those are, indeed, knowledge work problems.
So imitation isn’t opposed to expansion. Imitation is the pathway to expansion. The problem isn’t that we’re focused on knowledge work automation. The problem is that we’re treating it as the destination instead of the mechanism.
We’re measuring arrival when we should be measuring trajectory.
A Quick Reality Check: The World Is Still Unfinished
This is the part that makes the whole thing feel strange to me.
We have not cured most of the worst diseases. People still die from cancer and heart disease. People still die from malaria. Actually, if we want to really get down to it, there are 20,000 to 30,000 deaths because of the flu every year in the United States.
There is so much infrastructure that still needs to be built around the world. Clean water. Reliable power. Housing. Transportation. Healthcare access. Education. Logistics. Basic systems that make life work.
When I hear us talk about “AGI is coming” and the whole yardstick is “can it do a quarterly report” or “can it handle customer support” or “can it write code,” I think: okay, that’s real. But also, that’s not the whole picture.
We are confusing a revolution in symbols with mastery of the physical world.
And by the way, I’m not minimizing the symbols part. A lot of the world runs on symbols. Money is symbols. Law is symbols. Coordination is symbols. Bureaucracy is symbols. Entire institutions are basically language and trust and process.
Symbolic mastery unlocks physical mastery. I, wholeheartedly, believe that.
But I also think it’s a category error to equate “it can do office work” with “it is generally intelligent.”
To get down to it, our ability and willingness to define AGI by the proximity to which it can imitate our abilities at “office work” is an indictment on just how seemingly meaningless our day-to-day is and how capped we have been in our pursuit of greatness and limitless imagination.
We've defined the ceiling of machine intelligence as "can it do my job" — which says everything about how low we've set the ceiling for ourselves.
General intelligence should be measured by its ability to expand what humanity can build—not just automate knowledge work theatrics.
“AGI” Is a Moving Target
Part of why this conversation gets so messy is that “AGI” means totally different things to different people.
Some mean human-level breadth across domains.
Some mean economic replaceability—can it do most jobs?
Some mean scientific and engineering acceleration.
Some mean autonomous agency in the real world.
Some mean reliability, robustness, self-correction—the ability to not fall apart under weird conditions and edge cases.
Because we don’t agree on the definition, we end up arguing about vibes.
Job-replication became the default yardstick not because it’s correct but because it’s legible.
It maps cleanly onto fear. Onto status. Onto identity. Onto the thing everyone understands: labor.
But “economic disruption” and “general intelligence” are not the same thing.
Capability Isn’t the Only Bottleneck. Trust Is.
Even if you stay inside knowledge work, it’s not as simple as “can it do the task.”
The real world doesn’t run on “pretty good outputs.”
The real world runs on accountability.
Who is liable when it gets it wrong? What’s the error tolerance? How expensive is integration? What are the security and misuse risks? Can you trust it with access? Can you trust it with authority? Can you trust it under pressure and ambiguity and weird corner cases?
Society deploys capability only when it is governable.
A model being impressive is not the same as a model being trustworthy.
And here’s the thing that bothers me about my own “expansion” framing: this trust problem gets harder as the stakes go up. If we can’t trust AI to handle routine knowledge work reliably, why would we trust it to redesign supply chains, accelerate drug discovery, or take us to space?
The mundane deployments are the trust-building phase. We’re not just automating work—we’re building the institutional muscle to govern increasingly powerful systems.
Maybe that’s why the knowledge work yardstick persists. Not because we’re narcissists. But because it’s actually the right proving ground.
The Real Crux: The AGI Debate Is Secretly About Meaning
Okay. Here’s the part I actually care about.
There’s the definitional thing. The economic thing. The atoms-vs-symbols thing.
But underneath all of it, I think the AGI debate is less about intelligence and more about identity.
We’re using “will it take my job?” as a socially acceptable proxy for a deeper fear.
And that fear is: if work stops being the organizing principle of life, what replaces it?
Because work is NOT just income.
Work is structure. Work is identity. Work is social belonging. Work is community. Work is status. Work is the default script.
Even if you hate your job, it gives your life a shape. It gives you a reason to wake up. It gives you a story you can tell yourself about who you are.
So when we talk about “AI taking jobs,” what we’re really talking about is: what happens when the story breaks.
And I think that’s the question we’re not unpacking nearly enough.
What would we collectively do if all of our jobs were actually done for us?
Not as a fantasy question. As a real question.
Because not everybody is an explorer. Not everybody wants to foray into the unknown and invent new things and build new worlds.
Most people just want to live their lives.
So what happens when the default path is gone? When the thing society has organized around for the last few hundred years becomes optional, or unstable, or strange?
What do people pursue? What do people do with their time? What gives people dignity? What gives people meaning?
Meaning Doesn’t Disappear. It Just Moves.
Sometimes people talk about a post-work world like it’s automatically utopia.
I’m not saying it can’t be better. It obviously can.
But humans are still humans.
Even in abundance, people are incredibly adept to inventing new ways to compete for status.
People want belonging. People want prestige. People want to feel useful. People want to feel chosen. People want to feel like they matter.
Meaning and status do not disappear when work disappears.
They just move.
They move into other arenas. Community. Creation. Sport. Art. Spirituality. Exploration. Family. Local roles. Digital worlds. Who knows.
But the transition is not trivial. Because right now, work is the main sorting mechanism. The main identity mechanism. The main status mechanism.
If you pull that out from under society, you don’t just get “more leisure.”
You get a reorg of a fundamentally human operating system.
A Better Test for Progress Toward AGI
Here’s a simple way to make this less vibes-based.
If your definition of AGI is job automation, progress should track office benchmarks. How well do models replace tasks inside companies?
If your definition is expansion, the tell is different.
The tell is sustained autonomous loops that improve real-world systems.
Science. Engineering. Infrastructure. Healthcare. Energy. Supply chains. Systems that have atoms and consequences.
And here’s my actual position: both yardsticks matter, but they measure different things.
The job automation yardstick measures capability and trust readiness. It’s the proving ground.
The expansion yardstick measures civilizational impact. It’s the destination.
We need both. We just shouldn’t confuse them.
Closing: We Need a Better Conversation Than “Can It Do My Job”
I’m not saying job disruption isn’t real. It is. It’s already happening.
I’m saying our conversation about AGI is too self-centered.
We’re treating AGI as a labor market event instead of a civilization-building event.
And by doing that, we’re skipping the deeper, more human question.
If AGI exists, it won’t arrive as a better employee. It will arrive as a force that changes what is possible and, even, what is imaginable.
And yes, the first phase looks like knowledge work automation. That’s not vanity. That’s a key mechanism. That’s how symbolic mastery becomes physical mastery.
But we shouldn’t mistake the mechanism for the meaning.
Because the deeper problem isn’t whether AI can do your job.
The deeper problem is what humans do when knowledge work based productivity is no longer THE point.
What happens when work stops being the default answer to “what do you do with your life?”
That’s the conversation we should be having.
Because that’s the one that’s going to hit all of us—whether we call it AGI or not.
If we cap our definition of General Intelligence at 'managing an inbox' or 'generating reports,' we are drastically underselling the horizon. Real.
It gets me thinking about the maturity of AI perception globally versus locally. In hubs like SF or even NYC, the narrative is dominated by anxiety because the 'replacement' threat to knowledge work is visceral. But here in Nepal, where data maturity is still nascent (often data is literally stored in a daraaz with a lock), the friction is different.
Do you think this gives developing economies a unique advantage? Because we aren't as entrenched in the legacy systems of 'office work,' we might be able to leapfrog the 'imitation' phase and frame AGI immediately as a tool for 'expansion.' As someone working back in Nepal after traveling around, I see a unique, complex problem to solve here, and we are literally at the forefront of it.
Your vision of 'Silicon Peaks' resonates deeply. Would love to connect and chat more about this whenever you're in town. Hit me at nobelrimal@nyu.edu.