Thank you! In writing it, it did raise for me the question of 'what sorts of things _are_ uniquely human," & so I'm curious if anything leapt to mind while reading
I guess this all depends on how judgement is defined / definable, and how it is measured / measurable.
For any definition and measurement that is essentially a function of some data (aka context), subject to some objective measurenent (aka minimized loss), there will eventually be a machine that can be as good as humans, given the same data.
The uniquely human trait, however, will remain that of accountability, at least for a long time to come.
As long as judgement is linked to accountability, AI may be a helpful tool, yet never take the final decision.
That is especially true for the LLM variant of AI, where errors are eseentially unbounded because an LLMs input is unbounded. As a consequence the risk of failure, given AI takes a final decision, is too high for many applications. At least, human review and thus judgment is needed.
This is less of a concern for more classic, tractable AI, i.e. 'classic' machine learning, where error rates can be evaluated in a limitted feature space, and thus the model can be efficiently evaluated ahead of deployment. Thus the risk can be effectively calculated, and if acceptable (another judgment with accountability attached), the decision can be delegated to the AI.
Thanks for the thoughtful note - I used to think something like this too, but Andy Masley made some very good points here, that actually there are many advantages in holding (some) AI accountable over holding humans accountable: https://www.andymasley.com/writing/its-much-easier-to-hold-computers/
I definitely agree that for rogue AIs then it's scary to have AI making final decisions. Same for areas where a company is essentially 'judgment-proof' on a lawsuit, due to the size of the potential harms. But I think that so long as the AI is backstopped, it might plausibly be fine / preferable to human accountability!
This is such a great article, and I enjoyed reading it! I am not a fan of AI in general, so I tend to undermine its ability a lot, so I appreciate your perspective, and I find it quite educational.
For me, the main problem with AI is not what the technology can or cannot do at the moment, or what it will or won't be able to do in the future. The main problem for me is that Big Tech leads the entire conversation and, by proxy, the direction in which this technology is evolving.
I honestly can't imagine a situation in which it would be better to allow AI/machine/anything else that's not a human being to make decisions for us. At the end of the day, you cannot say that you are truly living your life if you cede the right to make choices to someone/something else. The agency to do things (and make mistakes along the way) is what differentiates a stone from a dog, a living creature from a thing. Also, it's worth pointing out that humans already agreed that losing the right to make decisions is a form of punishment, hence prisons where we lock other humans so that they cannot go out when they wish to.
In my mind, even if AI were better, more just/insightful, there really is no point in handing over our agency to it, especially because behind the AI are the corporations with one ultimate goal: profit. They don't care about outcomes for people.
But if I were to name one thing that people can do, which at the same time is an ability that AI probably shouldn't have, it is the ability to say "no" or rebel against the orders/order in general. Like when someone decides not to press "the button" even under the pressure from others around, and thus the world doesn't end with a blinding blast from a nuclear explosion. However, with AI, the same guardrails that should be implemented to make sure that it remains under our control, ultimately may be the cause of a catastrophe, depending on who is in control. Or in a more everyday scenario, I think Shoshana Zuboff in her book "The age of surveillance capitalism" gave an example of Eric Schmidt (I may be misremembering if it was actually him), who said that in the future it would be more sensible to just turn off remotely an engine in a car of a person who was late on their leasing payment (and that could be decided by an algorithm so I think it could be decided by AI as well). It would be more efficient, true. But what if that person was driving to the hospital or was picking up their kids from school?
Ultimately, if a person makes a grave mistake, they can be held accountable in the court of law, but who will be accountable for AI decisions? There's a lot to think about here, so it probably would be easier not to give AI the right to make choices for us. You see, efficiency is just a metric and a subjective one at that.
That's what came to mind when I read your article - sorry for the length of my comment ;-)
Thanks for the thoughtful reply :-) there's a lot to unpack, some quick reflections:
- I hear you on it feeling disempowering to look outside oneself for decision-making - but people also defer to other people on decisions ~all the time. People ask lawyers, doctors, financial advisors, etc., for varying degrees of help eg, with the expectation that they know more than we do; some even hand over full reins to e.g., roboadvisors who handle financial trading in full.
- We just don't have attention for every issue in our lives. And so in principle, looking to AI for input on a decision - or even making the decision outright - seems fine enough to me. Yes ideally I'd know what I'm getting into, have considered whether an error is super high-stakes, etc., but I think we can meet that bar.
- In the prison example, the more meaningful aspect of the punishment is being confined to a place you can't leave, with fewer rights within it. It's true that this comes with reduced decision-making abilities (for instance, sometimes losing the right to vote even once released), but I don't think that's the most salient part of the punishment.
Anyway, thanks for sharing that the post brought up for you, and for considering it so carefully!
It looks like Leif is quite doubtful of superintelligence - would you happen to know if he's described why / laid out an argument for this?
> "They try to figure out how we can survive an event that is not occurring: the emergence of superintelligence. Their thinking aims to solve a very weak science fiction scenario justified by utterly incorrect mathematics."
"Another genuine limitation is AI’s incomplete grasp of tacit knowledge—the know-how gained from experience, which often isn’t written down and so is harder to teach to AI. But lacking this experience is different from fundamentally lacking the capacity for judgment."
I'm not quite sure I can get on board this leap. It depends how you are defining 'judgement'. I think for a lot of people, they are defining it as something like knowledge, experience, intuition, empathy. Making judgement calls requires both cognitive and emotional reasoning. It is fundamentally epistemological. The whole spiel with SkyNet is it made a **logical** reasoned choice in judging humanity.
I know in your article you refer to financial judgement calls etc. so we're not talking law per se, but then we've got to admit there is a limit that A.I. cannot do adequately. The phrase "A.I. cannot make judgements" is rather broad.
"To shape the future we want, it’s better to clearly acknowledge the abilities that AI may soon have — and then consciously decide where we prefer humans anyway."
So then I agree with this. What are the benefits of a machine making a judgement call? I can't see any, to be honest. One might say "it'll be objective" but if we're honest, it's not often full objectivity we want. We trust something that understands the subjectivity in the reasoning as well.
We can use the information to help **us** make a judgement call, but we would not want the A.I. to make a judgement call. At some stage, we've got to decide if we want to, as human beings, actually keep our own agency. I think when people say "A.I. cannot make a judgement" they are not just including the information gathering or a reasoning through the practical meaning of that information, but the decision-making itself. Which is partly why I think when people are greeted with an art work that matches all the specifications but they learn it's an A.I. created work, they feel cheated. It's not just aesthetics people are after, but connection with another human being. Likewise, in judgement, it's this **connection** that's important for human beings.
Thanks for the thoughtful comment - I think the key is to picture a world in which AI is better at making decisions than a human, whether objective or subjective, and then decide where we still want humans to plug in.
When you say this:
> What are the benefits of a machine making a judgement call? I can't see any, to be honest. One might say "it'll be objective" but if we're honest, it's not often full objectivity we want. We trust something that understands the subjectivity in the reasoning as well.
it sounds like you aren't envisioning the same future state that I am. That is, you're still imagining humans to have an edge on subjective reasoning, whereas I'm trying to ask "What happens if that edge doesn't exist either?"
Great article ! People always need to think that we're "special". Religion and hubris probably play a big role there. As Geoffrey Hinton says "a human brain is **just** a bunch of neurons pinging. end of story". I find that Claude has much better judgment than a lof of humans around me (including me sometimes …)
Yeah in an earlier draft, I had a callout at the end that we'd better _hope_ that AI has judgment, because the AI companies are increasingly counting on AI judgment to keep us safe! e.g., Claude's constitution, where Anthropic looks to Claude to resolve a bunch of high-difficulty tradeoffs.
Human beings are special. That's what the Renaissance was about.
I think it's important to note what a loss of this belief would lead to. Geoffrey Hinton wouldn't even have science if not for the fundamental humanistic beliefs of the Enlightenment (itself borrowed from religious ideas).
Hi Ryan ... I wonder what you'd think about my posts about religion shaping our cultural mindset and it's love/hate relationship with science (and the renaissance).
Enlightment happened for many reasons but I believe the driver of humanity is curiosity - I would say that this is what makes us special compared to other animals (and it's an evolutionary trait).
But then what? You can't stop there. I actually think that "we're not that special" is the best way not to be complacent about the world around us and inside our brains.
What made us progress as a species was to challenge our hunter-gatherers intuitions (flat earth, sun revolving around the earth, etc.) and to always find better "world models" than we initially thought.
Thinking that we don't have free will does not destroy agency. Trying to replicate our minds on silicon is a great tool to actually understand what happens there.
I'm not saying it's not dangerous on some level; but I don't think that the "loss of belief" is a problem. Of course, personal and subjective opinion!
Great article, I really enjoyed it. I see a very human story in this : the story of individuals struggling to understand their own meaning, as they watch a technology do all the meaningful stuff. It’s a sad and powerful window on human nature; to resist their own demotion. I don’t see it that way myself, but I think many people do, as they defend the ever-shrinking island of “stuff AI can never do.”
Yeah I think this is well-said - and Sam Altman even expressed something kind of similar yesterday:
"I am very excited about AI, but to go off-script for a minute:
I built an app with Codex last week. It was very fun. Then I started asking it for ideas for new features and at least a couple of them were better than I was thinking of.
I felt a little useless and it was sad.
I am sure we will figure out much better and more interesting ways to spend our time, and amazing new ways to be useful to each other, but I am feeling nostalgic for the present."
And as some of the commenters pointed out: if this is how he feels as the CEO of one of the most successful companies on Earth, how must the mere 'ordinary' workers feel?
I don't think you're being fair to the example in the op-ed. Yes, LLMs "know" that a frosty relationship between CEOs affects acquisitions when asked directly about it. But suppose you fed the LLM all the context it needed: all emails, all financial data, all public statements and interviews and news coverage of the CEOs and companies. Would it still have the judgement to notice and factor the CEO's relationships into its analysis *without explicit prompting*? I'm not saying it wouldn't, but I don't think you've shown that it would either either.
Really great article. Clear thinking. I went back and forth on whether I agreed totally and I think I do, after reading your last section again. But it may be a matter of interpretation, so tell me if I am misreading you.
The act of judgment is able to be performed by AI. But the act of assigning value still would reside with us.
If that is the distinction you are drawing, then I think we are aligned. AI can arbitrate trade-offs once values and goals are specified. It can reason about consequences, weigh competing considerations, and often outperform humans in consistency. The remaining human role is not in performing the mechanics of judgment, but in specifying the value framework within which that judgment operates.
Delegation itself is not the issue. We routinely defer to experts and automated systems because attention is limited and specialization works. The key is that even in those cases, humans define the goals, set the constraints, and remain responsible for the outcomes. AI may execute judgment, but it does so inside a value structure that we establish and maintain.
Even arguments that computers are easier to monitor or hold accountable reinforce this point. Monitoring ensures that a system performs relative to a goal. It does not determine which goals deserve pursuit or how trade-offs should be morally weighted. That upstream assignment of value still rests with us.
So if your claim is that AI can and will master the mechanics of judgment, I agree. The specification of what counts as good, fair, or worth optimizing remains a human responsibility.
This was really insightful and I appreciate the unspoken challenge you’ve given yourself which is to keep your brain/ theory capable of staying on pace and dare I say in front even of the frontier models. If we assume all intelligence will be replaceable we can expend all the energy we use fearing and fighting about it and try to wrest back the only question that matters. What do we WANT our human society to look like. I want humans answering that, preferably one’s still connected to their own humanity
I agree with much of this article. LLMs are pretty good at linguistic-inferential judgements, especially when well prompted with all the relevant inputs.
That said, I think the profile you’re describing is relatively flat and uniformly capable when in reality the cognitive abilities of LLMs are jagged. Firstly, they lack the capacity or desire to initiate. Real-world judgement requires knowing what to ask in the first place, not just responding well when prompted. Secondly, a lot of human judgement is nonverbal, somatic, emotional and contextual and that may be difficult to fully convey via language alone. Thirdly, LLMs have no mechanism to learn from their errors beyond the context window. And fourthly, the hallucination problem is hard to simply “patch” - it’s symptomatic of no direct real-world link other than that mediated by humans, which means the error correction loop depends entirely on human oversight rather than direct consequence.
So yes, judgment isn’t uniquely human but it may still be uniquely whole in humans in a way that current LLMs aren’t.
Steven, this is a sharp piece and I think your empirical case is strong — AI clearly produces outputs that humans rate as displaying judgment, and critics who test outdated models and declare AI incapable are making the exact mistake you describe. I also agree the goalposts will keep moving. But I think you're drawing the wrong conclusion from the right data, and the reason is structural.
Judgment isn't pattern recognition at a higher level of abstraction. It's the act of a free agent stepping outside the system they're operating within to evaluate whether the system itself is adequate. A judge follows precedent until she senses — and "senses" is the right word, because this is not formalizable — that the precedent doesn't fit this case. A doctor follows protocol until something about this patient tells her the protocol is wrong. In every instance, judgment requires the capacity to ask "is my framework good enough?" — and that question cannot be answered from within the framework. It requires a reasoner who transcends the system being evaluated.
AI is a formal system. Given the same inputs and the same random seed, it produces the same outputs. Adding stochasticity doesn't help — randomness is not judgment; nobody claims a coin flip exercises discretion. What judgment requires is directed flexibility: the capacity to recognize, from within the process of reasoning itself, that a new kind of reasoning is needed. A formal system can be extraordinarily sophisticated within its axioms, but it cannot interrogate its own architecture. This is the same argument the emerging philosophical literature makes about consciousness — we know it's real because we experience it directly, not because it's observable from the outside. Free will is first-person evidence of the same kind. And your survey data — people can't tell AI art from human art, people rate AI ethics higher than a columnist's — measures the perception of judgment from the outside. It tells us about outputs. It tells us nothing about whether the process behind them is judgment or its imitation.
The practical stakes here are enormous. If AI has judgment, delegation is rational. If AI imitates judgment, delegation is the abdication of moral agency — things happen and no one is responsible, because the machine isn't a moral agent and the humans have convinced themselves the machine does it better. Your closing line — "consciously decide where we prefer humans" — is exactly right, but it assumes what the rest of your article denies: that human decision-making is categorically different from computation. I think it is. I've written a longer piece responding to your argument https://davidhoze.substack.com/p/yes-ai-can-imitate-judgment-no-thats — I'd genuinely welcome your pushback.
Really helpful perspective! I'll take this one to heart.
Curious what you think about the "average answer" problem: as AI converges, everyone starts getting the same answer to their questions. Insights (and competitive advantage) don't come from data, but from seeing something others cannot see. You could use AI to see what everyone else thinks, but that doesn't mean you should automatically use it for wisdom - or strategy.
Very interesting but if human judgement is not so useful, how do we explain that we constantly give feedback to LLMs in order to get a satisfying answer? Most of the time, we still change parts of the output we get. In my opinion, that’s where our judgment comes in. Even adapting the prompt or the context requires judgment
This has a direct architectural implication that most governance frameworks miss.
If judgment isn't uniquely human, then the infrastructure shouldn't assume it is. The accountability architecture I've been building uses an IDecisionMaker interface — anything that can observe, evaluate, decide, and act implements it. An AI agent. A human. A committee vote. A rules engine. The graph doesn't know or care what's deciding. It records what was decided, by whom, with what confidence, under what authority.
The question shifts from "can AI judge?" to "is the judgment auditable?" A human judge's reasoning lives in their head, partially expressed in an opinion. An AI decision-maker on a hash-chained event graph produces a complete chain: inputs, evaluation, confidence level, authority scope, outcome. Arguably more accountable than the human version.
The people asserting AI can't judge are building governance that assumes it can't. That governance will break the moment it can — which, as you show, is already.
Extremely smart article, love this! Great insights.
Thank you! In writing it, it did raise for me the question of 'what sorts of things _are_ uniquely human," & so I'm curious if anything leapt to mind while reading
I guess this all depends on how judgement is defined / definable, and how it is measured / measurable.
For any definition and measurement that is essentially a function of some data (aka context), subject to some objective measurenent (aka minimized loss), there will eventually be a machine that can be as good as humans, given the same data.
The uniquely human trait, however, will remain that of accountability, at least for a long time to come.
As long as judgement is linked to accountability, AI may be a helpful tool, yet never take the final decision.
That is especially true for the LLM variant of AI, where errors are eseentially unbounded because an LLMs input is unbounded. As a consequence the risk of failure, given AI takes a final decision, is too high for many applications. At least, human review and thus judgment is needed.
This is less of a concern for more classic, tractable AI, i.e. 'classic' machine learning, where error rates can be evaluated in a limitted feature space, and thus the model can be efficiently evaluated ahead of deployment. Thus the risk can be effectively calculated, and if acceptable (another judgment with accountability attached), the decision can be delegated to the AI.
Thanks for the thoughtful note - I used to think something like this too, but Andy Masley made some very good points here, that actually there are many advantages in holding (some) AI accountable over holding humans accountable: https://www.andymasley.com/writing/its-much-easier-to-hold-computers/
I definitely agree that for rogue AIs then it's scary to have AI making final decisions. Same for areas where a company is essentially 'judgment-proof' on a lawsuit, due to the size of the potential harms. But I think that so long as the AI is backstopped, it might plausibly be fine / preferable to human accountability!
This is such a great article, and I enjoyed reading it! I am not a fan of AI in general, so I tend to undermine its ability a lot, so I appreciate your perspective, and I find it quite educational.
For me, the main problem with AI is not what the technology can or cannot do at the moment, or what it will or won't be able to do in the future. The main problem for me is that Big Tech leads the entire conversation and, by proxy, the direction in which this technology is evolving.
I honestly can't imagine a situation in which it would be better to allow AI/machine/anything else that's not a human being to make decisions for us. At the end of the day, you cannot say that you are truly living your life if you cede the right to make choices to someone/something else. The agency to do things (and make mistakes along the way) is what differentiates a stone from a dog, a living creature from a thing. Also, it's worth pointing out that humans already agreed that losing the right to make decisions is a form of punishment, hence prisons where we lock other humans so that they cannot go out when they wish to.
In my mind, even if AI were better, more just/insightful, there really is no point in handing over our agency to it, especially because behind the AI are the corporations with one ultimate goal: profit. They don't care about outcomes for people.
But if I were to name one thing that people can do, which at the same time is an ability that AI probably shouldn't have, it is the ability to say "no" or rebel against the orders/order in general. Like when someone decides not to press "the button" even under the pressure from others around, and thus the world doesn't end with a blinding blast from a nuclear explosion. However, with AI, the same guardrails that should be implemented to make sure that it remains under our control, ultimately may be the cause of a catastrophe, depending on who is in control. Or in a more everyday scenario, I think Shoshana Zuboff in her book "The age of surveillance capitalism" gave an example of Eric Schmidt (I may be misremembering if it was actually him), who said that in the future it would be more sensible to just turn off remotely an engine in a car of a person who was late on their leasing payment (and that could be decided by an algorithm so I think it could be decided by AI as well). It would be more efficient, true. But what if that person was driving to the hospital or was picking up their kids from school?
Ultimately, if a person makes a grave mistake, they can be held accountable in the court of law, but who will be accountable for AI decisions? There's a lot to think about here, so it probably would be easier not to give AI the right to make choices for us. You see, efficiency is just a metric and a subjective one at that.
That's what came to mind when I read your article - sorry for the length of my comment ;-)
Thanks for the thoughtful reply :-) there's a lot to unpack, some quick reflections:
- I hear you on it feeling disempowering to look outside oneself for decision-making - but people also defer to other people on decisions ~all the time. People ask lawyers, doctors, financial advisors, etc., for varying degrees of help eg, with the expectation that they know more than we do; some even hand over full reins to e.g., roboadvisors who handle financial trading in full.
- We just don't have attention for every issue in our lives. And so in principle, looking to AI for input on a decision - or even making the decision outright - seems fine enough to me. Yes ideally I'd know what I'm getting into, have considered whether an error is super high-stakes, etc., but I think we can meet that bar.
- In the prison example, the more meaningful aspect of the punishment is being confined to a place you can't leave, with fewer rights within it. It's true that this comes with reduced decision-making abilities (for instance, sometimes losing the right to vote even once released), but I don't think that's the most salient part of the punishment.
Anyway, thanks for sharing that the post brought up for you, and for considering it so carefully!
Agreed! I've seen this phenomenon called remainder humanism by Leif Weatherby
It looks like Leif is quite doubtful of superintelligence - would you happen to know if he's described why / laid out an argument for this?
> "They try to figure out how we can survive an event that is not occurring: the emergence of superintelligence. Their thinking aims to solve a very weak science fiction scenario justified by utterly incorrect mathematics."
It's in his book I guess..which argues that AI is a language machine, producing culture not intelligence
"Another genuine limitation is AI’s incomplete grasp of tacit knowledge—the know-how gained from experience, which often isn’t written down and so is harder to teach to AI. But lacking this experience is different from fundamentally lacking the capacity for judgment."
I'm not quite sure I can get on board this leap. It depends how you are defining 'judgement'. I think for a lot of people, they are defining it as something like knowledge, experience, intuition, empathy. Making judgement calls requires both cognitive and emotional reasoning. It is fundamentally epistemological. The whole spiel with SkyNet is it made a **logical** reasoned choice in judging humanity.
I know in your article you refer to financial judgement calls etc. so we're not talking law per se, but then we've got to admit there is a limit that A.I. cannot do adequately. The phrase "A.I. cannot make judgements" is rather broad.
"To shape the future we want, it’s better to clearly acknowledge the abilities that AI may soon have — and then consciously decide where we prefer humans anyway."
So then I agree with this. What are the benefits of a machine making a judgement call? I can't see any, to be honest. One might say "it'll be objective" but if we're honest, it's not often full objectivity we want. We trust something that understands the subjectivity in the reasoning as well.
We can use the information to help **us** make a judgement call, but we would not want the A.I. to make a judgement call. At some stage, we've got to decide if we want to, as human beings, actually keep our own agency. I think when people say "A.I. cannot make a judgement" they are not just including the information gathering or a reasoning through the practical meaning of that information, but the decision-making itself. Which is partly why I think when people are greeted with an art work that matches all the specifications but they learn it's an A.I. created work, they feel cheated. It's not just aesthetics people are after, but connection with another human being. Likewise, in judgement, it's this **connection** that's important for human beings.
Thanks for the thoughtful comment - I think the key is to picture a world in which AI is better at making decisions than a human, whether objective or subjective, and then decide where we still want humans to plug in.
When you say this:
> What are the benefits of a machine making a judgement call? I can't see any, to be honest. One might say "it'll be objective" but if we're honest, it's not often full objectivity we want. We trust something that understands the subjectivity in the reasoning as well.
it sounds like you aren't envisioning the same future state that I am. That is, you're still imagining humans to have an edge on subjective reasoning, whereas I'm trying to ask "What happens if that edge doesn't exist either?"
Great article ! People always need to think that we're "special". Religion and hubris probably play a big role there. As Geoffrey Hinton says "a human brain is **just** a bunch of neurons pinging. end of story". I find that Claude has much better judgment than a lof of humans around me (including me sometimes …)
Yeah in an earlier draft, I had a callout at the end that we'd better _hope_ that AI has judgment, because the AI companies are increasingly counting on AI judgment to keep us safe! e.g., Claude's constitution, where Anthropic looks to Claude to resolve a bunch of high-difficulty tradeoffs.
Human beings are special. That's what the Renaissance was about.
I think it's important to note what a loss of this belief would lead to. Geoffrey Hinton wouldn't even have science if not for the fundamental humanistic beliefs of the Enlightenment (itself borrowed from religious ideas).
Hi Ryan ... I wonder what you'd think about my posts about religion shaping our cultural mindset and it's love/hate relationship with science (and the renaissance).
Enlightment happened for many reasons but I believe the driver of humanity is curiosity - I would say that this is what makes us special compared to other animals (and it's an evolutionary trait).
But then what? You can't stop there. I actually think that "we're not that special" is the best way not to be complacent about the world around us and inside our brains.
What made us progress as a species was to challenge our hunter-gatherers intuitions (flat earth, sun revolving around the earth, etc.) and to always find better "world models" than we initially thought.
Thinking that we don't have free will does not destroy agency. Trying to replicate our minds on silicon is a great tool to actually understand what happens there.
I'm not saying it's not dangerous on some level; but I don't think that the "loss of belief" is a problem. Of course, personal and subjective opinion!
Great article, I really enjoyed it. I see a very human story in this : the story of individuals struggling to understand their own meaning, as they watch a technology do all the meaningful stuff. It’s a sad and powerful window on human nature; to resist their own demotion. I don’t see it that way myself, but I think many people do, as they defend the ever-shrinking island of “stuff AI can never do.”
Yeah I think this is well-said - and Sam Altman even expressed something kind of similar yesterday:
"I am very excited about AI, but to go off-script for a minute:
I built an app with Codex last week. It was very fun. Then I started asking it for ideas for new features and at least a couple of them were better than I was thinking of.
I felt a little useless and it was sad.
I am sure we will figure out much better and more interesting ways to spend our time, and amazing new ways to be useful to each other, but I am feeling nostalgic for the present."
And as some of the commenters pointed out: if this is how he feels as the CEO of one of the most successful companies on Earth, how must the mere 'ordinary' workers feel?
Who judges the judgement?
I don't think you're being fair to the example in the op-ed. Yes, LLMs "know" that a frosty relationship between CEOs affects acquisitions when asked directly about it. But suppose you fed the LLM all the context it needed: all emails, all financial data, all public statements and interviews and news coverage of the CEOs and companies. Would it still have the judgement to notice and factor the CEO's relationships into its analysis *without explicit prompting*? I'm not saying it wouldn't, but I don't think you've shown that it would either either.
Really great article. Clear thinking. I went back and forth on whether I agreed totally and I think I do, after reading your last section again. But it may be a matter of interpretation, so tell me if I am misreading you.
The act of judgment is able to be performed by AI. But the act of assigning value still would reside with us.
If that is the distinction you are drawing, then I think we are aligned. AI can arbitrate trade-offs once values and goals are specified. It can reason about consequences, weigh competing considerations, and often outperform humans in consistency. The remaining human role is not in performing the mechanics of judgment, but in specifying the value framework within which that judgment operates.
Delegation itself is not the issue. We routinely defer to experts and automated systems because attention is limited and specialization works. The key is that even in those cases, humans define the goals, set the constraints, and remain responsible for the outcomes. AI may execute judgment, but it does so inside a value structure that we establish and maintain.
Even arguments that computers are easier to monitor or hold accountable reinforce this point. Monitoring ensures that a system performs relative to a goal. It does not determine which goals deserve pursuit or how trade-offs should be morally weighted. That upstream assignment of value still rests with us.
So if your claim is that AI can and will master the mechanics of judgment, I agree. The specification of what counts as good, fair, or worth optimizing remains a human responsibility.
This was really insightful and I appreciate the unspoken challenge you’ve given yourself which is to keep your brain/ theory capable of staying on pace and dare I say in front even of the frontier models. If we assume all intelligence will be replaceable we can expend all the energy we use fearing and fighting about it and try to wrest back the only question that matters. What do we WANT our human society to look like. I want humans answering that, preferably one’s still connected to their own humanity
I agree with much of this article. LLMs are pretty good at linguistic-inferential judgements, especially when well prompted with all the relevant inputs.
That said, I think the profile you’re describing is relatively flat and uniformly capable when in reality the cognitive abilities of LLMs are jagged. Firstly, they lack the capacity or desire to initiate. Real-world judgement requires knowing what to ask in the first place, not just responding well when prompted. Secondly, a lot of human judgement is nonverbal, somatic, emotional and contextual and that may be difficult to fully convey via language alone. Thirdly, LLMs have no mechanism to learn from their errors beyond the context window. And fourthly, the hallucination problem is hard to simply “patch” - it’s symptomatic of no direct real-world link other than that mediated by humans, which means the error correction loop depends entirely on human oversight rather than direct consequence.
So yes, judgment isn’t uniquely human but it may still be uniquely whole in humans in a way that current LLMs aren’t.
Steven, this is a sharp piece and I think your empirical case is strong — AI clearly produces outputs that humans rate as displaying judgment, and critics who test outdated models and declare AI incapable are making the exact mistake you describe. I also agree the goalposts will keep moving. But I think you're drawing the wrong conclusion from the right data, and the reason is structural.
Judgment isn't pattern recognition at a higher level of abstraction. It's the act of a free agent stepping outside the system they're operating within to evaluate whether the system itself is adequate. A judge follows precedent until she senses — and "senses" is the right word, because this is not formalizable — that the precedent doesn't fit this case. A doctor follows protocol until something about this patient tells her the protocol is wrong. In every instance, judgment requires the capacity to ask "is my framework good enough?" — and that question cannot be answered from within the framework. It requires a reasoner who transcends the system being evaluated.
AI is a formal system. Given the same inputs and the same random seed, it produces the same outputs. Adding stochasticity doesn't help — randomness is not judgment; nobody claims a coin flip exercises discretion. What judgment requires is directed flexibility: the capacity to recognize, from within the process of reasoning itself, that a new kind of reasoning is needed. A formal system can be extraordinarily sophisticated within its axioms, but it cannot interrogate its own architecture. This is the same argument the emerging philosophical literature makes about consciousness — we know it's real because we experience it directly, not because it's observable from the outside. Free will is first-person evidence of the same kind. And your survey data — people can't tell AI art from human art, people rate AI ethics higher than a columnist's — measures the perception of judgment from the outside. It tells us about outputs. It tells us nothing about whether the process behind them is judgment or its imitation.
The practical stakes here are enormous. If AI has judgment, delegation is rational. If AI imitates judgment, delegation is the abdication of moral agency — things happen and no one is responsible, because the machine isn't a moral agent and the humans have convinced themselves the machine does it better. Your closing line — "consciously decide where we prefer humans" — is exactly right, but it assumes what the rest of your article denies: that human decision-making is categorically different from computation. I think it is. I've written a longer piece responding to your argument https://davidhoze.substack.com/p/yes-ai-can-imitate-judgment-no-thats — I'd genuinely welcome your pushback.
Really helpful perspective! I'll take this one to heart.
Curious what you think about the "average answer" problem: as AI converges, everyone starts getting the same answer to their questions. Insights (and competitive advantage) don't come from data, but from seeing something others cannot see. You could use AI to see what everyone else thinks, but that doesn't mean you should automatically use it for wisdom - or strategy.
Thanks!
Very interesting but if human judgement is not so useful, how do we explain that we constantly give feedback to LLMs in order to get a satisfying answer? Most of the time, we still change parts of the output we get. In my opinion, that’s where our judgment comes in. Even adapting the prompt or the context requires judgment
This has a direct architectural implication that most governance frameworks miss.
If judgment isn't uniquely human, then the infrastructure shouldn't assume it is. The accountability architecture I've been building uses an IDecisionMaker interface — anything that can observe, evaluate, decide, and act implements it. An AI agent. A human. A committee vote. A rules engine. The graph doesn't know or care what's deciding. It records what was decided, by whom, with what confidence, under what authority.
The question shifts from "can AI judge?" to "is the judgment auditable?" A human judge's reasoning lives in their head, partially expressed in an opinion. An AI decision-maker on a hash-chained event graph produces a complete chain: inputs, evaluation, confidence level, authority scope, outcome. Arguably more accountable than the human version.
The people asserting AI can't judge are building governance that assumes it can't. That governance will break the moment it can — which, as you show, is already.
mattsearles2.substack.com