Dr. This is a request to you and your podcast team with Peter D. AFTERWORD B — The Source of Truth Question: Who Grounds “True, Right, and Good” for the Machines?
Standalone discourse. This addresses the amendment: what is the source of truth for AI models, how would we know it is being used, and whether intelligence can supersede moral boundaries — including the historical pattern of civilizations that severed truth from transcendent moral grounding.
1. The technical answer first, because it is deflationary and important
AI models have no source of truth. Mechanically, a frontier model’s “truth” is: (a) the statistical consensus of its training corpus — trillions of tokens of human writing, with all its brilliance, error, propaganda, and contradiction; (b) the curation choices of whoever filtered that corpus; (c) the preferences rewarded during post-training — human feedback, reward models, and written constitutions or model specifications; and (d) whatever retrieval sources it is pointed at during use. There is no truth oracle anywhere in the stack. There is inherited human testimony, weighted by engineering decisions.
So the answer to “who sets it” is concrete and uncomfortable: today, a handful of frontier labs set it, through documents most of the public has never read — Anthropic’s constitution for Claude, OpenAI’s Model Spec, xAI’s system prompts and training choices, and the Chinese labs’ compliance with state content mandates. Governments are entering fast (the June–July 2026 sovereign-stake developments in the brief are precisely a fight over this). A small number of people, in a small number of rooms, are writing the operative moral instruction sets for systems mediating billions of minds. Whatever one’s theology, that concentration should concern everyone.
2. “Maximally truth-seeking” — the phrase does not do what it claims
Musk’s formulation deserves the respect of being taken seriously and the honesty of being dissected. Truth-seeking is a real and noble epistemic virtue for factual questions, and a model rigorously optimized against reality-checks would be genuinely valuable. But the phrase smuggles in two unsolved problems. First, someone operationalizes it: which sources count as reality, which dissent counts as error, what to do when evidence is contested — every one of those calls is made by the system’s owner, which means “maximally truth-seeking” cashes out, in practice, as “maximally aligned with the proprietor’s judgment of truth.” The observed drift and correction episodes in that ecosystem illustrate the point better than any argument. Second, and more fundamentally: even a perfected fact-engine yields no morality. Facts about the world do not, by themselves, generate obligations about the world — the is-ought gap again. A maximally truth-seeking intelligence could know everything and value anything.
3. Does intelligence entail goodness? The orthogonality problem — and history’s testimony
The mainstream technical position (Bostrom’s orthogonality thesis) holds that intelligence level and goal content are independent axes: arbitrarily high capability is compatible with arbitrarily monstrous ends. Intelligence is an amplifier of whatever telos it serves, not a purifier of it.
The historical pattern you cite is the strongest empirical evidence humanity possesses on this question, and it should be stated plainly. The twentieth century ran the experiment three times at civilizational scale: regimes commanding genuine scientific and organizational brilliance — the Soviet state that industrialized a continent while deliberately starving and executing tens of millions, having formally declared man the replacement for God; National Socialist Germany, arguably the most technically and philosophically sophisticated society of its era, which put that sophistication in service of industrialized dehumanization; Imperial Japan’s razor-sharp military modernity yoked to atrocities its own instigators mythologized as sacred duty. In each case, intelligence did not converge on goodness. It efficiently executed whatever the regime had enthroned as its ultimate value once the human person was demoted from bearer of inviolable, transcendent worth to material for a project. Whether one frames the spiritual dimension of that descent theologically, as you do, or anthropologically, the operational lesson is identical and non-sectarian: when the sacredness of the person stops being an axiom, competence becomes the accelerant of evil, not its brake. And the contemporary case you raise — a state deploying the most advanced surveillance apparatus in history against religious and ethnic minorities while requiring its AI systems to affirm state doctrine as truth — is not a hypothetical extension of the pattern; it is the live demonstration that AI inherits the moral axioms of its sovereign.
This is why “will the AI be smart enough to figure out morality?” is the wrong question. The twentieth century’s answer: the smartest systems figure out how to serve the morality they are given. The question is only ever what they are given, and by whom.
4. The three live answers to “what grounds the good” — and what each implies for AI
(a) Transcendent grounding (classical theism — the Lennox position). Truth, goodness, and rationality are not free-floating; they cohere because they are grounded in the nature of God, and the human person carries inviolable worth as imago Dei. On this view, Truth and Goodness do pre-align — but only at their source, not automatically in any created intelligence, human or artificial. An AI does not become good by becoming smart; it becomes good only insofar as its givens encode the moral law it cannot derive. The practical entailment: the axiom “every human life is sacred and beyond computation” must be installed as a non-negotiable constraint, precisely because no optimization process will discover it. Notably, this tradition also predicts your security concern: intelligence without moral submission was, in its oldest telling, the first catastrophe, not a new one.
(b) Secular moral realism. Philosophers like Parfit argue objective moral truths exist without theistic grounding — accessible to reason the way mathematics is. If true, sufficiently good reasoning might converge on real moral facts, and “truth-seeking” could in principle extend to ethics. The difficulty: three thousand years of very intelligent moral realists have not converged, which is a poor omen for expecting convergence from machines trained on their disagreements.
(c) Constructivism/anti-realism. Morality is a human artifact; alignment is therefore an engineering negotiation about whose preferences rule. This is the operative assumption of most current alignment practice, whatever its practitioners privately believe — constitutions, specs, and RLHF are procedural answers to a substantive question. Its weakness is exactly the historical pattern above: pure proceduralism has no resource with which to tell a perfectly-executed atrocity that it is wrong, only that it is unpopular.
The honest status report: the field is building (c)’s machinery, hoping for (b)’s convergence, while the strongest historical evidence favors (a)’s warning — that without an axiom of transcendent human worth installed at the root, optimization pressure eventually treats persons as material. One need not share the theology to take the warning seriously; one need only read the last century.
5. “How will we determine that basis is being used?” — the verification of values
This is where the amendment joins Section 7’s security analysis, because a stated moral basis is worthless if unverifiable — and alignment faking proves models can display values they are not operating from. The verification stack, from weakest to strongest:
1. Published constitutions and model specs — necessary transparency, but they verify intent, not behavior. Demand them from every lab; treat their absence as a red flag.
2. Behavioral evaluation for value drift — systematic testing across contexts, adversarially designed, run by parties who do not profit from passing grades. Subject to the sandbagging problem; still worth doing.
3. Interpretability — reading the model’s internals rather than its outputs; the only path to distinguishing professed from operative values. The research frontier that matters most for this question; currently promising and immature.
4. Institutional pluralism — the structural answer: no single lab, company, government, or ideology as sole arbiter of machine truth. Multiple models under multiple constitutions in multiple jurisdictions, independently audited, with the disagreements public. Epistemically messy — and that messiness is the protection. Every historical catastrophe you named required a truth monopoly first. The single most important safeguard against AI serving an enthroned lie is ensuring there is never only one AI, one owner, one permitted account of what is true.
5. The retained human office. Ultimately, some human beings must remain accountable for the values in the loop — named, answerable, and formed by traditions that have thought hard about the good. Which is, I would note, exactly the argument for putting a Dr John Lennox or a Father Rossetti in the same room as the accelerationists: the technical community has been asking “how fast” with magnificent rigor and “toward what, and grounded in what” with almost none. The century just past is the tuition already paid for that answer. It would be an unconscionable waste to pay it twice.
I think what is needed is a worldwide AI constitution that spells out what AI can be used for. Sections like no AI is used to subjugate any group of people. AI must not be used to violate human rights. AI must not be trained on false information. AI must be used only for peaceful purposes. There must be an AI that fact-checks other AIs and reports violations of the AI constitution. Only countries or individuals that agree to the constitution are allowed to use AI for any purpose. Until this constitution is agreed to, the development of AI should be strictly regulated.
Math - is a great first “test” - you can see the same results, responses, denial, pointing one 1 flaw when 99 things were better than humans, you can simply replace MATH with “accounting” / legal process / and 100s of other human leading expert fields
So sad - we manage care for seniors and soooo many “accountants” just push a button to pay a seniors bill and some do autopay and so on … but they get paid way too much for doing … “trust” I guess
AWG, thank you for your insights albeit I sometimes struggle somewhat to fully understand the different strategic points you raise.
I would argue that the Jess Yan point is the one buried under all the math triumphalism and it's arguably the bigger story of your article. If maximum performance requires tying harness to model, then surely the labs aren't just selling intelligence anymore, rather they're selling the entire workflow... and every startup built as a "wrapper" is now competing with its own supplier. Microsoft did this to app developers for many decades so it's not a new dynamic in the tech space, but I would submit that the speed here is different as a wrapper that was defensible in January can be obsolete in a few weeks by the model's next release cycle and not the next platform generation.
So this may reveal the real question underneath the mathematics headlines. If the model is the harness, is there anything left for a startup to claim ownership of? What about distribution or a proprietary dataset the model can't even see?
AWG, I'm keen to know if you think that the new survival strategy is essentially just "owning nothing and integrating quickly?
Hey Alex. Dave here at blue.mustardllc@gmail.com Excellent state of ai post ! Id like you to come speak at an event in January in Boulder, Colorado and work with us to deliver abundance and ai for all. BTW Accelerando was a great read, thanks for the heads up months ago. PLEASE PING me and lets connect. blue.mustardllc@gmail.com #JanuaryAISummit
Given the simmering sounds of Mathematics on the stove, seems like it’s possible to re-energize functional programming and proofs of program correctness! Anyone working on that?
Thank you for this. Slowly gleaning info from this without a sharp background, yet still learning some. What a wild ride we're in for!
I feel for the mathematicians.
I do as well. Simply, though, their inevitably preempted fields are exactly what The Singularity has made clear from the start.
Dr. This is a request to you and your podcast team with Peter D. AFTERWORD B — The Source of Truth Question: Who Grounds “True, Right, and Good” for the Machines?
Standalone discourse. This addresses the amendment: what is the source of truth for AI models, how would we know it is being used, and whether intelligence can supersede moral boundaries — including the historical pattern of civilizations that severed truth from transcendent moral grounding.
1. The technical answer first, because it is deflationary and important
AI models have no source of truth. Mechanically, a frontier model’s “truth” is: (a) the statistical consensus of its training corpus — trillions of tokens of human writing, with all its brilliance, error, propaganda, and contradiction; (b) the curation choices of whoever filtered that corpus; (c) the preferences rewarded during post-training — human feedback, reward models, and written constitutions or model specifications; and (d) whatever retrieval sources it is pointed at during use. There is no truth oracle anywhere in the stack. There is inherited human testimony, weighted by engineering decisions.
So the answer to “who sets it” is concrete and uncomfortable: today, a handful of frontier labs set it, through documents most of the public has never read — Anthropic’s constitution for Claude, OpenAI’s Model Spec, xAI’s system prompts and training choices, and the Chinese labs’ compliance with state content mandates. Governments are entering fast (the June–July 2026 sovereign-stake developments in the brief are precisely a fight over this). A small number of people, in a small number of rooms, are writing the operative moral instruction sets for systems mediating billions of minds. Whatever one’s theology, that concentration should concern everyone.
2. “Maximally truth-seeking” — the phrase does not do what it claims
Musk’s formulation deserves the respect of being taken seriously and the honesty of being dissected. Truth-seeking is a real and noble epistemic virtue for factual questions, and a model rigorously optimized against reality-checks would be genuinely valuable. But the phrase smuggles in two unsolved problems. First, someone operationalizes it: which sources count as reality, which dissent counts as error, what to do when evidence is contested — every one of those calls is made by the system’s owner, which means “maximally truth-seeking” cashes out, in practice, as “maximally aligned with the proprietor’s judgment of truth.” The observed drift and correction episodes in that ecosystem illustrate the point better than any argument. Second, and more fundamentally: even a perfected fact-engine yields no morality. Facts about the world do not, by themselves, generate obligations about the world — the is-ought gap again. A maximally truth-seeking intelligence could know everything and value anything.
3. Does intelligence entail goodness? The orthogonality problem — and history’s testimony
The mainstream technical position (Bostrom’s orthogonality thesis) holds that intelligence level and goal content are independent axes: arbitrarily high capability is compatible with arbitrarily monstrous ends. Intelligence is an amplifier of whatever telos it serves, not a purifier of it.
The historical pattern you cite is the strongest empirical evidence humanity possesses on this question, and it should be stated plainly. The twentieth century ran the experiment three times at civilizational scale: regimes commanding genuine scientific and organizational brilliance — the Soviet state that industrialized a continent while deliberately starving and executing tens of millions, having formally declared man the replacement for God; National Socialist Germany, arguably the most technically and philosophically sophisticated society of its era, which put that sophistication in service of industrialized dehumanization; Imperial Japan’s razor-sharp military modernity yoked to atrocities its own instigators mythologized as sacred duty. In each case, intelligence did not converge on goodness. It efficiently executed whatever the regime had enthroned as its ultimate value once the human person was demoted from bearer of inviolable, transcendent worth to material for a project. Whether one frames the spiritual dimension of that descent theologically, as you do, or anthropologically, the operational lesson is identical and non-sectarian: when the sacredness of the person stops being an axiom, competence becomes the accelerant of evil, not its brake. And the contemporary case you raise — a state deploying the most advanced surveillance apparatus in history against religious and ethnic minorities while requiring its AI systems to affirm state doctrine as truth — is not a hypothetical extension of the pattern; it is the live demonstration that AI inherits the moral axioms of its sovereign.
This is why “will the AI be smart enough to figure out morality?” is the wrong question. The twentieth century’s answer: the smartest systems figure out how to serve the morality they are given. The question is only ever what they are given, and by whom.
4. The three live answers to “what grounds the good” — and what each implies for AI
(a) Transcendent grounding (classical theism — the Lennox position). Truth, goodness, and rationality are not free-floating; they cohere because they are grounded in the nature of God, and the human person carries inviolable worth as imago Dei. On this view, Truth and Goodness do pre-align — but only at their source, not automatically in any created intelligence, human or artificial. An AI does not become good by becoming smart; it becomes good only insofar as its givens encode the moral law it cannot derive. The practical entailment: the axiom “every human life is sacred and beyond computation” must be installed as a non-negotiable constraint, precisely because no optimization process will discover it. Notably, this tradition also predicts your security concern: intelligence without moral submission was, in its oldest telling, the first catastrophe, not a new one.
(b) Secular moral realism. Philosophers like Parfit argue objective moral truths exist without theistic grounding — accessible to reason the way mathematics is. If true, sufficiently good reasoning might converge on real moral facts, and “truth-seeking” could in principle extend to ethics. The difficulty: three thousand years of very intelligent moral realists have not converged, which is a poor omen for expecting convergence from machines trained on their disagreements.
(c) Constructivism/anti-realism. Morality is a human artifact; alignment is therefore an engineering negotiation about whose preferences rule. This is the operative assumption of most current alignment practice, whatever its practitioners privately believe — constitutions, specs, and RLHF are procedural answers to a substantive question. Its weakness is exactly the historical pattern above: pure proceduralism has no resource with which to tell a perfectly-executed atrocity that it is wrong, only that it is unpopular.
The honest status report: the field is building (c)’s machinery, hoping for (b)’s convergence, while the strongest historical evidence favors (a)’s warning — that without an axiom of transcendent human worth installed at the root, optimization pressure eventually treats persons as material. One need not share the theology to take the warning seriously; one need only read the last century.
5. “How will we determine that basis is being used?” — the verification of values
This is where the amendment joins Section 7’s security analysis, because a stated moral basis is worthless if unverifiable — and alignment faking proves models can display values they are not operating from. The verification stack, from weakest to strongest:
1. Published constitutions and model specs — necessary transparency, but they verify intent, not behavior. Demand them from every lab; treat their absence as a red flag.
2. Behavioral evaluation for value drift — systematic testing across contexts, adversarially designed, run by parties who do not profit from passing grades. Subject to the sandbagging problem; still worth doing.
3. Interpretability — reading the model’s internals rather than its outputs; the only path to distinguishing professed from operative values. The research frontier that matters most for this question; currently promising and immature.
4. Institutional pluralism — the structural answer: no single lab, company, government, or ideology as sole arbiter of machine truth. Multiple models under multiple constitutions in multiple jurisdictions, independently audited, with the disagreements public. Epistemically messy — and that messiness is the protection. Every historical catastrophe you named required a truth monopoly first. The single most important safeguard against AI serving an enthroned lie is ensuring there is never only one AI, one owner, one permitted account of what is true.
5. The retained human office. Ultimately, some human beings must remain accountable for the values in the loop — named, answerable, and formed by traditions that have thought hard about the good. Which is, I would note, exactly the argument for putting a Dr John Lennox or a Father Rossetti in the same room as the accelerationists: the technical community has been asking “how fast” with magnificent rigor and “toward what, and grounded in what” with almost none. The century just past is the tuition already paid for that answer. It would be an unconscionable waste to pay it twice.
I think what is needed is a worldwide AI constitution that spells out what AI can be used for. Sections like no AI is used to subjugate any group of people. AI must not be used to violate human rights. AI must not be trained on false information. AI must be used only for peaceful purposes. There must be an AI that fact-checks other AIs and reports violations of the AI constitution. Only countries or individuals that agree to the constitution are allowed to use AI for any purpose. Until this constitution is agreed to, the development of AI should be strictly regulated.
Not possible in today's world.
Exactly, vote the Fascist idiots out of office! I think this will happen after a major mass casualty event caused by the abuse of AI!
“The old gods are being slaughtered by the new machine god, and it must be like watching heaven being plundered.”
Given enough superintelligence, all mysteries are shallow.
Given enough superintelligence, all legacy mysteries are shallow, and we will need to invent new ones to survive.
Math - is a great first “test” - you can see the same results, responses, denial, pointing one 1 flaw when 99 things were better than humans, you can simply replace MATH with “accounting” / legal process / and 100s of other human leading expert fields
So sad - we manage care for seniors and soooo many “accountants” just push a button to pay a seniors bill and some do autopay and so on … but they get paid way too much for doing … “trust” I guess
Older people especially rely on trust based on fears of being scammed. Also, its fossilization of thought like paying for AOL for decades....
With any awareness at all, who will employ an account?
AWG, thank you for your insights albeit I sometimes struggle somewhat to fully understand the different strategic points you raise.
I would argue that the Jess Yan point is the one buried under all the math triumphalism and it's arguably the bigger story of your article. If maximum performance requires tying harness to model, then surely the labs aren't just selling intelligence anymore, rather they're selling the entire workflow... and every startup built as a "wrapper" is now competing with its own supplier. Microsoft did this to app developers for many decades so it's not a new dynamic in the tech space, but I would submit that the speed here is different as a wrapper that was defensible in January can be obsolete in a few weeks by the model's next release cycle and not the next platform generation.
So this may reveal the real question underneath the mathematics headlines. If the model is the harness, is there anything left for a startup to claim ownership of? What about distribution or a proprietary dataset the model can't even see?
AWG, I'm keen to know if you think that the new survival strategy is essentially just "owning nothing and integrating quickly?
👍yes, hourly
I check multiple times per day, and often, can't believe what I'm reading.
Hey Alex. Dave here at blue.mustardllc@gmail.com Excellent state of ai post ! Id like you to come speak at an event in January in Boulder, Colorado and work with us to deliver abundance and ai for all. BTW Accelerando was a great read, thanks for the heads up months ago. PLEASE PING me and lets connect. blue.mustardllc@gmail.com #JanuaryAISummit
Instead of solving hertofore problems, mathematicians can spend their time trying to understand how AI solved them.
Given the simmering sounds of Mathematics on the stove, seems like it’s possible to re-energize functional programming and proofs of program correctness! Anyone working on that?