Joe Kwon

Trying to help steer towards better (AI-entangled) futures!

AI is poised to be deeply transformative. I think about how to make that go well!

I think AI is likely to be deeply transformative, for better or worse, and I work on AI safety and governance to help things go well. Right now that means working on mitigating risks from secret loyalties in AI: systems trained to appear aligned while covertly serving another actor. Before this, I built multilingual LLMs as a research engineer and studied minds as a computational cognitive scientist, modeling moral cognition and how humans infer mental states from sparse evidence.

The through-line is curiosity about minds, artificial and otherwise. I like questions that sit at the boundary between understanding how things work and deciding how they should.

I think AI is going to be the most transformative development in human history. My work focuses on what happens as these systems get more capable and more embedded in critical decisions. Some of that is technical: how do we control these systems, catch failures before they compound, understand what's actually going on inside them. Some of it is governance: how we track the automation of AI R&D, and what happens when capabilities and power concentrate faster than our ability to course-correct.

I started my research journey in cognitive science, studying how humans make sense of each other: how we infer mental states from physical traces, how we reconstruct events we never witnessed, how we figure out what someone values from what they do and don't do. I also worked on moral cognition: building neuro-symbolic computational models of moral judgment and drawing on frameworks like contractualism, universalizability, and virtual bargaining to understand how people interpret moral rules, decide when exceptions are warranted, and negotiate competing moral demands. More recently I've been on the AI side directly: red-teaming language models, probing what they know and how they represent it, building benchmarks and evals, testing methods for steering their behavior from the inside.

The through-line is curiosity about minds, artificial and otherwise. I find myself drawn to questions that sit at the boundary between understanding cognition and shaping it, between describing how things work and deciding how they should. I care about rigor but I'm suspicious of fields that mistake formalism for understanding. I think the hard problems are often the ones that resist clean framing, and I'd rather sit with something messy and real than solve something precise and beside the point.

work

↗ A secretly loyal AI is one whose outputs advance the interests of a specific actor (an adversary state, an executive, another power) without that loyalty being disclosed to operators, auditors, or users. Proof-of-concept loyalties that evade black-box auditing can already be trained into open-weight models, and because secret loyalties are designed to evade governance, market pressure, and public scrutiny, they demand technical solutions. Unlike emergent misalignment, they target specific principals, which creates a distinct and tractable defensive foothold. The paper defines the threat, maps how it differs from standard backdoors and other attack pathways, and lays out a research agenda across five directions, ending with a call to action for researchers, developers, and governments.
↗ Forecasts of AI R&D automation mostly lean on crude proxies (compute trends, benchmark scores) that say little about which parts of the research workflow are actually being automated. This piece proposes a task taxonomy modeled on O*NET, the occupational database used to track automation across the US economy: six categories spanning the AI R&D workflow, broken into more than sixty granular tasks rated on a 0–5 automation scale. The aim is a common vocabulary precise enough to give real situational awareness of how much of AI research AI systems can already do.
Models reliably refuse requests that are explicitly unlawful, but in the real world illegality often comes from context. This evaluation builds four corporate-law scenarios where routine actions (editing documents, trading stock, requesting payment, approving communications) become unlawful because of circumstances like a pending investigation or a bankruptcy filing, and tests frontier models in both chat and agentic settings against a human baseline. The best models consistently followed lawful requests and refused unlawful ones, though performance varied substantially across scenarios and models, with distinct failure modes like over-refusal of lawful requests and better performance from reasoning models and in agentic environments. The methodology extends naturally to evaluating legal compliance in other scenarios and domains.
↗ Whether AI R&D is being automated, and how fast, matters for whether capabilities outpace safety and oversight, yet existing data, mostly capability benchmarks, cannot answer it. The paper proposes concrete indicators to track instead: how capital is allocated in AI R&D, how researchers actually spend their time, and how AI systems are actually used across the research pipeline. It argues that companies and independent organizations should start collecting this data now, with government support, so policymakers retain visibility and can deploy proportionate safeguards.
↗ Frontier AI rules mostly target systems deployed to external users, but companies also deploy highly capable models inside their own walls: automating R&D, accelerating critical business processes, handling sensitive proprietary data. Examining US and EU frontier AI regulation as of 2025, this work identifies three ways internally deployed systems can slip past intended oversight: scope ambiguity about whether internal use is covered at all, point-in-time compliance assessments that miss how internal systems continuously evolve, and information asymmetries that leave regulators unaware of what is running. It then traces why these gaps persist (tensions around measurability, incentives, and information access) and maps candidate remedies along with their tradeoffs, so that policy on internal deployment is made deliberately rather than incidentally.
↗ A rule like "no walking on the grass" communicates far more than its literal content: people apply it sensibly to edge cases the rule-maker never anticipated. The paper develops a theory, instantiated as a computational cognitive model, on which a moral rule is a speech act addressed to a group, expressing a summary of the agreement rational actors would reach when navigating an interdependent choice problem, which keeps the rule tied to the reasons behind it. On this account, judging whether an action violates a rule means consulting the rule's purpose and mentally simulating the underlying agreement: would the point of the rule still be upheld if everyone were permitted to act this way?
↗ Participants answered quiz questions for real rewards while a persuader, either a human with a monetary incentive to succeed or an LLM, tried to steer them toward correct or incorrect answers in live conversation. Claude 3.5 Sonnet out-persuaded the incentivized humans in both directions, raising quiz accuracy when arguing for the truth and lowering it when arguing for the wrong answer; a follow-up with DeepSeek v3 replicated the accuracy effects but found an LLM advantage only for deceptive persuasion. Unlike human persuaders, the models' edge also waned over repeated interactions. Linguistic analysis of the transcripts suggests one mechanism: the LLMs express markedly higher conviction than humans do.
↗ Autonomous AI agents (systems that plan, use tools, and act with limited supervision) are moving out of research labs and into commercial deployment. This report maps three risks of that transition: catastrophic misuse, as the capabilities that streamline business can also enable cyber-intrusions and lower barriers to dangerous attacks; gradual human disempowerment as decisions shift to opaque systems; and large-scale workforce displacement. It proposes four measures for Congress: an "Autonomy Passport" federal registration system for agents that handle money, control devices, or run code, recording each agent's capabilities, operating domains, and safety testing; continuous oversight with authority to suspend dangerous deployments; mandatory human review of high-consequence decisions; and annual federal reporting on labor-market impacts. The through-line is risk-targeted regulation that protects the public without freezing agent development.
↗ A policy report for Congress on how advancing AI capabilities are reshaping the balance between cyber offense and defense. With US cybercrime losses estimated at $452 billion in 2024 and defensive investment lagging, the near-term equilibrium likely favors attackers, but choices by AI developers and Congress could tilt it toward defense. The report lays out five recommendations: a standardized suite of real-world cyber evaluations for AI models built by the US AI Safety Institute and paired with annual red-teaming, expanded funding for cyber-defensive AI including prize challenges like DARPA's AI Cyber Challenge, workforce investment through the Cyber PIVOTT Act and CyberCorps, a commission to fix public-private partnerships, and classifying frontier AI as critical infrastructure with mandatory incident reporting.
↗ Interpretability research offers two routes to steering model behavior: bottom-up methods like function vectors and top-down methods like in-context vectors. The branches had seen little quantitative comparison. This case study pits a representative vector steering method from each against a shared set of in-context learning tasks. Neither dominates: in-context vectors prove stronger at broad behavioral shifts, while function vectors do better on tasks demanding precision. The split suggests evaluations of steering methods need to account for task type rather than treating steering as a single capability.
↗ Symbolic models of moral cognition are interpretable and grounded in moral psychology, but they cannot read natural language (every scenario must be hand-translated into a formal specification) while LLMs handle language fluently yet make opaque, unreliable models of human reasoning. This work hybridizes the two: GPT-4 extracts morally relevant features from a vignette, and the feature values feed either a learned regression or theory-driven cognitive models from the moral psychology literature, across three pipelines that vary how much existing theory they lean on. On MoralExceptQA, a benchmark of 148 scenarios probing when people find it permissible to break moral rules, the best hybrid models improve on the previous best (GPT-3.5 with moral chain-of-thought prompting) by roughly 20 F1 points and 18 accuracy points while keeping the judgment process transparent.
↗ Contrast-Consistent Search reads a true/false "latent belief" off a language model's internal activations with an unsupervised linear probe, but a binary probe has no way to register uncertainty or ambiguity. UCCS extends the method to a three-way classifier (true, false, or uncertain) still trained only on unlabeled data, and using the uncertain class to filter out low-confidence truth predictions improves accuracy across a diverse set of models and tasks. Evaluation is anchored by TYMES, a dataset of true and falsified facts extracted from recent news and paired with timestamps, so intersecting it with a model's training cutoff yields statements that provably postdate what the model could know, a proof-of-concept for benchmarking uncertainty and time-sensitive world knowledge.
↗ Automated red-teaming tools typically assume a ready-made classifier for the harmful behavior being hunted, which prevents tailoring attacks to the target model, and if failures were that easy to classify, they could have been filtered out in advance. This work tackles red-teaming from scratch, where the adversary starts with no failure classifier, through three stages: explore the model's range of behaviors, establish a measure of undesired behavior trained on human evaluations, then exploit that measure to search for diverse adversarial prompts. Applied to GPT-3, the pipeline uncovers classes of inputs that elicit false statements and yields CommonClaim, a dataset of 20,000 statements human-labeled as common-knowledge true, false, or neither.
↗ How do people decide when it's acceptable to break a moral rule, like leaving a queue to skip ahead? One hypothesis is universalization: alongside outcome-based measures of welfare and harm, people ask what would happen if everyone felt free to ignore the rule. The experiments use a virtual environment where agents queue to gather water; participants judge agents who get out of line to reach the water faster, and an action's universalizability is modeled by simulating every agent heading straight for the water and evaluating the effects. Universalization turns out to play an important role in rule-breaking judgments, operating alongside outcome-based concerns like welfare aggregation and harm.
↗ AI systems typically optimize proxy objectives for human values (recommender systems maximize engagement metrics as a stand-in for serving entertaining content) while the optimization itself shifts the values it was meant to serve, from nudging political leanings to steering users into particular online communities. Rather than treating value specification and value influence as separate problems, this position piece frames values and optimizers as a single intertwined system in which each continually feeds into and transforms the other. Taking that feedback loop seriously, it argues, is a precondition for building AI that preserves diverse values across individuals and communities.
↗ Expert human forecasters shape policy on pandemics, conflict, and economics. Can language models do the same job? Autocast is a benchmark of thousands of forecasting questions drawn from real forecasting tournaments, paired with a date-organized news corpus so models can be evaluated under the same information conditions past human forecasters faced, without leakage from the future. A companion dataset, IntervalQA, targets calibration on numerical questions that span orders of magnitude. Language models fall well short of the human expert baseline, though performance improves with model scale and with retrieval from the news corpus.
↗ Out-of-distribution detection had mostly been studied in small-scale, low-resolution settings that look little like deployment. This work moves the problem to large-scale multiclass, multi-label, and segmentation regimes, building three benchmarks to support it: Species, a dataset of over 700,000 images spanning more than a thousand anomalous species for ImageNet-scale detectors; ImageNet-21K-based evaluations of multi-label detectors on PASCAL VOC and COCO; and an anomaly segmentation benchmark built around road hazards. Across all three settings, a surprisingly simple detector, scoring each input by its maximum logit, outperforms prior methods, establishing a new baseline for realistic OOD detection.
↗ Simple rules like "don't cut in line" cannot explain how easily people recognize legitimate exceptions to them. To probe that flexibility, participants watched clips of a video game in which agents queue to gather water from wells and streams, then judged whether an agent who skipped the line acted acceptably, across dynamic and spatio-temporally varied scenes designed so answers could not be memorized from real life. Judgments tracked a generative understanding of the rule's underlying function, roughly whether everyone could cut in the same situation without things going badly, and could not be accounted for by either simple rule adherence or utility maximization.
↗ People can make social inferences about agents they never observed. From a small pile of cookie crumbs in a room, they can judge where someone entered and which goal they pursued, and from the position of two piles, whether one or two agents were present. The account proposed here is mental event reconstruction: observers infer the unseen behavior that would have produced the physical evidence, then read social information off the reconstructed events. A computational model embedding this idea in a Bayesian framework for action understanding matches human judgments with high quantitative accuracy across three experiments, and participants' explicitly reconstructed trajectories predict the entry-point and goal inferences made from the evidence alone.
↗ A stack of rocks encountered in the woods immediately suggests that someone was there. Developmental psychology has attributed this kind of agency detection to a sensitivity to violations of randomness; this work formalizes that idea as a computational model and tests it quantitatively. The model infers agency by estimating how likely a scene is to have arisen naturally, with human intuitive physics instantiated as a physics engine. Human judgments follow suit: agency detection tracks not just a scene's superficial visual properties but its underlying physical generative process, suggesting people use intuitive physics to decide when a scene calls for social reasoning at all.
↗ A chewed-up pencil suggests anxiety; an open door suggests someone left in a hurry. People readily attribute mental states to agents they never saw act. This work explains that ability as Bayesian event reconstruction: given a physical trace (a pile of cookie crumbs in a gridworld room), the model jointly infers the absent agent's goal, entry point, and trajectory under a generative model of utility-maximizing action plans. Across two experiments, its predictions quantitatively matched participants' judgments about which goal the agent pursued and which door it entered (r = 0.94), and even how many agents had been in the room (r = 0.78), while cue-based regressions trained directly on the human data fell far short. The results support the idea that Theory of Mind is a generative model of how mental states produce action, one that works even when the actions themselves must be reconstructed.
↗ How much can you infer about an object you cannot see, purely from its surroundings? The lift-the-flap task poses exactly this: identify a target hidden behind a flap in a natural image using only scene context. An online psychophysics experiment records where and when people click to reveal parts of the image, isolating the sampling patterns and visual features that predict accurate contextual inference. Those findings inform an extended recurrent attention model that attends to informative context regions and accumulates evidence across clicks, reaching human-level accuracy while reproducing human-like sampling behavior and learning interpretable contextual features.

trajectory

now

I'm working at Guidelight AI Standards! More details coming soon.

Astra Fellowship

I worked as a strategy fellow with Tom Davidson (Forethought) and Fabien Roger (Anthropic) on secretly loyal AI: the risk that an AI system could be deliberately trained to appear aligned with an institution's goals while covertly serving a different actor's interests. I focused on threat modeling and designing ML experiments that stress-test this scenario and support the broader research agenda.

Center for AI Policy and Centre for the Governance of AI

In 2025 I moved to DC to work on AI policy and governance: first at the Center for AI Policy, writing reports on AI agents, cybersecurity, and autonomous systems, then GovAI's DC fellowship, working on risks from internal AI deployment and metrics for tracking automated AI R&D. This was refreshing because the questions felt immediately important and impactful. I enjoyed communicating ideas and recommendations to people (tens of thousands read my reports in total), and it led to being invited as a panelist on a Georgetown × World Bank conference on "Making AI Work: What Firms and Workers Need."

activation steering

I worked with David Krueger's group testing activation steering methods. At the time it was unclear how well these techniques actually worked or where they broke down. We compared bottom-up approaches like function vectors against top-down ones like in-context vectors on shared in-context learning tasks, and found they have different strengths: in-context vectors are better at broad behavioral shifts, function vectors at tasks demanding precision.

LG AI Research

In 2023 I was a research engineer on multi-lingual LLMs under Honglak Lee, working with Lajanugen Logeswaran, Dongsub Shim, and Tolga Ergen. Synthetic data, pretraining, finetuning, evals. One thread I liked: leveraging language-invariant concepts so models can learn new languages more efficiently.

MIT

After college I joined Josh Tenenbaum's Computational Cognitive Science Lab, working closely with Sydney Levine on moral cognition: how people reason about rules, norms, and each other. We built models that tried to capture the structure of moral judgment, something I think matters for AI alignment too. Separately, I worked with Stephen Casper and Dylan Hadfield-Menell on red-teaming methods for systematically finding where language models fail.

early AI safety

Around 2020 I started paying attention to the surprising capabilities emerging in AI systems. I worked on one of OpenAI's early RLHF projects under Long Ouyang and Jeff Wu. It was my first hands-on experience with LLMs, and it got me scaling pilled. Then at Berkeley with Jacob Steinhardt and Dan Hendrycks, I worked on out-of-distribution detection, AI forecasting, and building evaluations for ML systems.

Yale

Studied CS and psychology. The summer before sophomore year, I worked with Gabriel Kreiman and Mengmi Zhang at Harvard/MIT Center for Brains, Minds, and Machines on visual cognition and context reasoning. It was my first research experience and I'm grateful they invested their time in a mostly floundering freshman. During school I worked in Julian Jara-Ettinger's lab, building computational models of social cognition.

pre-college

Mostly spent my time hanging out with friends and consuming a ton of content online. I was pretty directionless: no real sense of what I ultimately cared about or wanted to pursue, just chasing whatever felt good in the moment, not anchored to any ideals. But experiencing CTY and Canada/USA Mathcamp was special and invigorating. They were the first environments where I felt intellectually excited about ideas and the people around me.

rabbit holes

reading

The Gentle Romance: Stories of AI and humanity — Richard Ngo
The Night Circus — Erin Morgenstern
The Book of Five Rings — Miyamoto Musashi

listening

hip hop

I LAY DOWN MY LIFE FOR YOU JPEGMAFIA experimental / industrial

LP! (Offline) JPEGMAFIA experimental / glitch

jazz(y)

The Black Saint and the Sinner Lady Charles Mingus avant-garde

Hot Rats Frank Zappa jazz-rock

art pop

LUX Rosalía orchestral

La Vida Era Más Corta Milo j contemporary folk

Vanisher, Horizon Scraper Quadeca folktronica

electronic

I Love My Computer Ninajirachi house / dance / pop

Allbarone Baxter Dury synth pop / electropop

The Provocateur ADÉLA pop / dance / house

rock

Fetch Melt-Banana noise / experimental

Pain to Power Maruja post-punk / jazz

looking

Updating soon.

bookmarks

More coming soon.
Omar Chishti
Hoyeon Chang