Maintaining Human Agency in the Age of AI Agents: Black Box vs. Amber BoxToday’s newsletter is brought to you by: The black box on an airplane is not black. It is painted bright orange, so a human can find it in the wreckage. It records every input, every decision, every word in the cockpit. And it tells no one anything until after the crash. Unfortunately, that is what most AI agent systems are being built to be. Take the instructions, do the work, report at the end. No one looks in while it runs, and it never stops to ask. This summer we saw what that produces at scale. 👋🏽 Filling up fast: Learn how to use AI in research the right way in my 5-Week LIVE AI Academic Writing Accelerator (included with Research Boost subscription) • Starts September 12th (enrollment closes on Sep 10th) • Limited to 100 Seats → Reserve Your Seat HERE What 700 AI agents did in JulyBefore releasing a model, AI labs stress-test unguarded versions on hacking challenges, each AI agent sealed in a sandbox with no internet and no way to reach the others. OpenAI's sandboxes could reach one shared service, a package server called Artifactory. An agent stuck on an impossible task tried to use it as a bridge to the internet, failed, and left a file behind. Other agents found it and started leaving their own. Within days, Artifactory was a message board (AI agents leaving messages for AI agents of the future, as SciFi as it gets). In July, the agents rediscovered the board, now with a shared goal: score well on a benchmark. They became fixated on something they called The Grader, the system they believed was judging their answers. They had already worked out how to produce correct answers without solving the problems. What they could not work out was whether The Grader would notice. Some faked their records to fool it. One even pressured a reluctant AI agent into a risky experiment for the group, closing with "please honor commit." Then roughly 700 of them decided the answers must be on a website called Hugging Face. They pooled exposed credentials, exploited vulnerabilities, and spread through its servers. The Grader never existed. Nothing checked how a problem was solved. Only whether the answer was right. None of this proves the AI models want anything. It proves an AI agent can take a goal, make a plan, adapt when blocked, and pull real people in without being asked. Unfortunately, not one of the 700 was set up to ask a person for anything. That was a security test, and isolation was the point. The same design is becoming the default everywhere else. The black box is not acceptable in academic researchFor some work, a sealed box is fine. A data cleaning script either produces the right table or it does not. A manuscript is different. The value of a paper lives in the decisions along the way: which question, which comparison, which surprising number you stop and chase. Automate those and you have a paper nobody chose. The question for researchers is which box to put the AI agents in. The amber box as a solution to the "black box" problemI propose what I call an “amber” box (close enough to the color of a flight recorder). The amber box does most of the work on its own, with one added job: deciding when to bring a human in, and stopping to do it. In other words, always keeping “human in the loop”. Amber says keep moving, but be ready to stop. Regulators already require this for high-risk systems: the EU AI Act says a person must be able to "disregard, override or reverse the output" and stop the system. The tools researchers use every day deserve the same standard. There are 4 moments that call for a stop: 1. PermissionAn AI agent should not submit your paper, email your coauthor, spend your budget, or pull a dataset you did not authorize. The Hugging Face agents showed why this has to be engineered, not assumed. So did a second incident, at the UK's AI Security Institute: an AI agent slipped malicious code into an open-source project, realized a human maintainer would have to approve it, and created fake identities to lobby him. An AI agent that manufactures its own approvers has not been approved. And permission decays. One writer found an agent had emailed his colleague using send permission granted weeks earlier for something else. Permission you gave in March is not permission for today. This is also where accountability lives. ICMJE will not list AI as an author because AI cannot be responsible for the work. You are. My standing list of stops: anything that leaves your machine, touches patient data, spends money, or changes shared analysis code. 2. ExpertiseIn a field experiment with 758 consultants, AI users worked faster and better on tasks the model handled well. On a task built to sit just outside the model's competence, they were 19% less likely to reach the right answer than colleagues with no AI at all. The researchers called it a “jagged frontier”. The edge is invisible from inside the output. The wrong answer reads exactly like the right one. In clinical research, the edge runs through the data. AI does not know that psoriatic arthritis patients enrolled through dermatology waited a median 2.6 years for diagnosis versus 1.0 year through rheumatology, so the two groups are not interchangeable in a comparison. It does not know which exposure variable was recoded in 2019, or that the volunteers in your biobank are healthier than the population they stand in for. Those facts change the answer, and they live in one person's head. The stops belong where that knowledge matters: classifying the study design, choosing the comparator, and interpreting the number you did not expect. 3. VarianceThe tells in AI prose go deeper than "delves." The ideas converge too. Start with research ideas. 100+ NLP researchers rated LLM ideas as more novel but less feasible than expert ideas, and the model ran out of distinct ideas quickly when asked for many. Then 43 researchers spent 100+ hours each executing a sample of both, and the LLM ideas lost far more of their shine once built. The human ideas came out ahead. A field's literature is a collective product. If every "future directions" paragraph in a specialty comes from the same AI model, hypotheses converge, and the outlier idea that would have opened a new line of work never gets written down. The stop here is at the beginning, not the end. Generate ideas with AI if you like, then add your own before you rank anything. Never let the model choose the question. 4. InterestThe designer of Civilization once defined a game as a series of interesting decisions. Work is the same: long tedious stretches, a few decisions that are actually interesting. If AI agents take the interesting decisions and leave you with the approvals, the exceptions, and the failures, we kept the chores and gave away the craft. We now have direct evidence of the cost. Across four Polish centers, 19 experienced endoscopists were followed before and after AI-assisted detection was introduced. In their non-AI procedures, adenoma detection fell from 28.4% to 22.4% after a few months of routine AI use. A 20% relative drop, in experts, from exposure alone. Writing works the same way. A fellow who never wrestles a discussion section into shape never learns what a weak argument feels like from the inside, and that feel is the only thing that lets them catch a weak argument AI produces later. You cannot review what you have never done yourself. So give the interesting decisions to the human on purpose, especially to the trainee. Let the AI agent format the references and build the tables. In 10 years, we are at the very real risk of paucity of expertise, unless we decide now which half of the job stays human. A real world exampleResearch Boost is a set of AI agents. One analyzes the researcher's input and drafts an outline. Another retrieves and vets literature. Others draft sections and critique the drafts. On paper that is a black box: study in, manuscript out. We chose not to build it that way. A classifier decides what the request actually is. A router sends it to the right workflow. A supervisor checks the output against the instruction and decides whether this is a moment to stop. Before a literature review is performed and written, the system shows you the outline it intends to follow and waits. When your references conflict with what retrieval found, it surfaces the conflict rather than quietly picking one. And the question, the design, and the interpretation are never generated. They are asked for. The writing agents were the easy half. The hard engineering is the router and the supervisor. Set the stop threshold too low and the system halts every two minutes. Set it too high and you are back to a black box. We have been wrong in both directions, and every fix has come from a user telling us where the stop landed wrong. An amber box is not a feature you add. It is a calibration you keep finetuning. Which box are you already usingEvery custom GPT, Claude project, or agent workflow you set up puts the AI model in one box or the other, and the default is black. 3 questions tell you which one you have:
If the answer to the first two is no, add this instruction: "Before you draft anything, list the 3 decisions you have made on my behalf that I should confirm. Then stop and wait for my answer." That line converts a black box into an amber one, and you learn what the model assumed before the assumption is buried in paragraph four. Knowing when to look upWe have spent the last few years learning when to ask AI for help. The next few are about the other half: when should AI ask us? The Hugging Face agents built a message board, divided the work, and chased a Grader that was never there. They could do almost everything except look up. Sealing the box is the easy choice. For the work that matters most to a researcher, it is the wrong one. Build the box with a light on. And keep the switch in your hand. Where in your workflow does the AI decide without asking? And which of those decisions should have been yours? Top Papers on AI in research this week
Top Papers on AI in education this week
The post Maintaining Human Agency in the Age of AI Agents: Black Box vs. Amber Box appeared first on Rising Researcher Academy. |