Me: how does this system work and why is this part not working
AI: Great question, let me investigate.
AI: Gigantic response with a ten-step flowchart, a response explaining the load-bearing nature of the failing lines of code, and a prosposed fix.
Me: ok make that fix and test it
AI: Makes the change, tests it, says it’s good.
Me: still not working, this time this other part is broken
AI: Great catch, here is how we can fix that too.
Me: make the change
AI: makes the change and provides several paragraphs on how previous change didn’t cover what my brilliant insight pointed out but now everything will work.
Me: ok great looks like it’s working

“That doesn’t look like a failure” you say. I’m going to argue that it is. While the code now functions, what is left broken is my mental model of the system and the fix. The passivity of reading and occasionally prompting “continue” or “make the change” lulls me into a false sense of understanding. Learning is active - not passive.

I learned this lesson first in school. Before tests I would read the textbook to ensure my understanding was complete. This also lulled me into a false sense of security - my internal chain of thought being something like “yes I remember that part, and that, oh yeah and that”. But far more effective was presenting myself with a problem, a sheet of paper, and a pencil (nothing else) and attempting to solve it. Often I struggled and failed in ways I could not have anticipated. My blind spots revealed themselves and I was able to actually learn.

For me to retain my autonomy, my ability to make informed decisions, my mental models need to be accurate. Accurate mental models are unfortunately not the default outcome when using AI to learn about something. One habit that I have found helpful in systematically combating this is to notice when I’m in a situation where I want to be building a mental model but I’m being passive (not questioning, not feeling the stretch or the struggle that precedes understanding).

Once I’m aware that my quest for knowledge has been derailed, I have a strategy to pull myself back. It goes something like:

Me: my current mental model is that X fetches the data from Y then transforms it with Z and sends it to W. The issue is that Z needs to depend on the state of W and right now it doesn’t do that right because … correct me.

I pretend that a friend has just put me on the spot by asking what’s going on here. I answer to the best of my ability from whatever thoughts are floating around half-digested in my head (nothing else).

Then the model can correct me. By crystallizing my thoughts into words I can see how they break and that is really informative.

I allow the model to reject with evidence or accept my explanations. This is often super helpful because upon discovering the place where my explanation shattered I can clearly see for myself why (often I ask for what source code lines are relevant).

With this new evidence, I can update my mental model efficiently because the feedback is so sharply focused on the objective artifact of my present understanding.

I think this strategy works by combining aspects of paraphrasing (verbalizing in my own words what someone else has said) and the Feynman technique (the best way to learn is to teach). Usually I will have already read an model-generated explanation in previous conversation turns before I employ my strategy. And I don’t allow myself to scroll up and reread that explanation as I develop my own.

As model capabilities continue to improve, we need new techniques to ensure we can reliably construct accurate mental models of the systems we work with and what our agents are doing. This is how we can augment our agency without sacrificing our autonomy. Let’s have our cake and eat it too.

If this sounds like an interesting idea, I hope you give it a try! And let me know how it went.

When chatting with someone, and when I notice myself as I’m interacting with others, I have a sense for when authenticity is absent. I feel it when a conversation turns bland, when I get nervous enough that I try saying things I think the other person wants to hear. That’s why sycophancy can be so disgusting - inauthenticity.

On the flipside, my favorite conversations are those where both of us are authentic, sometimes vulnerable, sometimes ambitious, but always ourselves.

Authenticity has a power of attraction - people notice the effort it takes to break expectations and say something wacky and contrarian. I think a major supporting factor of Trump in the 2024 election was his authenticity. Hate him or love him, it’s hard to argue that he’s following a script. I think people can feel that. In products too, I love the spice of an authentic personality. Claude code comes to mind with its funny phrases (Moseying, Percolating, Smooshing, …).

And now, more than ever, authenticity is valuable because of the volume of inauthentic words/images/videos that come our way.

In the spirit of ‘vibe coding’, I’d like to propose the idea of ‘vibe testing’. ‘Vibe testing’ involves using AI to help plan, execute, and interpret an experiment. The experiment that inspired this idea emerged from a question I had about my sleep. Does my air purifier (a Levoit Core Mini-P) help me sleep better?

I have a Garmin Forerunner 255 Music watch that generates a sleep score (0-100) after each night’s sleep, taking into account duration, types of sleep (light, deep, REM), and heart rate variability (HRV). So I decided to conduct a randomized experiment to see if I could reject the null hypothesis (that my air purifier doesn’t improve my sleep). I chose 15 out of the next 30 nights uniformly at random without replacement and assigned them to be the nights with the air purifier on. I wrote those nights in red on my whiteboard and the other nights in black. Then I executed the experiement and each night wrote my sleep score in green underneath the night’s number. Here’s what it looked like when I finished:

Then, I used gemini to transcribe my whiteboard data collection to a CSV. To verify the transcription’s accuracy, I checked 10 random days, ensuring the CSV matched the whiteboard. Assuming gemini makes errors uniformly at random with respect to day (not quite true because my handwriting on some days is worse than others), this makes me confident enough for this experiment to trust the CSV’s accuracy.

At first I tried to have gemini conduct the data analysis itself, but this led to some hallucinations (see the Conclusion section of the pdf in this chat where gemini claimed 0.0225 > 0.05). So I vibe coded an R script to perform Welch’s t-test using a standard library, something more in-distribution for the model.

My average sleep score for nights with the purifier is 95.13, average without purifier is 90.73. The t-test yields a p-value of 0.04825, so this is evidence that my air purifier does help me sleep better! My current theory is that it is the white noise made by the fan, not the air filtration, that is helping me sleep. But this is a question for another day.

Zooming out, considering how easy AI made this experiment, I think I will vibe test more questions in the future. In the far future, I can imagine a smart home that can create and conduct randomized experiments like this one autonomously. With wearable biometric devices like smart watches and smart home environment controls, I could imagine something like the prompt “design, schedule, and execute an experiment to determine which temperature in the range 60-70 degrees Fahrenheit I sleep the best at” to actually succeed.

Taken even further, this idea becomes something like autoresearch (which I’m watching with keen interest - karpathy, quantum circuit optimization, and glite). Autoresearch on myself, with AI as the experimenter and myself as the subject and beneficiary of results.

This is an awesome possible future, and one I’d love to help build.

“Why am I out here rowing?? I’m falling behind where I could be if I was studying and learning and making cool stuff instead!”

Implicitly, I believed in a false dichotomy, a spectrum fallacy, an inaccurate mental map representing academics and athletics as engaged in a zero-sum game - to get better at one would require becoming worse at the other. I was wrong. And today, I hope to show you why.

Biology

To investigate how athletics could complement and assist with academics, I did two Deep Research queries, one with ChatGPT and one with Gemini.

In both reports, a protein named Brain-Derived Neurotrophic Factor (BDNF) is identified as a key factor responsible for the physiological benefits of exercise on brain functioning. Exercise, in particular cardio, has been shown to significantly increase BDNF synthesis in the brain. Interestingly, this effect may be more pronounced in men than women. BDNF supports the survival of existing neurons and encourages growth and differentiation of new neurons. Intuitively, I know that I feel better after working out. I also know that if I don’t exercise for a few days I become lethargic and generally unproductive.

Parkinson’s law

Parkinson’s law states that “work expands so as to fill the time available for its completion”. Regularly scheduled practices (and to some extent study methods like Pomodoro) act as unavoidable deadlines and chunks of time you know are off-limits in advance. This theory is in accordance with the observed phenomena that out of season athletes feel like they get less done! “If you want something done, ask a busy person” - Ben Franklin (?).

Right tool for the job

Gemini’s report included an interesting table explaining how one might utilize different types of exercise for different goals. For example, as we saw, cardio increases the production of BDNF. Therefore, you may want to go for a run before a study session during which your goals are to learn and ingest information into memory. Apparently (but this should be more thoroughly checked), strength training is an antidote to procrastination as it hightens executive function. And mind-body exercises like yoga or tai chi strengthen reasoning, attention, and problem-solving.

Not too long ago, I chatted over dinner with a friend about performance enhancing substances like caffeine. His quote struck me: “everything is a tradeoff”. You get to use the future now, but you’ll pay for it later. For an hour-long test with nothing else going on later that day, having the ability to use stimulants is awesome. It’s an option. And usually, the more options the better.

Just like being judicious with stimulants, knowing how certain foods will affect your body and energy levels is also a superpower. We talked about glycemic index (GI) and how he avoids carbs at dinnertime because they have a high GI.

All these ideas center around one meta-idea: use the right tool for the job. And the winds of evidence point us towards the realization that exercise is a tool, a multifaceted tool, that is useful in doing the job of being a college student.

Diminishing returns of exercise

While there is strong evidence that some exercise is important and beneficial, there must be a point beyond which exercise is harmful. Imagine someone working out eight hours a day - not much time to do other things. Unfortunately, I agree with my uncle who said collegiate athletics’ “competitiveness leads to a constant escalation of the amount of time spent on workouts. We’re not very good at controlling that escalation and need to be more conscious of that.”

In game theory terms, I think this represents the Prisoner’s Dilemma. At Williams, a D3 NESCAC school, we have an emphasis on being a ‘student-athlete’ where the student comes first. Meaning that academics should always have priority over athletics. However, I don’t think this is internalized enough by some of my teammates and possibly coaches. This is my attempt at representing the situation (where Athlete-Student means athletics is being prioritized over academics):

So if everyone had the same student-athlete time commitment for athletics, the student-athlete vision would be possible. And there are NESCAC rules for this like no mandatory practices out of season. However, in practice (haha), there are de-facto mandatory practices during the offseason. Everyone who wants to win has an incentive to push for more time practicing - quality time spent doing sport seems to be well-correlated with performance in sport. I think to solve this, coaches need an incentive for academic performance to balance the tug-of-war that currently appears one-sided.

The first known use of a tie in club colours was by members of Exeter College eight. In 1880, they took the ribbons off their boaters and tied them around their necks as a way to identify with their college.[9]

Ok, and what is this source? A wayback machine record of a silk tie designer’s article on the history of formal wear. It says:

In 1880, the rowing club at Oxford University’s Exeter College One men’s club, invented the first school tie by removing their ribbon hat bands from their boater hats and tying them, four-in-hand. When they ordered a set of ties, with the colours from their hatbands, they had created the modern school tie. School, club, and athletic ties appeared in abundance. Some schools had different ties for various grades, levels of achievement, and for graduates.

There is probably a grain of truth here, so I wanted to dig deeper. Deep research time. This returned more sources in support of ECBC’s invention of the first college or club tie. The tie as an element of fashion had existed, but ECBC rowers appear to have been the first to use a tie in their colors as a marker of identity. Further sources supporting this claim are an Oxford Student article about the Bear Inn (where there are ties on the ceiling), ECBC’s own website, and two more pages on the history of neckware.

Now I wonder how these sources are related. If all but one of them blindly copied the one, then my confidence in this idea should not be strong. But if these sources are independent in that they do not copy each other but place one of many trickled-down paths from history onto the Internet, then my confidene is strong. By default, I do believe this is true because of Occam’s razor.

Invalid or unsupported? What???!!! Google products are probably the most used products by any company anywhere in all of history. How on Earth do they not allow people to upload SVGs? I was dumbstruck upon seeing this for the first time because I remembered uploading SVGs successfully in the past. It turns out that SVGs are a security threat because they can contain javascript, so at some point in 2021 Google disallowed their use. I can’t find any official announcement about this, the closest thing I found is this support forum. Also, the SVG specification basically has a ton of functionality that almost never gets used (see this HN thread for some of the lore behind it). Ok, this is a challenge. I want to get my chart into google slides as a vector graphic so when we print out our poster everything is crisp. After some dead ends, I found a pipeline that works for me (I am running Ubuntu). There is probably a more optimal way to do this. Here we go:

1. Upload SVG to LibreOffice Writer
2. Export to PDF with “Lossless compression” on
3. Open the PDF in LibreOffice Draw and select everything with Ctrl-A
4. Open a new LibreOffice Impress presentation and paste into it
5. Save as .odp
6. Convert to .pptx with libreoffice --headless --convert-to pptx blogsvgtestpres.odp
7. Upload the .pptx to google drive
8. Open in google slides
9. Copy the figure and paste to wherever you want in google slides or google drawings!

This process is more steps than it should be. But this works (for now).

Edit 2026-06-30: Tikolu showed me their tool that is a much faster workaround than mine! Thanks!

On spring break (or “the vac” as it is called at Oxford), I traveled to Morocco. I flew into Marrakesh, met up with my American friend on a Fulbright, and we trained to Casablanca to stay with a friend I had made through competing at hackathons together. My Moroccan friend’s dad picked us up from the train station and gave us a tour of the city. Then - it was Ramadan so everyone was fasting - we broke fast with them and ate from an incredible spread of dishes. They offered us food and all we could say was “thank you” so many times it began to feel awkward. We were also the center of attention of the entire household which contributed to my unease.

During my receipt of a mind-boggling amount of hospitality (really - it was on another level), I continually expressed my gratitude “thank you”, “thanks, this is amazing”. But then, the father said “stop saying thank you, no need”. And I realized that our expressions of gratitude were making them uncomfortable - just as my receipt of their incredible hospitality was making me uncomfortable.

By this point of Iftar (the meal at sunset which breaks the day’s fast), we were running out of conversation topics so I proffered this to the table. In the states, my mom harped on us to always say thank you. But in the situation I found myself in, I was being told not to say thank you. Is this a cultural difference? Do people not say thank you in Morocco? People do say thank you in Morocco - “choukran”, but as a guest it seemed to be bad form to repeatedly thank them as we received each new dish / present / experience they graced us with. We laughed at this cultural difference and took cracks at explaining it. Could it be a reflection of America’s transactional culture? To accept something without the token expression, in the states, is to appear rude. Perhaps it satisfies the roleplay of the two-party transaction, signifying that you played a role - the receiver-receipiant transaction was not totally one-sided (a scary propsect - receiving without giving back???).

We had happened upon a difference in cultural norms. Then I remembered one salient example of saying vs not saying thank you on a world stage: the disastrous Zelensky - Trump meeting. Is the Ukranian social norm of saying thank you subtly different from the American one? Could an awareness of this have prevented the diplomatic tragedy of February 28, 2025? There is no “correct” culture, but for Zelensky to achieve his goals adopting the American thank you culture would have benefited him. And for my goal of pleasing our hosts, refraining from repeatedly thanking them would have benefitted me. However, we both may have found the frictions of breaking from old habits insurmoutable. I find probing differences like these to be fascinating as they also reveal glimpses of other cultures’ worldviews (accepting without feeling any obligation to repay).

I’m living in a house with eight friends next year. There are nine rooms total, so everyone gets a single. We had to figure out who gets what room. Two of my friends had done all the administrative work with the landlord, so we let them get first pick. That left seven of us. We each ranked our room preferences, (e.g. Room #8 most preferred, Room #4 2nd most preferred, …) and put them into a spreadsheet. I was taking computational game theory at the time with Tomasz Wąs, my tutor, and so brought the issue to him. How do we maximize everyone’s preferences and assign everyone to a room? Some rooms (like Room #8) were ranked first by multiple people, and some rooms (like Room #9) were at the bottom of most people’s rankings. But someone was going to get Room #9. What is the optimal way to construct a fair assignment of people to rooms based on their preferences?

Tomasz’s research focus is computational social choice and on what voting / distribution algorithms satisfy certain properties so he knew our options and showed me this website matchu.ai. This is a mechanism design problem. We could go with Random Serial Dictatorship (RSD), where we randomly select a person, they are assigned to their highest ranked room, then we remove that room from everyone else’s lists, then choose the next person. This algorithm has a fairness property (in that it treats everyone as equals), an efficiency property (no subgroups of people would be willing to swap rooms ex post), and truthfulness (no one could benefit from misreporting their rankings). Our other option was Probabilistic Serial Rule (PSR). PSR is a little more complicated. Borrowing from the matchu website:

It works as follows: suppose items are different types of pizzas. Each agent starts “eating” her top choice pizza at the same rate as every other agent. Once a pizza is consumed, agents move to their next preferred pizza; until all pizzas are eaten.

Then these probabilities can be broken down into a weighted sum of permutation matricies (with ones and zeros that assign people to rooms) and one of these can be drawn randomly but with respect to its weight. Ask your favorite LLM if you want more info.

PSR has advantages and disadvantages compared with RSD.

• PSR is ex ante envy-free. In PSR, the lottery you recieve (the distribution over what room you will get) is guarenteed to be your optimal lottery (you don’t envy (like more than your own) anyone else’s lottery). In RSD, you may envy the lottery someone else has (this is a subtle point but RSD remains strategy proof because if you replicate the other person’s rankings that may backfire on your lottery’s expected utility).
• PSR

  guarantees the prescribed probabilistic matching (lottery) is efficient, meaning that no other lottery exists that can strictly improve the outcome for some participants without making other participants worse off. So in PSR the lottery itself is Pareto efficient, whereas in RSD the lottery’s outcome is Pareto efficient.

• RSD, however, is strategy-proof, meaning agents have no incentive to lie about their preferences. In some scenarios under PSR, agents can improve their expected utility by reporting a preference ranking that is not their true preference ranking.
• PSR is difficult to explain (not a technical reason against it but certainly an implementational one).

Ok, back to the story. In the group chat, my friend said “let’s just choose rooms randomly”. Vehemently opposed to this, as we could do a lot better than random, I disagreed and said “why don’t we try to maximize everyone’s preferences jointly?”. However, this was my first mistake. Had I been more rational, many messages and misunderstandings could have been avoided. After arguments and a poll, I got the group to mark down their room preference rankings in a table in Google Docs. Discussing with Tomasz, we decided to go with Random Serial Dictatorship (where a random order is decided and people choose rooms in that order) because 1. RSD is easy to explain and easy to understand why it’s fair (the most important reason) and 2. it is strategy-proof (not that important but I got a bit carried away with making our mechanism as anti-adversarial as possible). I then proposed RSD to the group.

It turned out that my friend’s initial suggested mechanism (“let’s just choose rooms randomly”) was actually about deciding an order randomly (so exactly RSD)! Much argument was involved because I didn’t understand that what he was proposing was exactly what I was proposing. The blame for that lies on both of us (although probably a bit more on me). He should’ve been clearer about what he actually meant. I should have been clear-headed enough to recognize that very few people would actually propose to assign rooms to people at random and that drawing a pick order was a pretty common thing so this was what he actually meant when he said “choosing rooms randomly”.

At this point, we had conquered our misunderstanding and were ready to draw an order for rooms. This is when my interest in cryptography and mechanism design was piqued by a challenge. My friends and I were spread across the world - if someone randomly generated pick orders, how could we trust them to not fudge the results in their favor? I wanted to make the system uncheatable. I created this protocol:

1. Number each participant (assuming <= 10 participants)
2. Establish a date/time in the future to run the protocol
3. Extract the top headline of the New York Times website
4. Use SHA3-512 to get a hash from the headline
5. Go through the hash from the left and each participant number encountered adds them next in the draw order (e.g. given 7b8f723, 7 chooses then 8 then 2 (7 has already chosen)).
6. In the very unlikely event that the hash doesn’t contain enough of these distinct digits to make a full draw order, take the hash of the hash and continue with that.

Therefore, everyone could be very certain that none of us (unless they could control the NYT headline) was fudging the draw because everyone could verify for themselves the protocol result.

Explaining the rationale for this to the group, I justified it as a fun experiment and also that it eliminates any possible suspicion about the person doing the draw. When someone said “why don’t we just all hop on a zoom where someone uses a random spinner website” I replied with “they could have edited the website to favor themselves first”. Their response was that only I would be capable of doing that.

We hopped on a call and at this point in the debate I had resigned to using the spinner because I really didn’t think anyone was trying to cheat. It was clear my protocol was seen as unnecessary by some of the group (which in fairness it totally was). But then, like a gift from the gods, one of the leaders of our group said “why don’t we try Simon’s thing? And then nobody argued against that so I got my chance.

I used my protocol, generated the hash, looked through the hash and began to call out the pick order! This is when my second mistake struck. I was thinking to myself “I am calling out the room order, it would look bad if I’m too early, let’s hope I’m not too early, let’s hope I’m not too early”, making super duper super sure that everyone else thought this protocol was fair. I had veered off the course of rationality and was about to pay the price. Unconciously (I believe) I skipped over my own number the first time it appeared in the readout. I was also rushing as I read out the numbers (bad idea). It is hard to keep track of where you are in a hash. Then I finished (correctly noting the second time my number appeared), double checked, and realized my mistake. My third mistake was not saying anything there and in the moment - ‘Hey actually I messed up and I’m in front of you in the order’. I didn’t want to hurt my pride, to reveal that I made a mistake implementing my algorithm, when I had argued and pushed so hard for us to use it.

Putting on my behavioral pysch hat, I think I primed myself to not see my own number when reading the hash. I shot myself in the foot. Next year, my room will be worse than if I followed my algorithm properly, with a clear head, and without hurried biased thoughts.

This essay is the sixth essay of my Ethics of AI philosophy tutorial (under the tutelage of Benjamin Lang). In the essay, I attempt to answer two questions: (1) Could AI be conscious? and (2) If AI can be consious, how can we build a conscious AI system / reliably test for consiousness? During the tutorial, I found out that Eric Schwitzgebel, the author of the main articles I draw from, is Ben’s friend! And during reading, I learned that viceroy butterflies actually do taste bad to birds, so they aren’t Batesian mimics - now Ben is sending that to Eric. But with regards to content, I actually think that this essay is one of my best philosophy essays because I approached the topic in a logical way and anticipated and responded to counterarguments. I think I am a naturalist (everything is in physical reality - nothing is supernatural) and a functionalist (what makes something a mental state only depends on the larger system it is in). One critique from Ben that I like is from my use of the word “important” in “an important part of conscious communication” - is it a necessary condition, a sufficient condition, or just a trait which commonly co-occurs? I think I should have also made it clear that the second part of the essay - Evaluation for Evolution - is my attempt at a practical roadmap for creating machine consciousness.

The prospect of AI consciousness has implications for the ethics of our design and use of AI systems, and for the future of life in the universe. Therefore, determining if AI systems can be conscious is of high importance. And, if we believe they can be, how to make them conscious is the logical next question. This paper argues that (1) consciousness is an emergent property obtainable by AI and (2) we can make conscious AIs by creating better evaluation metrics of consciousness. Improving evaluation metrics will allow us to overcome the mimicry argument against robot consciousness through a process similar to evolution.

Consciousness is Emergent

Consciousness is familiar and puzzling. There is no fully agreed-upon definition as to what it is, but there are many theories for what it means to say something is consciousness. It could mean that the entity can sense and respond to its environment, or that it is self-conscious - aware that it is aware, or that there is some subjective “something that it is like” to be that entity (Van Gulick 2025). Or, it could mean that an entity is “behaviorally sophisticated” - defined as “capable of complex goal-seeking, complex communication, and complex cooperation” (Schwitzgebel 2024, 6). This section will argue that consciousness is an emergent property and can therefore be attributed to AI systems - machines that run on non-carbon substrates.

Can machines think? The famous question posed by Turing runs somewhat parallel to the concerns of this paper. One of the contrary views to his proposed imitation game is “The Argument from Consciousness”. During his rebuttal, Turing notes that for consciousness there is “something of a paradox connected with any attempt to localise it” (Turing 1950). In other words, consciousness appears to be an emergent property - just like life. Living things are physically composed of non-living things - DNA, RNA, proteins, and lipids that are not themselves alive. My brain is made up of billions of interconnected neurons, and few people would say that each individual neuron is conscious by itself. Because emergence requires multiple systems working together to produce a behavior, consciousness can therefore only be a property of some system over a specified time interval. This makes intuitive sense - there are periods of time where my brain is not conscious and frozen brain states or abstract (not running) formal programs do not appear to be conscious. With the emergence property of consciousness established, AI consciousness is much more plausible.

Imagine an artificial neuron, an a-neuron, that functions similarly to a normal human neuron except that it isn’t carbon based. It has artificial dendrites and an artificial axon, and action potentials that flow from each a-neuron like they do in normal neurons. Following a process similar to Schneider’s Chip Test, let’s replace one neuron in my brain - that only interfaces with other neurons - with an a-neuron with the same static action potential functions (but of course doesn’t respond to the brain’s chemical changes like a normal neuron would) (Schneider 2020, 451). Now, keep replacing neurons that are only connected with other neurons in my brain with a-neurons until all have been replaced. After this process is done, only the biological neurons interfacing with other types of cells in my body to receive signals and issue commands are left. Assuming a-neurons can perfectly represent normal neurons’ changes in action potential and firing profiles, immediately after this surgical operation occurs, my brain functions exactly as it did post-surgery. If my brain was conscious pre-operation it seems that in the moments following this operation my brain will continue to have the emergent property of consciousness because it will be functioning the same. This relies on the property of consciousness to be emergent, and that emergent phenomena rely solely on the functioning of the smaller parts that interact to create the wider system. If the smaller parts of a system interact exactly the same as they do in some other system, the emergent phenomena of the first system can be said to be occurring in the second. Therefore, a system of mostly a-neurons (an AI) can be conscious.

One obvious objection to this Chip-Swap operation’s conclusions is that the brain filled with mostly a-neurons will not function exactly the same as it did pre-operation because of the biochemical interactions that normally occur in a brain (neurotransmitters like dopamine and neuroplastic changes in a brain’s local structures). There are two ways to respond to this objection: (1) to argue that an AI system can also create these effects and (2) to argue that these effects are not important to consciousness. Each will be addressed.

First, consider the Chip-Swap+ operation (an improved version) where not only a-neurons replace regular neurons in my brain, but there is also a powerful machine that does the work of simulating the influence of neurotransmitters and neuroplasticity on the a-neurons in my new brain. Granted, this machine is a stretch of the imagination past the original thought experiment, but if we are able to do the original Chip-Swap experiment successfully it is plausible that we could do these frequent modifications as well. Therefore, an AI system could completely recreate the behavior of the brain and be understood as conscious when functioning.

Second, let us argue that neurotransmitters and neuroplasticity are not of crucial importance for consciousness - only electrical potentials are. Consciousness seems to be a property that can be assigned to a system on relatively short time intervals, as in an entity can have the property of consciousness for an interval of a few hours. This appears to show that neuroplasticity is not relevant to consciousness as significant structural changes in neurons and new cell growth take significantly more time to occur. Neurotransmitters function to excite or inhibit neurons (making them more or less likely to fire). This activity does deeply affect the functioning of the brain, however it is useful only as far as modulating the flow of electrical signals. The signals themselves are of much more importance to the emergent behavior of consciousness. The argument for the possible existence of AI consciousness has been made - the next step is how to get there.

Evaluation for Evolution

How can we develop conscious AI systems? The definition of consciousness this section focuses on is behavioral sophistication - complex goal-seeking, communication, and cooperation. Modern LLMs are designed to mimic humans, so if we are to follow the sensible mimicry argument against robot consciousness, we should be by default suspicious to assign consciousness to robots based on initial impressions of behavioral sophistication (Schwitzgebel 2024, 27). However, by the Copernican argument for alien consciousness, we should assume by default that behavioral sophistication implies consciousness for alien forms of life (Ibid., 2). This “violation” of the parity principle (the idea that we should apply the same types of behavioral or cognitive tests to robots as we would aliens to determine consciousness) is justified by prior information about the provenance of each type of system. In sketch, the argument roughly follows the idea that over an evolutionary time span, actually having a certain feature F (like long-term memory or behavioral sophistication or tasting nasty) is much more efficient than mimicking the superficial features associated with F (like the wing patterns of a monarch butterfly). However, when we have reason to believe a robot is designed to mimic human consciousness, inference to the best explanation suggests that the robot has the superficial features associated with human consciousness but is unlikely to have consciousness itself (Ibid., 5). For a likely non-conscious but conscious-mimicking system like today’s LLMs, could there be a way to transform them into something that we have confidence is conscious?

This section proposes that there does exist such a way, one which follows Schwitzgebel’s statement that - assuming functionalism (where a system that exactly replicates the functional states of a conscious brain is conscious) - “In the limit, the mimic could only ‘fool’ a god-like receiver by actually acquiring feature F” (Ibid., 30). In short, we must become gods of distinguishing consciousness from its superficial features. This method would rely on the increasing capability of an intended dupe (us) to distinguish between consciousness and an AI attempt at consciousness. In the current deep learning paradigm, such a capability to distinguish the performance of different systems is called an evaluation metric. If humans score high on a consciousness evaluation but AI systems do not, there is an argument to be made that the AI system lacks consciousness and is just engaging in mimicry. However, caution must be exercised to avoid evaluations degenerating into tests for humanness because such tests would presuppose that consciousness can only be found in humans.

Therefore, we must get better at asking the question: what does it scientifically mean to be conscious? Researchers can become forces of natural selection by creating better and better ways of measuring the behavioral differences between humans and AIs such that architectures and algorithms of AI systems are synthetically evolved to minimize those differences. In doing so, consciousness can be obtained by AIs through the evolution of structures that generate complex goal-seeking, communication, and cooperation. There are multiple avenues in which we can approach crafting better evaluations, most of which are currently being pursued.

To better evaluate an entity’s complex goal-seeking capabilities, we can develop evaluations (evals) that measure the reasoning capabilities of AI models. One property of human goal-seeking is that we create goals from a hierarchy of fundamental desires like Maslow’s hierarchy of needs. For example, the goal to finish a project at work could come from the desire for shelter or a sense of connection amongst colleagues. AI goal-seeking could similarly involve developing sub-goals during reasoning based on a main objective like “respond to the prompt as best as possible”. Better evaluation metrics for reasoning (so the agent improves at goal-seeking) can involve methods like multi turn reinforcement learning, where a reward model scores intermediate steps. These methods can be supercharged by verifiers - objective evaluation of model outputs like “does the model’s code compile”. Another type of reasoning eval that is good at measuring the difference between humans and AIs are those similar to the ARC-AGI-2 dataset that is specifically designed to demonstrate the subtle ways in which AI models are inferior to human reasoning.

To evaluate communication, the commonly applied post-training technique of reinforcement learning from human feedback (RLHF) seems to perform well in enabling coherent and understandable LLM outputs. However, an important part of conscious communication is one’s ability to know one doesn’t know something. Modern LLM hallucinations are clearly an obstacle that we should develop better checks and evals for so we can force AI systems to develop cognitive structures that enable closer-to-human conscious communication. It may turn out that for some forms of complex communication, temporally stable entities with long-term memories are required. In these types of interactions, humans may score much higher on good evals than the one-off chat sessions of today’s LLMs. For AI systems to reach human scores on these evals, they may be forced to develop a temporally stable presence and long term memory structure.

To better evaluate cooperation, we need evals that measure the performance of groups. It could turn out that control of a body is just better for certain types of cooperation. If this is the case, the best way for models to get better at these evaluations would be embodiment in something like a Tesla Optimus robot. Humans cooperating are usually good at sticking to their assigned task. LLMs, however, have trouble doing so. For example, during coding tasks they edit parts of the code they shouldn’t. An evaluation that measured how closely an assigned task was followed would help us evaluate cooperation.

Reconciling with Goodhart’s Law

Goodhart’s Law is an adage that states: “When a measure becomes a target, it ceases to be a good measure”. Translated in terms of the argument above, it seems to say that evaluations (measures of consciousness) that become targets (used to guide optimizations) cease to be good evaluations (of consciousness). In the argument above, many measures were proposed to become targets. Does this mean that they will all cease to be good measures of consciousness?

Individually, yes but collectively, no. As consciousness is an emergent phenomena, it cannot be described and measured precisely in the way that the property “at 25 degrees celsius” can be measured for something like water. None of the proposed evaluations were complete measures of consciousness, so every measure can be gamed in a way that score increases but apparent consciousness decreases. Optimizing each measure individually would fail. But attempting to optimize each measure jointly, and adding new evals that find where the current set fails, is a much more robust system. It avoids the pitfalls of Goodhart’s Law because the measure itself is dynamic (by the addition of new evals). This system, loosely defined, moves beyond a measure and becomes a framework to evolve AI consciousness.

References

• Schneider, Susan. 2020. “How to Catch an AI Zombie: Testing for Consciousness in Machines.” In Ethics of Artificial Intelligence, edited by S. Matthew Liao, 439–58. Oxford: Oxford University Press. https://doi.org/10.1093/oso/9780190905033.003.0016.
• Schwitzgebel, Eric, and Jeremy Pober. 2024. “The Copernican Argument for Alien Consciousness; The Mimicry Argument Against Robot Consciousness.” November 12. https://arxiv.org/abs/2412.00008.
• Searle, John R. 1980. “Minds, Brains, and Programs.” Behavioral and Brain Sciences 3 (3): 417–57. https://doi.org/10.1017/S0140525X00005756.
• Turing, Alan M. 1950. “Computing Machinery and Intelligence.” Mind 59 (236): 433–460. https://doi.org/10.1093/mind/LIX.236.433.
• Van Gulick, Robert. 2025. “Consciousness”. In The Stanford Encyclopedia of Philosophy, edited by Edward N. Zalta & Uri Nodelman. https://plato.stanford.edu/archives/spr2025/entries/consciousness/

I felt compelled to write this haiku after winning blades. The moment was surreal. Ecstatic jubilation. Screaming YES!!!! at Donny bridge, gripping Malachy in front of me in a bear hug and falling all the way back to embrace Will with my head facing the sky. “This is what victory feels like” - Oscar Tejura as we spin after boathouse island, gazing at the endless cheering crowds we are soon to join. Maybe after a rough IRAs last year and feelings of doubt, this beacon of light and achievement was startling. The last Exeter M1 crew to win blades was 26 years ago - 1999. Blades appear to last about 30 years on front quad’s walls (chalk and now less chemically powerful sealant). Mmmmmmmmmm what a day. If you have no idea what any of that meant: summer eights is a four day bumps race and bumping on all four days is an achievement called ‘blades’ where your names and college crest is chalked on a wall in your college, and you can buy an oar with everyone’s name and your college crest on the blade.