Autonomy at Stake

This essay is my second essay for my Ethics of AI philosophy tutorial (under the tutelage of Benjamin Lang). In the essay, I talk about nudges as mostly used in ways against your interests, but in conversation we talked about how we might allow a greater potency for nudges thought to be in your interests (like a youtube screen time popup).In retrospect, my arugment here is not strong because my definition of what counts as autonomy infringement is too general. We talked about medical ethics cases like Dax Cowart’s right to refuse treatment and new possibilities of LLMs like as a surrogate decision maker after being fine-tuned on things you’ve written.

We live our lives online to an increasing extent. Private companies like Google, Meta, and Amazon harvest vast and astonishingly detailed quantities of data from our usage of their platforms. From gleaning what we might be tempted to buy from milliseconds of lingering on an image to who we are leaning towards voting for from posts we like and users we follow, our preferences are laid bare by our actions - captured through the devices we use to interact in the modern world.

This paper will attempt to argue that the current trend of data collection, processing, and use has the potential to threaten our autonomy by enabling the manipulation of our behaviors through targeted nudges. Autonomy, the capacity to live one’s own life, is like an axiom in moral frameworks (Christman 2020). Therefore, potential violations of personal autonomy are important ethical issues to consider. Establishing criteria to determine whether some action is persuasion, which does not violate autonomy, or manipulation, which may violate autonomy, will help us understand how targeted advertising or “nudges” based on behavioral psychology could be problematic. The capacity of surveillance technologies to utilize data is expanding, as is the amount of data available. Microtargeted advertising and nudging, the craft of hyper-personalized messaging, is a method by which personal data can be used to sway behavior. Does it threaten our autonomy? And, even if it does, do the costs of allowing it to do so really outweigh the benefits we enjoy from platforms that give us enormous value?

Personal Autonomy

Personal autonomy is understood as the capacity “to live one’s life according to reasons and motives that are taken as one’s own and not the product of manipulative or distorting external forces, to be in this way independent” (Christman 2020). To precisely define what personal autonomy is, it is helpful to clarify the difference between it and freedom. Freedom concerns the ability to act effectively; autonomy concerns the “independence and authenticity of desires that move one to act” (Christman 2020). There is also a difference between ideal autonomy and “basic” autonomy, where basic autonomy is held by most normal adult humans and ideal autonomy is a state of full authenticity, free from any influence that may alter their desires from those of their “true” self. There are few people, if any, that could obtain ideal autonomy - where nothing external influences their desires and decisions. This paper will focus on autonomy in the sense of personal basic autonomy, so there is some reasonable extent to which external influences on an individual’s decisions may occur without violation of their personal autonomy. For example, a friend invited you to join the crew team so you try it out and end up doing it for years. Most people would not think that your friend violated your autonomy. However, consider this scenario: both your parents were athletes in college and they have been signing you up for sports teams since childhood. Now you are in college and you feel pressure from them to do something athletic. In this situation, where the reason for your decision to join the crew team has been influenced by external forces, is one where it is reasonable to state that your autonomy has been infringed upon.

Autonomy is important with regards to ethics. Kant thinks that the capacity to impose moral laws on ourselves is the “ultimate source of all moral value” (Christman 2020). John Stuart Mill claims that autonomy is “one of the elements of well-being” (Christman 2020). Therefore, if we are interested in our ethics we should take seriously any threat to personal autonomy.

Rational Persuasion vs. Manipulation

To accomplish the goal of influencing someone to do something, there are two broad courses of action available: persuade them without the use of force or force them into complying either physically or with threats (coercion). Within the non-coercion category, rational persuasion and manipulation exist. Rational persuasion is the use of logic, facts, and reasoning to persuade someone to do something. Manipulation is harder to define. Three main categories of manipulation have been put forward: “those that characterize manipulation as an influence that bypasses reason; those that treat it as a form of trickery, and those that treat it as a form of pressure” (Noggle 2022). We will focus mainly on the “bypassing reason” category.

This paper is concerned with the impact these different forms of influence have on personal autonomy. Rational persuasion, in general, respects personal autonomy as it relies on one to think rationally through the argument that has been presented and freely come to a conclusion. Manipulation, on the other hand, influences decision making through non-rational means, and works against autonomy by definition. Some cases exist where intuition points toward the conclusion that manipulation can actually increase autonomy, like the case where a teacher manipulates a student into taking a class that ends up giving her more career options. While this form of manipulation does increase autonomy overall, it decreases it in the short term. For the purposes of this argument, we will focus on the short-term direct effects of manipulation because targeted advertising and nudges are usually attempting to influence people over a relatively short time. The growing power, unlocked by more data and better algorithms, to predict and manipulate human behavior is a threat to our autonomy.

The Power of Data

“Knowledge is Power” - Francis Bacon. And what is knowledge? Data. As “Data Vultures” points out, user data is harvested from us just about all the time in our modern daily lives (Veliz 2020). Shoshana Zuboff analyzes the competitive pressures of the “surveillance capitalists” and finds that their means of production are increasingly becoming “means of behavioral modification” (Zuboff 2019). One illustrative example of the value of user data and opportunities to influence user behavior is that Google reportedly pays Apple $18 billion a year so that it will remain the default search engine on Apple devices.

Technological advances have accelerated the amount of data ingested by and the capabilities of data processing techniques. OpenAI trains on free-tier user conversations. A company named Bee.computer has made a wearable microphone that records your conversations throughout the day and makes them searchable and generates insights for you. Simon Willinson, a prolific AI blogger, wrote that it would only cost $1.68 total to generate detailed text descriptions for the 68,000 images in his personal photo library using gemini-1.5-flash. But these advances have potential downsides. Surveillance has the potential to become increasingly pervasive and capable, enabling authoritarian states to oppress and censor more people. The use of data to control behavior for political aims has also occurred in the United States, above and beyond normal advertisements.

U.S. elections have massive stakes, so any advantage or edge is highly valuable. Cambridge Analytica, a British political consulting firm, harvested data from millions of Facebook profiles to analytically assist the 2016 presidential campaigns. They targeted potential voters with personalized messages, playing a crucial role in a close election. Did those personalized messages infringe on their recipients’ autonomy?

When a Nudge Infringes on Autonomy

Nudges are typically defined as subtle, non-coercive influences into people’s decision-making (Noggle 2022). For this paper, nudges will also include messaging done for the goal of influencing behavior through the principles of behavioral psychology. So, nudges include advertising that utilizes emotions. A graphic political video of violent migrants will be more effective at instilling a xenophobic mindset than a graph showing an upward trend of criminal cases.

When do nudges become manipulations? There is a significant lack of agreement on this topic. Based on the definitions presented in this essay, nudges are subtle and rely on psychology, thereby taking advantage of the non-rational ways in which our brains work. Therefore, nudges are manipulations because they sidestep our reasoning capabilities through the backdoor of emotional or other heuristic-based methods of cognition that evolution has left us with. Then the question becomes: when are manipulations problematic? One plausible answer seems to require some measure of the extent to which the manipulation altered your personal autonomy - a comparison to what your “true self” would have decided. If your behavior after being subjected to the manipulation is exactly the same as what it would have been without being subjected to the manipulation, it seems like the manipulation is not problematic. On the other hand, if your behavior under manipulation would be completely different from your behavior without manipulation, your personal autonomy seems to have been infringed upon. Each choice we face is under different circumstances, so it is hard to measure what an individual’s behavior would have been in the counterfactual scenario. It may be more useful to think about the “potency” of one manipulation over a population rather than the power of a manipulation over a single individual. Measuring the potency of a manipulation to affect personal autonomy in a population can be accomplished through a randomized controlled trial. Then it would be up to society to determine which levels of influence are acceptable and which are not. One note is that in the case of microtargeted manipulations these populations may actually be quite small and could even reduce back to the individual case. Targeted advertising is shown to be more effective than non-targeted advertising, and interestingly popups warning against potential microtargeting had “no meaningful impact on persuasiveness” (Carrella 2025). This finding shows that people usually undercorrect for their cognitive shortcomings. Therefore, if we were to regulate manipulations based on potency, perhaps we should choose to set stricter standards than we think are necessary.

The argument above attempts to make the case that our autonomy is at stake in the world of modern surveillance. Data collected by private companies is used, in some cases legally and some illegally, to craft potent personalized nudges that are manipulative and will further infringe on our personal autonomy as more data and algorithms make them more effective - unless we develop and enforce regulations to protect ourselves from exploitable features/bugs of our hardware.

Do the Benefits Outweigh the Costs?

One criticism of the argument that our personal autonomy is at stake in today’s world is that people freely consent to the major platforms’ Terms and Conditions. Another criticism claims the extent to which personal autonomy is threatened by manipulative nudges is actually quite small in the real world, and therefore is not a cause for concern. Both of these criticisms rely on the plausible argument that we benefit more from major tech platforms than they cost us, in the sense of well-being. Proponents of these views might voice the following thought experiment: choose between living in today’s world OR living in a world with no Google, no Amazon, no Instagram but with no microtargeted ads or nudges. Most people would choose today’s world, reasoning that the benefits of convenience like Amazon knowing your credit card details, deep insights like Google Maps’ Timeline feature, and personal connections like Facebook’s People You May Know section outweigh the costs they may have on our autonomy. We think the pros outweigh the cons.

There are issues with this line of reasoning. First, there is strong evidence to suggest that tech platforms violate our autonomy in getting us to agree to the Terms. As Veliz puts it, “there is no room for negotiating … you are being bullied” (Veliz 2020). The Terms seem to meet the conditions for all three of the definitions of manipulation. They bypass reason because most humans do not care to read and try to understand such an intimidating wall of text (and so we are not informed either). They are a form of trickery - most have clauses allowing them to change their terms at any time. They are also a form of pressure - if all of your friends use one platform, like Instagram, you are peer-pressured into using it because to not use it would jeopardize your place in the group.

Do the benefits actually outweigh the costs? What are the benefits? And what are the costs? There are long-term and short term views of both, further broken down into the individual and societal perspective. This response will take the stance that over the long term, the potential risks outweigh the benefits for society and therefore everyone. The benefits are mostly clear: convenience, access to valuable platforms. The costs are less clear, but there is potentially more at stake. The story of Cambridge Analytica is evidence that microtargeted messaging has already been weaponized successfully to steer people’s behavior. One could argue that it was a contributing factor to the democratic backsliding we are seeing in America today. In its section on Harm in the Ethics of Manipulation, the Stanford Encyclopedia of Philosophy says “Systematic political manipulation may weaken democratic institutions and perhaps even lead to tyranny” (Noggle 2022). As rules and norms appear to regulate competition for control over the American government less and less, it is reasonable to assume that microtargetted nudges and messaging will become more widespread.

Finally, the thought experiment proposed above is a false dichotomy. We can have the wonderfully useful platforms of today without many of the issues raised in this essay. Government involvement is the way forward because the incentives of market competition are currently leading us down a dangerous road. Perhaps metrics like “potency levels” will regulate attempts to manipulate us (which are problematic only when it exceeds a reasonable extent). This path is an uphill battle against very strong interests but we have some recent hope, in the antitrust suit against Meta, that we are moving in a positive direction.

References

• Carrella, F., A. Simchon, M. Edwards, and S. Lewandowsky. 2025. “Warning People That They Are Being Microtargeted Fails to Eliminate Persuasive Advantage.” Communications Psychology 3 (1): 15. https://doi.org/10.1038/s44271-025-00188-8 .
• Christman, John. 2020. “Autonomy in Moral and Political Philosophy.” In The Stanford Encyclopedia of Philosophy, edited by Edward N. Zalta. Fall 2020 ed. https://plato.stanford.edu/archives/fall2020/entries/autonomy-moral/ .
• Gaukroger, Cressida. 2020. “Privacy and the Importance of Getting Away with It.” In Journal of Moral Philosophy 17 (4): 416-439. https://doi.org/10.1163/17455243-20202987
• Marmor, Andre. 2015. “What Is the Right to Privacy?” USC Law Legal Studies Paper No. 14-13. University of Southern California Gould School of Law. https://ssrn.com/abstract=2422380 .
• Noggle, Robert. 2022. “The Ethics of Manipulation.” In The Stanford Encyclopedia of Philosophy, edited by Edward N. Zalta. Summer 2022 ed. https://plato.stanford.edu/archives/sum2022/entries/ethics-manipulation/ .
• Veliz, Carissa. 2020. “Data Vultures.” In Privacy is Power: Why and How You Should Take Back Control of Your Data. London: Bantam Press.
• Zuboff, Shoshana. 2019. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. New York: PublicAffairs.