We can do cultural engineering openly, ethically, and sustainably
This is an open letter to the researchers at the University of Zurich: the work you're doing is extremely important, but there is a better way
There is a subreddit called r/ChangeMyView where people go to understand each other’s beliefs. When someone’s successfully changes their mind or has a perspective shift, they celebrate & mark that as a “delta”. On April 26th, this community found out that researchers from the University of Zurich had been testing how LLM bots can be used to change real people’s minds. They had been doing this for months, they found out now because the research team was disclosing it before publication.
Across the internet, there are two categories of response to this incident, like in these two HackerNews comments:
❌ “this is terrible, and unethical, and should not have happened”
✅ “it’s ALREADY happening. At least these people share their results with us”
Last November, we saw this same dynamic happen on Blue Sky, when ML researchers made a dataset of 1 million posts, and the community backlash was so severe that it made HuggingFace delete the dataset:
❌ “I didn’t consent to be part of this dataset, this is terrible”
✅ “but you already *are*, you’re being studied, and targeted in live social engineering experiments. We, the open research community, are trying to illuminate this!”
This was the reasoning the University of Zurich team gave for performing this research, that there is an “unacceptable gap in the body of knowledge”. The gap they’re referring to is between those who study & perform experiments en-masse on human populations in secret, and those who do so openly & publish their results.
The researchers argue that psychological manipulation of OPs on this sub is justified because the lack of existing field experiments constitutes an unacceptable gap in the body of knowledge.
I agree that this gap is unacceptable, and in fact dangerous. Imagine if you discovered one day that geoengineering is not only possible, but a bunch of private companies are actively doing it, with no oversight, and no coordination. Meanwhile, the general public is completely unaware to what degree their landscape is engineered, by whom, and for what purpose.
This is the current state of culture science. I think it has enormous potential for solving the existential crises that humanity faces today. But it may also be our downfall if we can’t transition away from the current wild west of narrative warfare into something like bottom up & decentralized cooperation.
No way to study culture without changing it
This is a foundational principle of our field: there is no such thing as a neutral, objective researcher. You cannot study culture without changing it.
This is obvious in the case of the University of Zurich experiment. They found something true about the current memetic landscape: you can use bot swarms to change people’s minds.
In a first field experiment on AI-driven persuasion, we demonstrate that LLMs can be highly persuasive in real-world contexts, surpassing all previously known benchmarks of human persuasiveness. […] our experiment confirms the challenge of distinguishing human- from AI-generated content. Throughout our intervention, users of r/ChangeMyView never raised concerns that AI might have generated the comments posted by our accounts.
Source (draft of the paper they disclosed to the subreddit): https://regmedia.co.uk/2025/04/29/supplied_can_ai_change_your_view.pdf
Do you see how publishing this changes the landscape? The next time someone repeats this experiment, anywhere on the internet, the conditions are different. People have less trust in these types of interactions. Their awareness has changed.
Is the change that this study caused in our society good or bad? We can debate that, but the point is that, studying culture ALWAYS changes it, for the same reason that observing quantum particles changes them (not because it’s magic, but because “observing” == “interacting with”).
Even if the subject is never aware of the study, it still changes culture, because you, the researcher, belong to a specific culture. You have a relationship to those being studied. You gained predictive power upon studying them. This gives you the ability to influence, and steer. You are not neutral.
The fact that studying the culture changes it is NOT a problem to be fixed: it’s a property of the reality we are studying. Michael Smith has an excellent thread about this being a key bottleneck, and how we must solve the problem of embracing this necessary subjectivity while still being rigorous.
Active participation with the subject is necessary
I propose a simple rule of open memetics: transparency accelerates good faith cultural study, and hinders nefarious engineering attempts.
If people actually want to find out the answers to your research questions, they will help you get those answers. When the Washington Post wanted to research how much time people were spending on TikTok, and any patterns in their watch history, they just asked people to give them that data, and they got like 1000 people overnight.
The main problem with the Zurich study is the deception, and how the study shifts the culture in a negative direction, in a way that makes it hard for the next person to do similar kind of work. It doesn’t scale.
My first ever live culture experiment was “Anatomy of an Internet Argument”. I got into internet arguments with a goal of turning angry interactions into positive, wholesome ones. I discovered a formula for this, and it is reproducible.
I wrote about the techniques I discovered because I wanted people to try & reproduce it, or falsify it. This is a science experiment without deception. You genuinely want to change people’s minds, because you believe your position is correct. I needed to find “sustainable methods” for mass adoption, something that still works even if everyone did it. But that also still had benefit even if only YOU did it (to avoid the cold start problem of mass behavior change).
This method requires that you understand the other guy before you can change their mind. This works great because you always win something useful from the interaction even if you fail to change them. Sometimes you change your mind from what you learn. This is what makes me a subject in my own experiment, not an objective researcher. I have changed my mind on a lot of topics playing this game.
Doing this work is how I encountered my first “self fulfilling narrative”: people assume that internet arguments are toxic, so they don’t put in effort, which confirms their belief that internet arguments suck. That was my hypothesis. How do we prove that? We can seed a new narrative, and see if it works. If we’re right, then merely changing the belief will change the outcome. If we’re wrong, then it will not give us the outcome we expect (that counts as a failed prediction, which invites us to update our model).
Turns out: I was right. The more I did these internet argument case studies, the more perception began to shift. I started getting messages like this, where people try the methods & report back when it works. Which itself further spread the method, and further shifted the perception in the direction I envisioned.
This is propaganda. This is the science of shifting perception. But it’s GOOD propaganda, it’s positive, it’s transparent. If my methods didn’t work, they wouldn’t spread. It becomes bad propaganda when you start hiding things (like paying people to say good things about it even if it’s not true). Transparency accelerates what is good, and hinders what is bad.
It got even more fun when people in my network started changing their behavior because they knew I was on the lookout for good examples of wholesome internet arguments:
So, yes, the subjects being aware of the study WILL change their behavior, but that’s a feature, not a bug.
What does open memetics / open cultural engineering look like?
I think active participation is currently the bottleneck. This is itself blocked on explaining to people what open memetics is & how it works. This is what I’ve been working on for the past year and I consider it “cultural infrastructure” work. My goal is to pave the memetic landscape such that we can do this maintenance openly. I do this by:
doing small experiments that I expect will shift culture in a certain direction
writing about how it went & asking others to reproduce
On one level, I do this with my IRL friend group. We’ve been prototyping ways to create a culture of “drop by unannounced”. I’d categorize this as under the umbrella of “fixing the loneliness epidemic”. If we find something that works, we can write it up. Others can then “peer review” this by trying it themselves & reporting back.
This is similar to what
does, documenting a case study of how, having coffee regularly in front of their house fostered a community “third space” in their neighborhood. Or Renee talking about posting on her local subreddit to invite anyone new to town & looking for friends to meetup. I have seen this reproduced in my local community, it works!
On another scale I try to do it on the internet. Anatomy of an Internet Argument is a protocol for how to engage with people outside of your tribe respectfully, and how to write about them in a way that feels good for both of you. In doing so I get to study the evolving “memome”1 of different cultures. Sometimes I try to introduce a meme that I think is good & useful.
The easiest way to introduce a meme is to demonstrate successful use of it by someone they trust. If you are a member of that tribe, you can be patient zero. If you are not, you need to find someone who can be. For most of the things I want to introduce, there’s usually a minority who already have that meme2, and all I need to do is bring attention to them.
This is what I was doing when I wrote:
In the first one, I wanted blue tribe to shift from identifying as “pro regulation” to identifying as “pro making things better, using whatever tools work”. I saw this pivotal moment where a democrat won with a specific anti-regulation position, and the surrounding debate/confusion around that. If the “Open Memetics Institute” had existed at the time, we could have launched a coordinated campaign to amplify this3.
A week after I wrote that,
independently wrote the same thing in “Stop telling constituents they're wrong”, perhaps sensing the same pivotal cultural moment. This kind of open, decentralized coordinated campaign is more likely to be successful if we have a solid theoratical memetics foundation, AND empirical knowledge about the memome we’re operating on. Like, what you need to understand here is that people will resist this change if it erodes their sense of group cohesion. What you need to do is create new narratives4 that take on that function of group cohesion, as scaffolding, before you can dismantle the other thing. As in, being “pro regulation” is how you signal you’re in the blue tribe, so any attempt to nudge that will face resistance, until that signalling function is fulfilled another way.Open memetics is self reflection for the human superorganism
I used to think that buddhism / meditation was about sitting still & stopping all your thoughts. Now I understand it to be more of a metacognition practice: it’s about figuring out where your thoughts come from. Just by doing this, the nature of the thoughts change.
Example: I used to impulsively reach for chocolate for stress relief. This was bad because I’d do it even if I was full, which feels bad, more stress, downward spiral. Fighting against the impulse also didn’t work, because I’d be too strict and end up relapsing. What worked was growing discernment through observing my thoughts: I can tell why I am reaching for the chocolate, I can identify if it’s a source of stress, and just move away from that, or if I genuinely want a piece of chocolate, and in which case, just have it.
Meditation is the process of a mind putting its attention on itself.
Memetics is the process of the collective mind putting its attention on itself.
Meditation can be dangerous. The things that you reveal about yourself may be destabilizing, you may not be ready for them. The same is true for memetics. Whatever we shine attention on changes. It’s a very powerful effect. And it’s not always possible to control the resulting feedback loops.
I think the type of insighta that were unlocked for me with meditation, about my own mind, is what memetics can unlock for us collectively. What does enlightenment look like in a collective mind?
A meme is like a gene, so a memome is like a genome, it’s a set of beliefs & behavioral patterns that are consistently found together in a specific culture.
This is validation that the meme you’re trying to introduce is actually useful, it will likely have emerged naturally, it just may have trouble spreading due to the hostile memetic landscape we live in.
The beautiful thing about open coordinated campaigns is you can see them coming. If you think it’s good, you can join in and amplify. If it’s bad, you can oppose it. If you are a dark actor with malicious intentions, you can try to subvert it (revealing your presence). Win, win, win.
Historically, the job title of those who do this work, of noticing the effect of narratives on their community, and creating new ones or maintaining existing ones was a priest. We have others fulfilling these roles now (youtubers like Hank Green for example), we just don’t think of them as “priests” & their audience as a congregation.
Yes!! People are only so easily psyop’d because they haven’t ever heard of memetic engineering.
They thus haven’t developed antibodies to attack vectors which they aren’t aware exists & don’t even know to watch out for
Once I exited my first “cult” & gained awareness of this dimension it was indescribably easier to spot future currents & dodge them or flow in & out.
🤯
This is the future