Can we use Twitter Radar to empirically study & predict culture?
It's pretty cool, but it doesn't have the granularity I hoped for
Imagine if you could semantically search internet discourse and graph it over time. You’d be able to watch how ideas spread across society & where they get stuck. You could take a mainstream idea and chart its course back to wherever blog or book it came from.
What patterns could you find with such a system? Could you engineer ideas that spread more effectively? Could you render propaganda ineffective if you could see the origin of every idea?
This is what I thought “Twitter Radar” was. I paid for premium & gave it a shot. It’s extremely limited compared to this grand vision, but I want to document it here as the first publicly available tool I’m aware of that of that gives the general public this kind of information.
How it works
It’s keyword search, so you can’t search for “gender discourse” generally, but you can search for specific things. Here’s a graph of “your body, my choice”, which captures the heated gender discussions that went viral after the US elections on Nov 6, 2024.
In addition to the graph, you get three tabs: “latest”, “top”, “trends”. The first two are the same thing you get with regular twitter search.
“Trends” is the most useful. It summarizes the topics of discussion, but it doesn’t work for a lot of the queries I’ve tried. It returns nothing for “your body, my choice” even though it had half a million posts.
It returns nothing for “isreal” (which has 6 million posts). It returns 1 trend for “Palestine” with 2 million posts.
These “trends” are not paywalled. They appear to be the same as the auto-generated topics that show up in the “explore / trending” page. Here’s the one for Palestine at the time of writing.
The data it’s searching is always the last 7 days, there’s no way to search for older data, or limit it to the last 24 hours.
How they’re marketing it
The default suggested query is “$NVDA (sell OR buy)”, so they’re targeting stock traders, probably hobbyists & amateurs?
I think it’s clever. I think there’s a ton of value for society in having greater visibility over our cultural discourse, but it’s not yet clear to everyone. One way to pave the way for this is to find early adopters who have a financial incentive to find something novel & useful.
In the twitter community archive project, we’ve collected an open dataset of ~4 million tweets and I’m trying to get more people to use it, explore it, find something valuable for all of us in there. Meanwhile, twitter has people paying them to do the work of analyzing the data & prove there’s value here. Very clever.
Please don't take any of this as criticism. I'm just exploring your idea out loud. I'm genuinely trying to be constructive.
Twitter is probably coursed. Elon Musk is now linked to Trump. People are moving off the platform to things like BlueSky and Mastodon. Your scope of the W.E.I.R.D. people has possibly changed.
I also wonder what it means "to predict culture" on a social network. I define culture to mean "the things that we do."
Typically, there are two categories of users: producers and consumers.
The producers are usually going to be people looking for engagement for their business. It might be people posting interesting things that they find. Or it's people shitposting. What percentage of users producing on Twitter is a person just trying to express themselves versus trying to be a persona?
So when we measure culture (the way we behave), are we measuring what humans really do or are you measuring how they interact with this tool?
And if you discover interesting behaviours on Twitter, is that a real reflection of how humans behave offline?
Maybe I'm worrying about the wrong thing.