Lower Quantity, Higher Quality: Auditing News Content and User Perceptions on Twitter/X Algorithmic versus Chronological Timelines

TL;DR

This study conducts a sociotechnical audit of Twitter/X to compare algorithmic and chronological timelines in terms of news content and user effects. Using a three-week, randomized, within-subject design with 243 real users and a custom browser extension, the authors collect over 800,000 tweets to assess content outputs and deploy four surveys to track perceptions. Key findings show the algorithmic timeline reduces news exposure but yields less ideologically congruent, less extreme, and slightly more reliable news, while user behaviors shift toward greater engagement under the algorithmic feed; perceptions of the platform remain largely stable, revealing a gap between behavior and attitude. The work highlights methodological value in combining content audits with user-centered analysis and calls for more user-focused investigations to align technical performance with user experiences and democratic objectives.

Abstract

Social media personalization algorithms increasingly influence the flow of civic information through society, resulting in concerns about "filter bubbles", "echo chambers", and other ways they might exacerbate ideological segregation and fan the spread of polarizing content. To address these concerns, we designed and conducted a sociotechnical audit (STA) to investigate how Twitter/X's timeline algorithm affects news curation while also tracking how user perceptions change in response. We deployed a custom-built system that, over the course of three weeks, passively tracked all tweets loaded in users' browsers in the first week, then in the second week enacted an intervention to users' Twitter/X homepage to restrict their view to only the algorithmic or chronological timeline (randomized). We flipped this condition for each user in the third week. We ran our audit in late 2023, collecting user-centered metrics (self-reported survey measures) and platform-centered metrics (views, clicks, likes) for 243 users, along with over 800,000 tweets. Using the STA framework, our results are two-fold: (1) Our algorithm audit finds that Twitter/X's algorithmic timeline resulted in a lower quantity but higher quality of news -- less ideologically congruent, less extreme, and slightly more reliable -- compared to the chronological timeline. (2) Our user audit suggests that although our timeline intervention had significant effects on users' behaviors, it had little impact on their overall perceptions of the platform. Our paper discusses these findings and their broader implications in the context of algorithmic news curation, user-centric audits, and avenues for independent social science research.

Figures (5)

• Figure 1: Full timeline of the study. In the observational phase users view Twitter/X timelines as usual. In the first intervention we randomly assign users into the algorithmic or chronological condition. In the second intervention, for each user, we flip their condition. The figure also shows the timing of the survey deployments.
• Figure 2: Algorithmic effects on news curation: Users assigned to the algorithmic timeline were exposed to tweets with less congruent and less extreme news links. The effect on reliability is not significant.
• Figure 3: Algorithmic effects on user behavior. Users assigned to the algorithmic timeline used and engaged with Twitter/X more, however, they saw less tweets with news links. The effect on their engagement with news tweets is not significant.
• Figure 4: Summary of participants (N=243) by self-reported partisanship and ideology. Partisanship is reported on a 7-point scale from ('strong democrat' to 'strong republican'). Ideology is reported on a 7-point scale from ('extremely liberal' to 'extremely conservative').
• Figure 5: Summary of participant demographics (N=243) by age, gender, race, education level and income level.