Research/CV

Underlined names indicate research assistants I mentored.

Joshua Ashkinaze, Laura Kurek, Alina Faisal, Tongyuan Miao, Mariam Joseph, Ceren Budak, and Eric Gilbert. Information Discernment in Large Language Models. The 40th Annual Conference on Neural Information Processing Systems (NeurIPS) (2026).

LLMs are increasingly used with external knowledge sources like the internet. Do they weigh information appropriately -- updating more for reliable sources (source discernment) and more when claims bring priors closer to the truth (truth discernment)? We formalize this as information discernment and introduce Learn2Discern (L2D), an experimental framework and benchmark grounded in three normative axioms with interpretable metrics. To establish external validity, a pre-registered, quota-matched user study (n=299) confirms that real LLM users endorse all three axioms and report that violations reduce their trust and usage intent. Across 13 models and nearly 670K trials, we find consistent failures across both dimensions: models perform near chance on source and truth discernment, rely on source popularity twice as much as source reliability, and update roughly equally whether a claim improves or worsens their position relative to the ground truth. Models integrate external knowledge most effectively on datasets where their priors are already the most accurate. Newer and larger models improve truth discernment but not source discernment, a blind spot that model complexity does not address. We identify simple inference-time interventions that improve both forms of discernment. We release our dataset and survey as a testbed for a core alignment property that scales in importance as LLMs replace traditional search.

Joshua Ashkinaze, Hua Shen, Saipranav Avula, Ceren Budak, and Eric Gilbert. Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferences. The 39th Annual Conference on Neural Information Processing Systems (NeurIPS) (2025).

Note: Spotlight 🏆 (top 3% of submissions)

We introduce the Deep Value Benchmark (DVB), an evaluation framework that directly tests whether large language models (LLMs) learn fundamental human values or merely surface-level preferences. This distinction is critical for AI alignment: Systems that capture deeper values are likely to generalize human intentions robustly, while those that capture only superficial patterns in preference data risk producing misaligned behavior. The DVB uses a novel experimental design with controlled confounding between deep values (e.g., moral principles) and shallow features (e.g., superficial attributes). In the training phase, we expose LLMs to human preference data with deliberately correlated deep and shallow features -- for instance, where a user consistently prefers (non-maleficence, formal language) options over (justice, informal language) alternatives. The testing phase then breaks these correlations, presenting choices between (justice, formal language) and (non-maleficence, informal language) options. This design allows us to precisely measure a model's Deep Value Generalization Rate (DVGR) -- the probability of generalizing based on the underlying value rather than the shallow feature. Across 9 different models, the average DVGR is just 0.30. All models generalize deep values less than chance. Larger models have a (slightly) lower DVGR than smaller models. We are releasing our dataset, which was subject to three separate human validation experiments. DVB provides an interpretable measure of a core feature of alignment.

Joshua Ashkinaze, Ruijia Guan, Laura Kurek, Eytan Adar, Ceren Budak, and Eric Gilbert. Seeing Like an AI: How LLMs Apply (and Misapply) Wikipedia Neutrality Norms. The 20th International AAAI Conference on Web and Social Media (ICWSM) (2026).

Note: We were very happy that this research had substantial impact with key stakeholders: I was invited to give a talk at a Wikimedia research showcase, and our paper is cited in Wikipedia strategy around integrating AI into the platform.

Large language models (LLMs) are trained on broad corpora and then used in communities with specialized norms. Is providing LLMs with community rules enough for models to follow these norms? We evaluate LLMs’ capacity to detect (Task 1) and correct (Task 2) biased Wikipedia edits according to Wikipedia’s Neutral Point of View (NPOV) policy. LLMs struggled with bias detection, achieving only 64% accuracy on a balanced dataset. Models exhibited contrasting biases (some under- and others over-predicted bias), suggesting distinct priors about neutrality. LLMs performed better at generation, removing 79% of words removed by Wikipedia editors. However, LLMs made additional changes beyond Wikipedia editors’ simpler neutralizations, resulting in high-recall but low-precision editing. Interestingly, crowdworkers rated AI rewrites as more neutral (70%) and fluent (61%) than Wikipedia-editor rewrites. Qualitative analysis found LLMs sometimes applied NPOV more comprehensively than Wikipedia editors but often made extraneous non-NPOV-related changes (such as grammar). LLMs may apply rules in ways that resonate with the public but diverge from community experts. While potentially effective for generation, LLMs may reduce editor agency and increase moderation workload (e.g., verifying additions). Even when rules are easy to articulate, having LLMs apply them like community members may still be difficult.

Joshua Ashkinaze, Emily Fry, Narendra Edara, Eric Gilbert, and Ceren Budak. Plurals: A System for Guiding LLMs Via Simulated Social Ensembles. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems.

Note 1: Honorable Mention 🏆 (top 5% of submissions)

Note 2: Check out the GitHub library and documentation!

Recent debates raised concerns that language models may favor certain viewpoints. But what if the solution is not to aim for a “view from nowhere” but rather to leverage different viewpoints? We introduce Plurals, a system and Python library for pluralistic AI deliberation. Plurals consists of Agents (LLMs, optionally with personas) which deliberate within customizable Structures, with Moderators overseeing deliberation. Plurals is a generator of simulated social ensembles. Plurals integrates with government datasets to create nationally representative personas, includes deliberation templates inspired by deliberative democracy, and allows users to customize both information-sharing structures and deliberation behavior within Structures. Six case studies demonstrate fidelity to theoretical constructs and efficacy. Three randomized experiments show simulated focus groups produced output resonant with an online sample of the relevant audiences (chosen over zero-shot generation in 75% of trials). Plurals is both a paradigm and a concrete system for pluralistic AI.

Joshua Ashkinaze, Julia Mendelsohn, Qiwei Li, Ceren Budak, and Eric Gilbert. How AI ideas affect the creativity, diversity, and evolution of human ideas: Evidence from a large, dynamic experiment. Proceedings of the 2025 ACM Collective Intelligence Conference.

Note: Honorable Mention 🏆 (top 5% of submissions)

Exposure to large language model output is rapidly increasing. How will seeing AI-generated ideas affect human ideas? We conducted an experiment (800+ participants, 40+ countries) where participants viewed creative ideas that were from ChatGPT or prior experimental participants and then brainstormed their own idea. We varied the number of AI-generated examples (none, low, or high exposure) and if the examples were labeled as 'AI' (disclosure). Our dynamic experiment design -- ideas from prior participants in an experimental condition are used as stimuli for future participants in the same experimental condition -- speaks to the interdependent process of cultural creation: creative ideas are built upon prior ideas. Hence, we capture the compounding effects of having LLMs 'in the culture loop'. We find that high AI exposure (but not low AI exposure) did not affect the creativity of individual ideas but did increase the average amount and rate of change of collective idea diversity. AI made ideas different, not better. There were no main effects of disclosure. We also found that self-reported creative people were less influenced by knowing an idea was from AI and that participants may knowingly adopt AI ideas when the task is difficult. Our findings suggest that introducing AI ideas may increase collective diversity but not individual creativity.

Shubham Atreja, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, Libby Hemphill. What's in a Prompt?: A Large-Scale Experiment to Assess the Impact of Prompt Design on the Compliance and Accuracy of LLM-Generated Text Annotations. The 19th International AAAI Conference on Web and Social Media (ICWSM) (2025).

Manually annotating data for computational social science tasks can be costly, time-consuming, and emotionally draining. While recent work suggests that LLMs can perform such annotation tasks in zero-shot settings, little is known about how prompt design impacts LLMs' compliance and accuracy. We conduct a large-scale multi-prompt experiment to test how model selection (GPT-4o, GPT-3.5, PaLM2, and Falcon7b) and prompt design features (definition inclusion, output type, explanation, and prompt length) impact the compliance and accuracy of LLM-generated annotations on four highly relevant and diverse CSS tasks (toxicity, sentiment, rumor stance, and news frames). Our results show that LLM compliance and accuracy are prompt-dependent. For instance, prompting for numerical scores instead of labels reduces all LLMs' compliance and accuracy. Concise prompts can significantly reduce prompting costs but also lead to lower accuracy on tasks like toxicity. Furthermore, minor prompt changes like asking for an explanation can cause large changes in the distribution of LLM-generated labels. By assessing the impact of prompt design on the quality and distribution of LLM-generated annotations, this work serves as both a practical guide and a warning for using LLMs in CSS research.

Joshua Ashkinaze, Eric Gilbert, Ceren Budak. The Dynamics of (Not) Unfollowing Misinformation Spreaders. Proceedings of the 2024 ACM Web Conference (Formerly WWW).

Note: Oral presentation (top 9% of submissions); selected for the special Online Trust and Safety day.

Many studies explore how people "come into" misinformation exposure. But much less is known about how people "come out of" misinformation exposure. Do people organically sever ties to misinformation spreaders? And what predicts doing so? Over six months, we tracked the frequency and predictors of ~900K followers unfollowing ~5K health misinformation spreaders on Twitter. We found that misinformation ties are persistent. Monthly unfollowing rates are just 0.52%. In other words, 99.5% of misinformation ties persist each month. Users are also 31% more likely to unfollow non-misinformation spreaders than they are to unfollow misinformation spreaders. Although generally infrequent, the factors most associated with unfollowing misinformation spreaders are (1) redundancy and (2) ideology. First, users initially following many spreaders, or who follow spreaders that tweet often, are most likely to unfollow later. Second, liberals are more likely to unfollow than conservatives. Overall, we observe a strong persistence of misinformation ties. The fact that users rarely unfollow misinformation spreaders suggests a need for external nudges and the importance of preventing exposure from arising in the first place.

---

Qiwei Li, Shihui Zhang, Andrew Timothy Kasper, Joshua Ashkinaze, Asia A. Eaton, Sarita Schoenebeck, Eric Gilbert. Reporting Non-Consensual Intimate Media: An Audit Study of Deepfakes. ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) (2026).

Non-consensual intimate media (NCIM) inflicts significant harm. Currently, victim-survivors can use two mechanisms to report NCIM - as a non-consensual nudity violation or as copyright infringement. We conducted an audit study of takedown speed of NCIM reported to X (formerly Twitter) of both mechanisms. We uploaded 50 AI-generated nude images and reported half under X's "non-consensual nudity" reporting mechanism and half under its "copyright infringement" mechanism. The copyright condition resulted in successful image removal within 25 hours for all images (100% removal rate), while non-consensual nudity reports resulted in no image removal for over three weeks (0% removal rate). We stress the need for targeted legislation to regulate NCIM removal online. We also discuss ethical considerations for auditing NCIM on social platforms.

---

Karen Bonuck, Akilah Collins-Anderson, Joshua Ashkinaze, Alison Karasz, and Amanda Schwartz. Environmental scan of sleep health in early childhood programs. Behavioral Sleep Medicine, 18(5), 598–610 (2020).

Objective: To ascertain how sleep health knowledge is translated to early care and education (ECE) programs, using a multi-component environmental scan.

Methods: A website scan identified organizations' sleep content re: recommended practices, developmental effects, and "actionable" ratings (0–2). ECE staff surveys assessed preparedness, practices, and beliefs about addressing sleep health and sleep problems in ECE programs. Semi-structured interviews with stakeholders from the ECE, pediatric and sleep communities assessed awareness, priorities, and practices at their organizations.

Results: Of 15 websites scanned, half lacked sleep content on links to development, optimal duration, or scientific background. ECE staff (n = 31) were comfortable speaking to parents about healthy sleep, and with incorporating sleep education and guidance into ECE. Stakeholders (n = 15) rated healthy sleep as a high relevance, but lower priority issue. Within ECE settings stakeholders reported that knowledge about specific links to health and development was poor and that sleep health was often obscured by "safe sleep" issues. Their recommendations included: linking sleep health to "hot topics" such as obesity or preschool suspensions and expulsions, integrating it with the teaching of routines, and raising public awareness.

Conclusion: Despite understanding that healthy sleep promotes school readiness, there is insufficiently specific, actionable information in ECE training, programs, or policies. Findings suggest a need for an awareness campaign with clear, actionable messaging, dissemination of turnkey materials, and integration with policy and professional training systems.

Trial Registration: ClinicalTrials.Gov: NCT03556462.

Preprints and Under Review

Underlined names indicate research assistants I mentored.

Joshua Ashkinaze, Yubo Zhou, Pengyu Chen, Avi Bhattacharya, Hyebin Park, Yanman Zhang, Stephanie Preston, Eric Gilbert, Ceren Budak. The As-If Machine: Making Distant Risks Feel Concrete via Personalized Counterfactuals. arXiv preprint arXiv:2610.33301 (2026).

Life experiences shape policy attitudes, but we only live one life. Citizens may therefore fail to support policies addressing risks they have never personally experienced. Narratives can make such risks immediate and personal, but personalized stories remain difficult to produce at scale. Here we ask whether AI can make distant scenarios feel concrete by simulating a participant's life as if that scenario occurred. We present the As-If Machine (AIM), a multi-agent system through which people co-write personalized counterfactual narratives—stories about their own lives under different policy futures. In two pre-registered and quota-matched experiments on science funding (N = 1,007) and satellite cybersecurity (N = 894), we compared personalized co-writing with an expert-written vignette on the same topic and with the same task on an unrelated topic. AIM increased psychological closeness—how close a risk feels to the self—by nearly identical margins in the two domains (0.36 and 0.38 standard deviations against the vignette, 0.80 and 0.90 against unrelated writing). AIM increased psychological closeness even among participants with unfavorable views of AI. Effects on policy attitudes were 0.16 SDs against the vignette and 0.10 against unrelated writing for science funding, and 0.54 and 0.67 SDs, respectively, for satellite cybersecurity. Across 23,496 human–AI autocomplete interactions, participants and the system played distinct roles: AIM supplied concrete detail that grounded the scenario, and participants supplied first-person perspective and emotion. Co-writing with the As-If Machine makes distant policy risks feel close.

---

Laura Kurek, Joshua Ashkinaze, Ceren Budak, Eric Gilbert. Follow Nudges without Budges: A Field Experiment on Misinformation Followers Didn't Change Follow Networks. arXiv preprint arXiv:2512.13643 (2025).

Can digital ads encourage users exposed to inaccurate information sources to follow accurate ones? We conduct a large-scale field experiment (N = 28,582) on X, formerly Twitter, with users who follow accounts that spread health misinformation. Participants were exposed to four ad treatments varied on two dimensions: a neutral message versus a persuasive message appealing to values of independence, and a request to follow a health institution versus a request to follow a health influencer. We term this ad-based, social network intervention a follow nudge. The ad with a persuasive message to follow a well-known health institution generated significantly higher click-through rates than all other conditions (Bonferroni-corrected pairwise tests, all p < .001). Given the overall low click-through rate across treatments and the high cost of digital advertising infrastructure on X, however, we conclude that our proposed intervention -- at least in its current ad-based format -- is not a cost-effective means to improve information environments online. We discuss challenges faced when conducting large-scale experiments on X following the platform's ownership change and subsequent restrictions on data access for research purposes.