Center for Advanced Intelligence Project

Date and Time: June 5, 2025: 9:30 – 10:30 (JST)
Venue: Online and Open Space at the RIKEN AIP Nihonbashi office
*Open Space is available to AIP researchers only

Title: Uncertainty Quantification for LLM-Based Survey Simulations

Speaker: Kaizheng Wang (Columbia University)

Abstract: We investigate the use of large language models (LLMs) to simulate human responses to survey questions, and perform uncertainty quantification to gain reliable insights. Our approach converts imperfect LLM-simulated responses into confidence sets for population parameters of human responses, addressing the distribution shift between the simulated and real populations. A key innovation lies in determining the optimal number of simulated responses: too many produce overly narrow confidence sets with poor coverage, while too few yield excessively loose estimates. To resolve this, our method adaptively selects the simulation sample size, ensuring valid average-case coverage guarantees. It is broadly applicable to any LLM, irrespective of its fidelity, and any procedure for constructing confidence sets. Additionally, the selected sample size quantifies the degree of misalignment between the LLM and the target human population. We illustrate our method on real datasets and LLMs. The talk is based on joint work with Chengpiao Huang and Yuhang Wu.

Bio: Kaizheng Wang is an assistant professor of Industrial Engineering and Operations Research, and a member of the Data Science Institute at Columbia University. He works at the intersection of statistics, machine learning, and optimization.