HCII at UIST 2025

HCII at UIST 2025 kharlan Tue, 10/21/2025 - 11:54 October 27, 2025 News What’s next in the future of user interfaces? Look no further for some possibilities. Faculty and students from the Human-Computer Interaction Institute applied creativity to the technology of fabricating textiles and interactive things, sensing and pose estimation, managing tasks, privacy, and a variety of other fields at the recent UIST conference. The Association of Computing Machinery (ACM) Symposium on User Interface Software and Technology (UIST) is known for its interdisciplinary focus and hands-on format. UIST is a gathering place for the dreamers and the makers, uniting the strengths of their diverse fields to push the boundaries of human-computer interaction (HCI). For many UIST (pronounced “wist”) attendees, one favorite aspect of the conference is the annual Demo Reception. Accepted authors showcase their novel interactive concepts, new interaction techniques, devices and systems, while attendees get the opportunity to engage with researchers and enjoy hands-on experiences with the new technologies. One HCII demo associated with the paper “Personalized Bistable Orthoses for Rehabilitation of Finger Joints” was awarded the Best Demo Honorable Mention Award (Jury’s Choice). HCII participants contributed to UIST this year as paper authors, as well as participated in workshops, demos and the doctoral consortium. Jump below for a list of accepted papers with HCII contributing authors. A search for Carnegie Mellon in the UIST 2025 program returns a full list of involvement. For a more visual overview of HCII works at UIST, view this YouTube playlist . HCI PhD student Hyunsung Cho was selected to participate in the Doctoral Consortium, where she shared her work on “Perceptually Intelligent UI for Augmented Everyday Interaction.” Professor Scott Hudson gave a keynote talk at the workshop, "Computational Tailor-Making for Personalized, Shape-changing, and Sustainable Fabrics." UIST was held in Busan, Republic of Korea, from September 28 - October 1, 2025. Alexandra Ion Scott Hudson Chris Harrison Mayank Goel Patrick Carrington Franklin (Mingzhe) Li Carolyn Rosé Sauvik Das Vimal Mollyn Nathan DeVrio Yuyu Lin Kyzyl Monteiro Riku Arakawa Yuvraj Agarwal Violet Han Jesse Gonzalez Yi-Hao Peng Jeffrey Bigham Hyunsung Cho Haiyi Zhu Hong Shen Alice Qian Dominik Moritz Ken Holstein Hamza El Alaoui Atieh Taheri Physical Interfaces – Sensing, Haptics, Fabrication Human-Centered AI Tools and Programming Accessibility and Assistive Technology AR / VR / XR Accepted Papers with HCII Contributing Authors Jump to a category below or scroll down to view all: Textiles and Fabrication Managing Tasks Sensing and Pose Estimation Privacy UI Agents, Gen AI Agents, Object Agents Textiles and Fabrication Personalized Bistable Orthoses for Rehabilitation of Finger Joints Yuyu Lin, Dian Zhu, Anoushka Naidu, Kenneth Yu, Deon Harper, Eni Halilaj, Douglas Weber, Deborah Kenney, Adam J. Popchak, Mark Baratz, Alexandra Ion Orthoses are essential components of rehabilitation yet limited in functionality. To facilitate both rehabilitation and dexterity, we introduce a novel multifunctional yet unpowered finger orthosis design. Our design supports easy switching between two distinct states: a stiff state for immobilization and a flexible state for mobilization. A key benefit is that it can be customized using our computational design tool, and 3D printed in one piece. 📄 Paper ▶ Video 🔗 Coverage: CMU Researchers Develop Customizable Finger Brace for Injury Recovery Kinethreads: Soft Full-Body Haptic Exosuit using Low-Cost Motor-Pulley Mechanisms Vivian Shen, Chris Harrison Our bodies experience a wide variety of kinesthetic forces as we go about our daily lives, including the weight of held objects, contact with surfaces, gravitational loads, and acceleration and centripetal forces while driving, to name just a few. These forces are crucial to realism, yet are simply not possible to render with today's consumer haptic suits, which primarily rely on arrays of vibration actuators built into vests. In this work, we present Kinethreads: a new full-body haptic exosuit design built around string-based motor-pulley mechanisms, which keeps our suit lightweight (<5kg), soft and flexible, quick-to-wear (<30 seconds), comparatively low-cost (~$400), and yet capable of rendering expressive, distributed, and forceful (up to 120N) effects. 📄 Paper ▶ Video 🔗 Project page Sculptable Mesh Structures for Large-Scale Form-Finding Jesse T Gonzalez, Yanzhen Zhang, Dian Zhu, Alice Yu, Sapna Tayal, Nazm Furniturewala, Ziying Qi, Somin Ella Moon, Leyi Han, Alexandra Ion, Scott E Hudson To better capture the interplay between real-world objects and a designer’s work-in-progress, practitioners will often go through a sequence of low-fidelity prototypes (paper, clay, foam) before arriving at a form that satisfies both functional and aesthetic concerns. While necessary, this model-making process can be quite time-consuming, particularly at larger scales, and the resulting geometry can be difficult to translate into a CAD environment, where it will be further refined. This paper introduces a user-adjustable, room-scale, "shape-aware" mesh structure for low-fidelity prototyping. A user physically manipulates the mesh by lengthening and shortening the edges, altering the overall curvature and sculpting coarse forms. 📄 Paper ▶ Video 🔗 Project page Transforming Everyday Objects into Dynamic Interfaces using Smart Flat-Foldable Structures Violet Yinuo Han, Amber Yinglei Chen, Mason Zadan, Jesse T Gonzalez, Dinesh K Patel, Wendy Fangwen Yu, Carmel Majidi, Alexandra Ion Dynamic physical interfaces are often dedicated devices designed to adapt their physical properties to user needs. In this paper, we present an actuation system that allows users to transform their existing objects into dynamic physical user interfaces. We design our actuation system to integrate as a self-contained locomotion layer into existing objects that are small-scale, i.e., hand-size rather than furniture-size. We envision that such objects can act as collaborators: as a studio assistant in a painter's palette, as tutors in a student's ruler, or as caretakers for plants evading direct sunlight. 📄 Paper ▶ Video Using an Array of Needles to Create Solid Knitted Shapes Francois Guimbretiere, Victor F Guimbretiere, Amritansh Kwatra, Scott E Hudson Textiles offer many advantages as a fabrication material, particularly when viewed from the perspective of human use. We present a different machine design using a 2D bed of knitting needles to fabricate additional solid knitted forms. This approach provides substantially more flexibility on how to structure stitches as yarn paths inside a volume. We describe a small prototype 6x6 needle machine, and demonstrate that it can create traditional knits, horizontal knits (knitted in the plane of the needles), and solid knits, including overhangs, and pyramidal forms. 📄 Paper ▶ Video Designing Looms as Kits for Collaborative Assembly Samantha Speer, Nickolina Yankova, Joey Huang, Carolyn Rosé, Kylie A Peppler, James McCann, Melisa Orta Martinez Both engineering kits and collaboration skills are increasingly prevalent in education and are often used in conjunction, yet little research addresses how kit hardware can be designed to support collaboration. This paper posits that certain design features of kits can promote or hinder collaboration. We present a user study evaluating the collaborative assembly of two loom kits: one for higher education (RoboLoom) and one for individuals (Ashford Loom). We developed a coding scheme from the 3Cs framework (coordination, cooperation, and communication) to analyze which hardware features influenced collaboration. We find five design feature categories that may influence collaboration: repetitiveness, specificity, difficulty, parallelizability, and physicality. 📄 Paper ▶ Video DropPop: Designing Drop-to-Deploy Mechanisms with Bistable Scissors Structures Yibo Fu, Emily Guan, Jianzhe Gu, Dinesh K Patel, Justin U Soza Soto, Yichi Luo, Carmel Majidi, Josiah Hester, Lining Yao Deployable structures often rely on complex deployment mechanisms such as external pneumatic pumps, electric motors, or manual assembly. These conventional methods, which are intended for applications in shape morphing architectures, robotics, and product design, can be bulky and unwieldy for everyday interaction and daily use. We introduce a new class of deployable structures that harness the locomotion of a single bistable cap to drive the expansion of a scissor-like mechanism. We explore various input modalities for deployment such as hand dropping, and drone deployment, and showcase demo applications. 📄 Paper Managing Tasks StepWrite: Adaptive Planning for Speech-Driven Text Generation Hamza El Alaoui, Atieh Taheri, Yi-Hao Peng, Jeffrey P Bigham People frequently use speech-to-text systems to compose short texts with voice. However, current voice-based interfaces struggle to support composing more detailed, contextually complex texts, especially in scenarios where users cannot visually track progress. We introduce StepWrite, a large language model-driven voice-based interaction system that augments human writing ability by enabling structured, hands-free and eyes-free composition of longer-form texts while on the move. StepWrite decomposes the writing process into manageable subtasks and reduces cognitive load by offloading the context-tracking and adaptive planning tasks to the models. This work highlights the potential of structured, context-aware voice interactions in enhancing hands-free and eye-free communication in everyday multitasking scenarios. 📄 Paper ▶ Video Scaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initiative Dialogue Riku Arakawa, Prasoon Patidar, Will Page, Jill Lehman, Mayank Goel Daily tasks such as cooking, machine operation, and medical self-care often require context-aware assistance, yet existing systems are hard to scale due to high training costs and unpredictable and imperfect performance. This work introduces the PrISM framework, which streamlines the process of creating an assistant for users' own tasks using demonstration and dialogue. Evaluated through multiple studies involving several examples of daily tasks in a user's life, our approach demonstrates improved step-tracking accuracy, enhanced user interaction, and an improved sense of collaboration. 📄 Paper ▶ Video AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and Videos Zheng Ning, Leyang Li, Daniel Killough, JooYoung Seo, Patrick Carrington, Yapeng Tian, Yuhang Zhao, Franklin Mingzhe Li, Toby Jia-Jun Li Videos offer rich audiovisual information that can support people in performing activities of daily living (ADLs), but they remain largely inaccessible to blind or low-vision (BLV) individuals. In cooking, BLV people often rely on non-visual cues--such as touch, taste, and smell--to navigate their environment, making it difficult to follow the predominantly audiovisual instructions found in video recipes. To address this problem, we introduce AROMA, an AI system that provides timely responses to the user based on real-time, context-aware assistance by integrating non-visual cues perceived by the user, a wearable camera feed, and video recipe content. 📄 Paper ▶ Video Sensing and Pose Estimation EclipseTouch: Touch Segmentation on Ad Hoc Surfaces using Worn Infrared Shadow Casting Vimal Mollyn, Nathan DeVrio, Chris Harrison The ability to detect touch events on uninstrumented, everyday surfaces has been a long-standing goal for mixed reality systems. Prior work has shown that virtual interfaces bound to physical surfaces offer performance and ergonomic benefits over tapping at interfaces floating in the air. A wide variety of approaches have been previously developed, to which we contribute a new headset-integrated technique called EclipseTouch. We use a combination of a computer-triggered camera and one or more infrared emitters to create structured shadows, from which we can accurately estimate hover distance (mean error of 6.9 mm) and touch contact (98.0% accuracy). We discuss how our technique works across a range of conditions, including surface material, interaction orientation, and environmental lighting. 📄 Paper ▶ Video 🔗 Project page IMUCoCo: Enabling Flexible On-Body IMU Placement for Human Pose Estimation and Activity Recognition Haozhe Zhou, Riku Arakawa, Yuvraj Agarwal, Mayank Goel IMUs are regularly used to sense human motion, recognize activities, and estimate full-body pose. In this paper, we rethink IMU-based motion sensing by acknowledging that signals can be captured from any point on the human body. We introduce IMU over Continuous Coordinates (IMUCoCo), a novel framework that maps signals from a variable number of IMUs placed on the body surface into a unified feature space based on their spatial coordinates. Our evaluations demonstrate that IMUCoCo supports accurate pose estimation in a wide range of typical and atypical sensor placements. Overall, IMUCoCo supports significantly more flexible use of IMUs for motion sensing than the state-of-the-art, allowing users to place their sensors-laden devices according to their needs and preferences. 📄 Paper ▶ Video Privacy Imago Obscura: An Image Privacy AI Co-pilot to Enable Identification and Mitigation of Risks Kyzyl Monteiro, Yuchen Wu, Sauvik Das Users often struggle to navigate the privacy vs. publicity boundary in sharing images online. To address this challenge, we introduce and evaluate Imago Obscura, an intent-aware AI-powered image-editing copilot that enables users to identify and mitigate privacy risks in images they intend to share. Imago Obscura enables users to articulate their image-sharing intent and privacy concerns, and then recommends and facilitates application of a suite of obfuscation techniques found to be effective in prior literature - e.g., inpainting, blurring, and generative content replacement. We evaluated Imago Obscura with 15 end-users in a lab study and found that it improved users' awareness of image privacy risks and their ability to address them, enabling more informed sharing decisions. 📄 Paper ▶ Video UI Agents, Gen AI Agents, Object Agents Morae: Proactively Pausing UI Agents for User Choices Yi-Hao Peng, Ding Li, Jeffrey P Bigham, Amy Pavel User interface (UI) agents promise to make inaccessible or complex UIs easier to access for blind and low-vision (BLV) users. However, current UI agents typically perform tasks end-to-end without involving users in critical choices or making them aware of important contextual information, thus reducing user agency. To address this problem, we introduce Morae, a UI agent that automatically identifies decision points during task execution and pauses so that users can make choices. Morae uses large multimodal models to interpret user queries alongside UI code and screenshots, and prompts users for clarification when there is a choice to be made. This work exemplifies a mixed-initiative approach in which users benefit from the automation of UI agents while being able to express their preferences. 📄 Paper ▶ Video POET: Supporting Prompting Creativity and Personalization with Automated Expansion of Text-to-Image Generation Evans Xu Han, Alice Qian, Haiyi Zhu, Hong Shen, Paul Pu Liang, Jane Hsieh State-of-the-art visual generative AI tools hold immense potential to assist users in the early ideation stages of creative tasks. We introduce POET, a real-time interactive tool that (1) automatically discovers dimensions of homogeneity in text-to-image generative models, (2) expands these dimensions to diversify the output space of generated images, and (3) learns from user feedback to personalize expansions. Focusing on visual creativity, POET offers a first glimpse of how interaction techniques of future text-to-image generation tools may support and align with more pluralistic values and the needs of end users during the ideation stages of their work. 📄 Paper ▶ Video Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors Michelle S. Lam, Fred Hohman, Dominik Moritz, Jeffrey P Bigham, Kenneth Holstein, Mary Beth Kery AI policy sets boundaries on acceptable behavior for AI models, but this is challenging in the context of large language models (LLMs). We introduce policy maps, an approach to AI policy design inspired by the practice of physical mapmaking. Instead of aiming for full coverage, policy maps aid effective navigation through intentional design choices about which aspects to capture and which to abstract away. With Policy Projector, an interactive tool for designing LLM policy maps, an AI practitioner can survey the landscape of model input-output pairs, define custom regions (e.g., “violence”), and navigate these regions with if-then policy rules that can act on LLM outputs (e.g., if output contains “violence” and “graphic details,” then rewrite without “graphic details”). 📄 Paper ▶ Video Towards Unobtrusive Physical AI: Augmenting Everyday Objects with Intelligence and Robotic Movement for Proactive Assistance Violet Yinuo Han, Jesse T Gonzalez, Christina Yang, Zhiruo Wang, Scott E Hudson, Alexandra Ion What if familiar objects could proactively assist users? For example, a stapler moving across the table to help users organize documents. In this paper, we build on the qualities of tangible interaction and focus on recognizing user needs in everyday tasks to enable ubiquitous yet unobtrusive tangible interaction. To achieve this, we introduce an architecture that leverages Large Language Models (LLMs) to perceive users’ environment and daily activities, perform spatial-temporal reasoning, and generate object actions aligned with inferred user intentions and object properties. 📄 Paper ▶ Video 🔗 Coverage: This Stapler Knows When You Need It New projects advance fabrication, sensing, task completion and more HCI PhD student Yuyu Lin presents "Personalized Bistable Orthoses for Rehabilitation of Finger Joints" at UIST 2025. Preserve My Content Edits Off