Tag Archives: selected

One Rating to Rule Them All? Evidence of Multidimensionality in Human Assessment of Topic Labeling Quality

Status

Amin Hosseiny Marani, Joshua Levine, Eric P. S. Baumer. 2022. One Rating to Rule Them All? Evidence of Multidimensionality in Human Assessment of Topic Labeling Quality. In Proceedings of the ACM International Conference on Information & Knowledge Management (CIKM).

Abstract

Two general approaches are common for evaluating automatically generated labels in topic modeling: direct human assessment; or performance metrics that can be calculated without, but still correlate with, human assessment. However, both approaches implicitly assume that the quality of a topic label is single-dimensional. In contrast, this paper provides evidence that human assessments about the quality of topic labels consist of multiple latent dimensions. This evidence comes from human assessments of four simple labeling techniques. For each label, study participants responded to several items asking them to assess each label according to a variety of different criteria. Exploratory factor analysis shows that these human assessments of labeling quality have a two-factor latent structure. Subsequent analysis demonstrates that this multi-item, two-factor assessment can reveal nuances that would be missed using either a single-item human assessment of perceived label quality or established performance metrics. The paper concludes by suggesting future directions for the development of human-centered approaches to evaluating NLP and ML systems more broadly.

DOI | pdf

“It’s Like the Value System in the Loop”: Domain Experts’ Values Expectations for NLP Automation

Status

Dilruba Showkat and Eric P. S. Baumer. 2022. “It’s Like the Value System in the Loop”: Domain Experts’ Values Expectations for NLP Automation. In Proceedings of the ACM Conference on Designing Interactive Systems (DIS), 100–122.

Abstract

The rise of automated text processing systems has led to the development of tools designed for a wide variety of application domains. These technologies are often developed to support non-technical users such as domain experts and are often developed in isolation of the tools primary user. While such developments are exciting, less attention has been paid to domain experts’ expectations about the values embedded in these automated systems. As a step toward addressing that gap, we examined values expectations of journalists and legal experts. Both these domains involve extensive text processing and place high importance on values in professional practice. We engaged participants from two non-profit organizations in two separate co-speculation design workshops centered around several speculative automated text processing systems. This study makes three interrelated contributions. First, we provide a detailed investigation of domain experts’ values expectations around future NLP systems. Second, the speculative design fiction concepts, which we specifically crafted for these investigative journalists and legal experts, illuminated a series of tensions around the technical implementation details of automation. Third, our findings highlight the utility of design fiction in eliciting not-to-design implications, not only about automated NLP but also about technology more broadly. Overall, our study findings provide groundwork for the inclusion of domain experts values whose expertise lies outside of the field of computing into the design of automated NLP systems.

DOI

Of Course it’s Political! A Critical Inquiry into Underemphasized Dimensions in Civic Text Visualization

Status

Eric P. S. Baumer, Mahmood Jasim, Ali Sarvghad, and Narges Mahyar. 2022. Of Course it’s Political! A Critical Inquiry into Underemphasized Dimensions in Civic Text Visualization. Computer Graphics Forum 41, 3 (EuroVis).

Abstract

Recent developments in critical information visualization have brought the field’s attention to political, feminist, ethical, and rhetorical aspects of data visualization. However, less work has explored the interplay between design decisions and political ramifications—structures of authority, means of representation, etc. In this paper, we build upon these critical perspectives and highlight the political aspect of civic text visualization especially in the context of democratic decision-making. Based on a critical analysis of survey papers about text visualization in general, followed by a review on the status quo of text visualization in civics, we argue that civic text visualization inherits an exclusively analytic framing. This framing leads to a series of issues and challenges in the fundamentally political context of civics, such as misinterpretation of data, missing minority voices, and excluding the public from decision making processes. To span this gap between political context and analytic framing, we provide a series of two-pole conceptual dimensions, such as from singular user to multiple relationships, and from complexity to inclusivity of visualization design. For each dimension, we discuss how the tensions between these poles can help surface the political ramifications of design decisions in civic text visualization. These dimensions can thus help visualization researchers, designers, and practitioners attend more intentionally to these political aspects and inspire their design choices. We conclude by suggesting that these dimensions may be useful for visualization design across a variety of application domains, beyond civic text visualization.

DOI | preprint

Where Do Stories Come From? Examining the Exploration Process in Investigative Data Journalism

Status

Dilruba Showkat and Eric P. S. Baumer. 2021. Where Do Stories Come From? Examining the Exploration Process in Investigative Data Journalism. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2: 390:1-390:31.

Abstract

Investigative data journalists work with a variety of data sources to tell a story. Though prior work has indicated that there is a close relationship between journalists’ data work practices and that of data scientists. However, these relationships and data work practices are not empirically examined, and understanding them is crucial to inform the design of tools that are used by different groups of people including data scientists and data journalists. Thus, to bridge this gap, we studied investigative reporters’ data work practices with one non-profit investigative newsroom. Our study design includes two activities: 1) semi-structured interviews with journalists, and 2) a sketching activity allowing journalists to depict examples of their work practices. By analyzing these data and synthesizing them across related prior work, we propose the major phases in the data-driven investigative journalism story idea generation process. Our study findings show that the journalists employ a collection of multiple, iterative, cyclic processes to identify journalistically “interesting” story ideas. These processes both significantly resemble and show subtle nuanced differences with data science work practices identified in prior research. We further verified our proposal through a member check with key informants. This work offers three primary contributions. First, it provides a close glimpse into the main phases of investigative journalists’ data-driven story idea generation technique. Second, it complements prior work studying formal data science practices by examining data-driven investigative journalists, whose primary expertise lies outside computing. Third, it identifies particular points in the data exploration processes that would benefit from design interventions and suggests future research directions.

DOI

A Review on Strategies for Data Collection, Reflection, and Communication in Eating Disorder Apps

Status

Anjali Devakumar, Jay Modh, Bahador Saket, Eric P. S. Baumer, and Munmun De Choudhury. 2021. A Review on Strategies for Data Collection, Reflection, and Communication in Eating Disorder Apps. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 1–19.

Abstract

Eating disorders (EDs) constitute a mental illness with the highest mortality. Today, mobile health apps provide promising means to ED patients for managing their condition. Apps enable users to monitor their eating habits, thoughts, and feelings, and offer analytic insights for behavior change. However, not only have scholars critiqued the clinical validity of these apps, their underlying design principles are not well understood. Through a review of 34 ED apps, we uncovered 11 different data types ED apps collect, and 9 strategies they employ to support collection and refection. Drawing upon personal health informatics and visualization frameworks, we found that most apps did not adhere to best practices on what and how data should be collected from and reflected to users, or how data-driven insights should be communicated. Our review offers suggestions for improving the design of ED apps such that they can be useful and meaningful in ED recovery.

DOI

Misfires, Missed Data, Misaligned Treatment: Disconnects in Collaborative Treatment of Eating Disorders

Status

Lauren C. Taylor, Kelsie Belan, Munmun De Choudhury, and Eric P. S. Baumer. 2021. Misfires, Missed Data, Misaligned Treatment: Disconnects in Collaborative Treatment of Eating Disorders. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1: 31:1-31:28.

Abstract

Technology bears important relationships to our health and wellness and has been utilized over the past two decades as an aid to support both self-management goals as well as collaboration among treatment teams. However, when chronic illnesses such as eating disorders (ED) are managed outside of institutionalized care settings, designing effective technology to support collaboration in treatment necessitates that we understand the relationships between patients, clinicians, and support networks. We conducted in-depth, semi-structured, interviews with 9 ED patients and 10 clinicians to understand the ED journey through the lens of collaborative efforts, technology use, and potential detriments. Based on our analysis of these 19 interviews, we present novel findings on various underlying disconnects within the collaborative ED treatment process – disconnects among clinicians, between treatment foci, among preferences in tracking, within support networks, and in patients’ own identities. Our findings highlight how these various disconnects are concomitant with and gaps can stem from a lack of collaboration between different stakeholders in the ED journey. We also identify methods of facilitating collaboration in these disconnects through technological mediators.

DOI

Who Has a Choice?: Survey-Based Predictors of Volitionality in Facebook Use and Non-use

Status

Patrick Skeba, Devansh Saxena, Shion Guha, and Eric P. S. Baumer. (2021). Who Has a Choice?: Survey-Based Predictors of Volitionality in Facebook Use and Non-use. Proceedings of the ACM on Human-Computer Interaction 5, GROUP: 223:1-223:25.

Abstract

This paper examines volitionality of Facebook usage, that is, which individuals feel they have a choice about whether or not to use the site. It analyzes data from two large surveys, conducted three years apart. Across the two surveys, a variety of factors impacted whether or not respondents saw their Facebook usage as a matter of their own choice, such as engaging in non-use behaviors, measures of Facebook addiction, a sense of their own agency, and, across both studies, level of education. These results expand on prior literature around technology use and non-use, especially in terms of which populations may feel obligated to use, or be unwillingly prevented from using, social media such as Facebook. Furthermore, they provide potential implications both for future work and for technology policy.

DOI

Informational Friction as a Lens for Studying Algorithmic Aspects of Privacy

Status

Patrick Skeba and Eric P. S. Baumer. (2020). Informational Friction as a Lens for Studying Algorithmic Aspects of Privacy. Proceedings of the ACM on Human-Computer Interaction 4, CSCW.

Abstract

This paper addresses challenges in conceptualizing privacy posed by algorithmic systems that can infer sensitive information from seemingly innocuous data. This type of privacy is of imminent concern due to the rapid adoption of machine learning and artificial intelligence systems in virtually every industry. In this paper, we suggest informational friction, a concept from Floridi’s ethics of information, as a valuable conceptual lens for studying algorithmic aspects of privacy. Informational friction describes the amount of work required for one agent to access or alter the information of another. By focusing on amount of work, rather than the type of information or manner in which it is collected, informational friction can help to explain why automated analyses should raise privacy concerns independently of, and in addition to, those associated with data collection. As a demonstration, this paper analyze law enforcement use of facial recognition, andFacebook’s targeted advertising model using informational friction and demonstrate risks inherent to these systems which are not completely identified in another popular framework, Nissenbaum’s Contextual Integrity.The paper concludes with a discussion of broader implications, both for privacy research and for privacy regulation.

DOI

Evaluating Design Fiction: The Right Tool for the Job

Status

Eric P. S. Baumer, Mark Blythe, and Theresa J. Tanenbaum. (2020). Evaluating Design Fiction: The Right Tool for the Job. in ACM Conference on Designing Interactive Systems (DIS). (Eindhoven, the Netherlands; held virtually). [24.0% acceptance rate; Honorable Mention Award]

Abstract

Design fiction has become so widely adopted that it regularly appears in contexts ranging from CEO speeches to dedicated tracks at academic conferences. However, evaluating this kind of work is difficult; it is not clear what good or bad design fiction is or what the judgment criteria should be. In this paper we assert that design fiction is a heterogeneous set of methods, and practices, able to produce a diversity of scholarly and design contributions. We argue locating these diverse practices under the single header of “design fiction” has resulted in epistemological confusion over the appropriate method of evaluation. We identify different traditions within the HCI literature—critical design; narratology and literary theory; studio-based design “crits”; user studies; scenarios and persona development; and thought experiments—to articulate a typology of evaluative frames. There is often a mismatch between the standards to which design fiction is held and the knowledge that speculative methods seek to produce. We argue that evaluating a given instance of design fiction requires us to properly select the right epistemological tool for the job.

DOI | pdf

Topicalizer: Reframing Core Concepts in Machine Learning Visualization by Co-designing for Interpretivist Scholarship

Status

Eric P. S. Baumer, Drew Siedel, Lena McDonnell, Jiayun Zhong, Patricia Sittikul, and Micki McGee. (2020). Topicalizer: Reframing Core Concepts in Machine Learning Visualization by Co-designing for Interpretivist Scholarship. Human-Computer Interaction, Special Issue on Unifying Human Computer Interaction and Artificial Intelligence 35(5-6): 452-480.

Abstract

Computational algorithms can provide novel, compelling functionality for interactive systems. However, designing such systems for users whose expertise lies outside computer science poses novel and complex challenges. This paper focuses specifically on the domain of designing interactive topic modeling visualizations to support interpretivist scholars. It describes a co-design process that involved working directly with two such scholars across two different corpora. The resultant visualization has both several similarities and key differences with other topic modeling visualizations, illustrated here using both the final design and discarded prototypes. The paper’s core contribution is an attention to how our emphasis on interpretation played out, both in the design process and in the final visualization design. The paper concludes by discussing the kinds of issues and tensions that emerged in the course of this work, as well as the ways that these issues and tensions can apply to much broader contexts of designing interactive algorithmic systems.

DOI | pdf