Tag Archives: text

One Rating to Rule Them All? Evidence of Multidimensionality in Human Assessment of Topic Labeling Quality

Status

Amin Hosseiny Marani, Joshua Levine, Eric P. S. Baumer. 2022. One Rating to Rule Them All? Evidence of Multidimensionality in Human Assessment of Topic Labeling Quality. In Proceedings of the ACM International Conference on Information & Knowledge Management (CIKM).

Abstract

Two general approaches are common for evaluating automatically generated labels in topic modeling: direct human assessment; or performance metrics that can be calculated without, but still correlate with, human assessment. However, both approaches implicitly assume that the quality of a topic label is single-dimensional. In contrast, this paper provides evidence that human assessments about the quality of topic labels consist of multiple latent dimensions. This evidence comes from human assessments of four simple labeling techniques. For each label, study participants responded to several items asking them to assess each label according to a variety of different criteria. Exploratory factor analysis shows that these human assessments of labeling quality have a two-factor latent structure. Subsequent analysis demonstrates that this multi-item, two-factor assessment can reveal nuances that would be missed using either a single-item human assessment of perceived label quality or established performance metrics. The paper concludes by suggesting future directions for the development of human-centered approaches to evaluating NLP and ML systems more broadly.

DOI | pdf

Topicalizer: Reframing Core Concepts in Machine Learning Visualization by Co-designing for Interpretivist Scholarship

Status

Eric P. S. Baumer, Drew Siedel, Lena McDonnell, Jiayun Zhong, Patricia Sittikul, and Micki McGee. (2020). Topicalizer: Reframing Core Concepts in Machine Learning Visualization by Co-designing for Interpretivist Scholarship. Human-Computer Interaction, Special Issue on Unifying Human Computer Interaction and Artificial Intelligence 35(5-6): 452-480.

Abstract

Computational algorithms can provide novel, compelling functionality for interactive systems. However, designing such systems for users whose expertise lies outside computer science poses novel and complex challenges. This paper focuses specifically on the domain of designing interactive topic modeling visualizations to support interpretivist scholars. It describes a co-design process that involved working directly with two such scholars across two different corpora. The resultant visualization has both several similarities and key differences with other topic modeling visualizations, illustrated here using both the final design and discarded prototypes. The paper’s core contribution is an attention to how our emphasis on interpretation played out, both in the design process and in the final visualization design. The paper concludes by discussing the kinds of issues and tensions that emerged in the course of this work, as well as the ways that these issues and tensions can apply to much broader contexts of designing interactive algorithmic systems.

DOI | pdf

Speaking on Behalf of: Representation, Authority, and Delegation in Computational Text Analysis

Status

Eric P. S. Baumer and Micki McGee. (2019). Speaking on Behalf of: Representation, Authority, and Delegation in Computational Text Analysis. in Proceedings of the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES). (Honolulu, HI).

Abstract

Computational tools can often facilitate human work by rapidly summarizing large amounts of data, especially text. Doing so delegates to such models some measure of author- ity to speak on behalf of those people whose data are being analyzed. This paper considers the consequences of such delegation. It draws on sociological accounts of representation and translation to examine one particular case: the application of topic modeling to blogs written by parents of children on the autism spectrum. In doing so, the paper illustrates the kinds of statements that topic models, and other computational techniques, can make on behalf of people. It also articulates some of the potential consequences of such statements. The paper concludes by offering several suggestions about how to address potential harms that can occur when computational models speak on behalf of someone.

DOI

Interpretive Impacts of Text Visualization: Mitigating Political Framing Effects

Status

Eric P. S. Baumer, Jaime Snyder, and Gery Gay. (2018). Interpretive Impacts of Text Visualization: Mitigating Political Framing Effects. ACM Transactions on Human-Computer Interaction (ToCHI), 25(4), 20:1–20:26.

Abstract

Information visualizations are often evaluated as a tool in terms of their ability to support performance of a specific task. This article argues that value can be gained by instead evaluating visualizations from a communicative perspective. Specifically, it explores how text visualization can influence the impacts that framing has on the perception of political issues. Using data from a controlled laboratory study, the results presented here demonstrate that exposure to a text visualization can mitigate framing effects. Furthermore, it also shows a transfer effect, where participants who saw the visualization remained uninfluenced by framing in subsequent texts, even when the visualization was absent. These results carry implications for the methods used to evaluate information visualization systems, for understanding the cognitive and interpretive mechanisms by which framing effects occur, and for exploring the design space of interactive text visualization.

DOI

Comparing Grounded Theory and Topic Modeling: Extreme Divergence or Unlikely Convergence?

Status

Eric P. S. Baumer, David Mimno, Shion Guha, Emiy Quan, and Geri Gay. (2017). Comparing Grounded Theory and Topic Modeling: Extreme Divergence or Unlikely Convergence? Journal of the Association for Information Science and Technology (JASIST), 68(6): 1397–1410.

Abstract

Researchers in information science and related areas have developed various methods for analyzing textual data, such as survey responses. This article describes the application of analysis methods from two distinct fields, one method from interpretive social science and one method from statistical machine learning, to the same survey data. The results show that the two analyses produce some similar and some complementary insights about the phenomenon of interest, in this case, nonuse of social media. We compare both the processes of conducting these analyses and the results they produce to derive insights about each method’s unique advantages and drawbacks, as well as the broader roles that these methods play in the respective fields where they are often used. These insights allow us to make more informed decisions about the tradeoffs in choosing different methods for analyzing textual data. Further- more, this comparison suggests ways that such methods might be combined in novel and compelling ways.

DOI

Machine Learning and Grounded Theory Method: Convergence, Divergence, and Combination

Status

Michael Muller, Shion Guha, Eric P. S. Baumer, David Mimno, and N. Sadat Shami (2016). Machine Learning and Grounded Theory Method: Convergence, Divergence, and Combination. in Proceedings of the ACM Conference on Supporting Group Work (GROUP). (Sanibel Island, FL).

Abstract

Grounded Theory Method (GTM) and Machine Learning (ML) are often considered to be quite different. In this note, we explore unexpected convergences between these methods. We propose new research directions that can further clarify the relationships between these methods, and that can use those relationships to strengthen our ability to describe our phenomena and develop stronger hybrid theories.

DOI

Testing and Comparing Computational Approaches for Identifying the Language of Framing in Political News

Status

Eric P. S. Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, & Geri K. Gay. (2015). Testing and Comparing Computational Approaches for Identifying the Language of Framing in Political News. in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL – HLT) (pp. 1472–1482). Denver, CO.

Abstract

The subconscious influence of framing on perceptions of political issues is well-document in political science and communication research. A related line of work suggests that drawing attention to framing may help reduce such framing effects by enabling frame reflection, critical examination of the framing underlying an issue. However, definite guidance on how to identify framing does not exist. This paper presents a technique for identifying frame-invoking language. The paper first describes a human subjects pilot study that explores how individuals identify framing and informs the design of our technique. The paper then describes our data collection and annotation approach. Results show that the best performing classifiers achieve performance comparable to that of human annotators, and they indicate which aspects of language most pertain to framing. Both technical and theoretical implications are discussed.

pdf (ACL)

Data Set – This includes all 75 annotated articles, as well as descriptions of the format and instructions on use.

Broadening Exposure, Questioning Opinions, and Reading Patterns with Reflext: a Computational Support for Frame Reflection

Status

Eric P. S. Baumer, Claire Cipriani, Mitchell Davis, Gary He, Jaclyn Jeffrey-Wilensky, James Kang, Jinjoo Lee, Justin Zupnick, and Geri K. Gay. (2014). Broadening Exposure, Questioning Opinions, and Reading Patterns with Reflext: a Computational Support for Frame Reflection. Journal of Information Technology and Politics, 11(1), 45-63.

ABSTRACT

Vast amounts of political coverage are generated daily online. Some tools have been developed to help keep track of what is being said, but fewer efforts focus on how things are being said, i.e., how issues are framed. This article presents a study of Reflext, an interactive visualization tool that leverages computational linguistic analysis to support reflection on the framing of political issues. This system was deployed in a field study, during which the tool was used by regular readers of political news coverage during the 2012 U.S. election campaign. The results describe the tool’s support for a variety of activities related to frame reflection, how users integrated tool use with their existing reading practices, and broader issues in how participants interpreted the computational analysis and visualization. These findings contribute to our understanding of how algorithmically based interactive systems might mediate both the practical experiences of and situated interpretation of framing in political content.

DOI | pdf