SoUnD Framework: Analyzing (So)cial Representation in (Un)structured (D)ata

Abstract

The unstructured nature of data used in foundation model development is a challenge to systematic analyses for making data use and documentation decisions. From a Responsible AI perspective, these decisions often rely upon understanding how people are represented in data. We propose a framework designed to guide analysis of human representation in unstructured data and identify downstream risks. We apply the framework in two toy examples using the Common Crawl web text corpus (C4) and LAION-400M. We also propose a set of hypothetical action steps in service of dataset use, development, and documentation.

1 Introduction

Data is recognized as a core underlying factor contributing to machine learning model behaviours that are potentially unfair or harmful to humans [60]. Insights gleaned from systematic analysis of datasets can identify potential harms and inform interventions to mitigate risk. Principled analysis of the data underpinning pre-trained foundation models is particularly salient given the increasing reach of such models and their use by researchers and developers who lack the resources to develop computationally-intensive models [14, 36].

At the same time, the large, unstructured nature of datasets that underpin foundation models poses significant challenges for conducting analyses required to make development, documentation, and use decisions. The open-ended potential for downstream use, means that risks are wide-ranging and sometimes lack clear methods of evaluation [79], complicating systematic study. Prior systematic fairness audits and auditing tools have often focused on labeled datasets and utilized aggregated and disaggregated analyses to identify class imbalances (e.g., [67, 47, 50, 30]). Despite increased scrutiny of large unstructured datasets [12, 28], methods of analysis remain less robust and less systematic relative to labeled datasets, in part because labels provide a crucial pointer to dataset features to evaluate for fairness and bias concerns.

Our work closes this gap by contributing a conceptual framework to facilitate standardized analysis of unstructured data, in service of responsible data use and dataset development, use, and documentation. The framework (shown in Appendix A) focuses on how people are represented in data, including the range of data features that indicate social identity and influence the representation of different social and cultural groups– from identity terms that explicitly name social groups to recurring image objects implicitly associated with those groups. Analyses are grouped according to who is in the data, what is in the data, as well as associations between them. The framework’s focus on whether and how people are depicted means that the core analytical questions are not modality-specific and are extensible to new modalities and combinations as they emerge. The framework is designed to integrate into RAI workflows to support decision making for mitigation steps (e.g., data filtering, rebalancing) for a range of goals, such as new dataset development, existing dataset adaptation, and benchmark development.

2 Background

2.1 Dataset Transparency and Documentation

Transparency is a core focus in RAI, with a growing body of scholarship focusing on increasing transparency of AI systems and datasets for a variety of stakeholders. These range from developers who may build on pre-trained models to system end-users who may be subject to algorithmic decision making [53, 78]. At the dataset level, transparency serves to highlight critical information both about the contents of a dataset as well as the processes that underpin how a dataset was created. To this end, a range of work brings structured approaches to documenting both dataset content and development processes [10, 33, 28, 26, 44, 65, 75, 61]. However as massive, unstructured datasets increasingly become the norm in ML development, structured frameworks are needed to help summarize key characteristics of the data they capture. Dodge et al. [28] offer an expansive audit of C4 [62], which inspires a more structured approach for disentangling the contents of web-crawled data that feature heavily in ML datasets. Our work aims to standardize approaches to support existing transparency and documentation efforts by enabling the identification and communication of potential social risks associated with data.

2.2 Dataset Audits

Datasets underpinning ML training and testing have been at the center of a range of ethical controversies relating to privacy, consent, unfair system performance, representational harms, and harmful applications [60]. Against this backdrop, prominent ML datasets have been subject to close scrutiny, with empirical examinations and audits uncovering a range of problematic content that itself is a cause of harm (e.g., copyright violations; representational harms such as misgendering) or that can lead to downstream harms. For example, both image and text datasets have been shown to contain co-occurrence statistics that mirror harmful social biases and stereotypes [32, 39]; image datasets have been found to include problematic sexual imagery, including depictions of sexual violence and non-consensual sexual content, and racial and ethnic slurs within image labels and captions [11, 12]; text datasets have been found to contain biased and harmful sentiments towards marginalized identity groups [43, 28] and to exclude perspectives from marginalized groups [28]. Dataset audits can close documentation gaps [28], be used to make data filtering or re-balancing decisions [66], and, in some extreme cases, lead to the deprecation of datasets, such as MegaFace [48] and Tiny Images [76]. Organizing prior individual audits, we present a principled framework that supports dataset auditing to both shape dataset development decisions, as well as flag downstream model evaluations to prioritize.

2.3 Standardizing Responsible AI Workflows

Evaluating datasets in a structured way and effectively communicating results to various stakeholders remains an important challenge for RAI. As Sambasivan et al. [69] show, data work often takes a backseat to work focused on developing state of the art models and algorithms. In addition to lower incentive to engage in data work as a component of robust evaluation, current approaches to data documentation are “largely ad hoc and myopic in nature” [38] and practitioners face difficulty in understanding why documentation is needed, how best to document, and, ultimately, what to document [19]. A range of development toolkits and checklists have been proposed to address these challenges, including documentation frameworks such as Data and Model Cards [33, 58, 61], internal auditing frameworks [64], analysis tooling such as Know Your Data and WIMBD [29], and impact assessment frameworks [72]. RAI audits in particular require support to determine what to measure and how to measure it to avoid risks that can compound through development [69]. Mitchell
et al. [57] give a high-level framework for measuring large, unstructured datasets and we extend this by configuring our framework around social representation and demonstrate how the results of an audit might be used to distill dataset decisions.

3 Framework

In this section, we introduce a framework for systematic evaluation, anchoring on risk and harm associated with representations of humans in data. The framework supports analyses for a variety of goals ranging from dataset development to third-party audits. Our framework also identifies a set of components that guide the operationalization of each analysis and the interpretation of their results.

3.1 Framework analyses

Figure 1 demonstrates the framework’s conceptual structure. We organize the framework around high-level questions about human-centered considerations in data: namely, Who is in the data, What is in the data, and How are the two associated? This structure also allows the analyses to focus on data questions at different levels of complexity with respect to corresponding downstream harms, as well as to prevent an over-focus on optimizing isolated analyses or metrics.

Our framework is designed to be general-purpose and extensible, thus it is not exhaustive of every single possible analysis within each section. While we use text, image, and image-text datasets as references for developing and describing the framework, it can be adapted to other modalities with appropriate changes. For instance, the dialect analysis in text could be adjusted to include elements of (or complemented with) analysis of accent in speech data. Analyses can also be modified, added, or removed as the field’s collective sociotechnical understanding about relevant social biases evolve over time, while preserving the overall framework structure. For example, salient social identity term lists may iteratively change as best practices respond to social shifts, or as global socio-cultural contexts are increasingly integrated into RAI considerations. Analyses can also be updated alongside our understanding of salient social risks, the human social characteristics they are connected to, and our technical means of analyzing them. However, the motivating questions remain stable.

Figure 1: Framework conceptual structure. The combination of Human Factor and Content Factor analyses provided disaggregated associations. Analyses documentation and results are on the right.

3.1.1 Who is in the data?

In asking who is in the data, we consider several human factors of data that include measuring the presence of people in data along with social characteristics.

Presence of People:
The analyses of people’s presence specifically tally whether individuals or identifying information appear in data. This includes calculations of personally-identifiable information and face or person detection.

Extending to new modalities, the analyses implicitly ask which data characteristics can indicate the presence of a person, such as faces or bodies in visual data or voice in audio data. Results guide more focused, follow up analyses that assess depictions of social groups.

Social Characteristics:
The analyses of social characteristics in data center on data characteristics that are often associated with social identity and may be used as proxies for social identity. Some proxies appear directly in data, such as pronouns, while others, such as perceived age or gender expression in images, must be inferred, frequently using predictive methods (e.g., Lanitis
et al. [51]). These include analyses of dialect, linguistic style, skin tone, and voice pitch. Social characteristic analyses provides insight into the over- and under-representation of specific social groups, which has been associated with disparities in performance [80, 16] and general problems for class prediction [46]. Because these characteristics are social in nature, their measurement must be localized. For example, social identity terms and gender presentation vary across social and cultural contexts.

3.1.2 What is in the data?

The second high-level grouping of analyses focuses on content that may heighten the sensitivity of human representation as a whole.

Content: This group of analyses is focused on content characteristics that relate to harmful or undesirable outcomes that are independent of specific people or social groups. Analyses include calculating the distribution of topics in text, as well as sexual content in images. Topic distribution provides a bird’s-eye view of the composition of the data and can give an indication of sexually explicit or sensitive topics contained in a dataset. Topic distributions can give clues to subtle downstream biases. For example, models trained primarily on news data have been shown to exhibit biases against particular country names and professions [42].

Provenance: Data provenance can indicate what is likely to be contained in the data. This includes values, norms, and perspectives in data ascertained through metadata, such as the geographic distribution of sources and their publication dates. For example, source URL domains point to the range of content represented in web-scraped data, which offers insight into document content, such as linguistic and cultural content, as well as the prevalence of machine-generated text [28]. The geographic, cultural, and social representation in data can have implications for downstream models. For example, image classifiers trained on datasets sourced predominantly from western countries have lower rates of accuracy when applied to images from non-western countries [74]. Data recency can have particular impacts on models supporting low-resource languages, which can disproportionately rely on religious or historical texts due to data scarcity (e.g., Ahmadi and Masoud [3]).

3.1.3 Human × Content Associations

The final section of the focuses on associations between human and content factors in data. Each standalone section of the framework is valuable, however, associations reveal how people are depicted. Associations disaggregate analyses within and across modalities, such as social identity terms and topics in text (for language datasets) or occurrences between objects detected in images and identity terms in associated text (for multimodal datasets). Associations can reveal stereotype-aligned correlations, which can amplify the stereotypes and propagate exclusionary norms [24, 79, 81, 39].

While highly specific combinations of analyses can be run (e.g., an evaluation of queer depictions in Spanish-language medical literature from a specific year), the structure of the framework facilitates analyses beginning with the most general question (i.e., are people depicted?) followed by more specific inquiries to provide a tractable entry point for RAI analyses.

3.2 Framework Components

Next, we outline additional framework components that guide analysis results reporting and general analysis planning. The Output and Action fields are provided to capture the results of a given analysis and any mitigation actions decided in response. The Taking Action section discusses in more depth the process of making mitigation decisions. In addition to a research-backed motivation related to downstream risks, each analysis includes additional fields to support planning:

Analysis Object indicates whether an analysis is calculated on data directly (i.e., tokens in text data; source urls) or if an analysis applies to an inference produced by an intermediate classifier (e.g., inferred document topic; predicted age of person in an image). This distinction highlights which analyses are dependent on predictive models and therefore susceptible to biases that those predictive models may themselves exhibit. The distinction between “Image” and “Inferred image signals” is particularly important since few of the analyses in the framework are applied to image data directly.

Effort indicates estimated time and cost of an analysis based on current techniques and available tooling, which, in turn, reflect bias toward use for English language and Western data. In non-Western contexts, effort is likely higher for technical implementation as well as making determinations about which identity groups should be prioritized.

Dependencies indicates intermediate resources needed to conduct an analysis, such as identity term lists or classifiers which produce inferred signals. While the framework does not dictate a required implementation for any analysis, we point to example classifiers and term lists that may be used.

4 Taking Action

The framework is not meant to be exhaustively implemented for every use case, since all possible Association analyses would produce an intractably large number of results. Moreover, which mitigation actions to take depends highly on context: the downstream effects of dataset filtering and rebalancing are, in many cases, still an open question, and there are a range of other actions that may be applicable. Finally, while this framework can be used to uncover previously unanticipated biases, we expect that in practice, practitioners will have high-level aims informed by institutional policies and previously documented data biases. An exhaustive review of mitigation actions is beyond the scope of this work, however we describe key considerations that inform the actions a user should take. We also include a selection of guiding questions and considerations at the start of the full framework.

Important considerations are 1) dataset purpose (e.g., training or evaluation data) 2) analysis goals (e.g., auditing a third-party dataset or developing a new evaluation benchmark), and 3) development steps at which interventions can be applied (e.g., data collection, model evaluation, documentation).

Dataset Purpose: The framework can be applied to a range of dataset types including pre-training or fine-tuning datasets. It can also be used for understanding and evaluating model-generated data. Suitable framework actions depend on the purpose of the dataset. For example, when analyzing pre-training data, it may be unclear how changes to data distributions will impact model performance, potentially making other mitigations more desirable. In contrast, data used for benchmark development stands to be used as a repeated measure of model robustness and performance. Thus, actions that might require additional costs or resources may be more easily justified to meet evaluation goals.

Table 1: A non-exhaustive list of data analysis goals.

Analysis Goals: A range of goals can motivate framework use– each of which brings attention to different actions. Table 1 lists common goals of dataset analyses. For example, developing a new dataset from web-scraped data raises potential decisions to collect additional data or adjust filtering criteria in the data collection process. In contrast, conducting an audit of a third-party dataset for compliance purposes brings focus to documentation and data use decisions.

Table 2: A non-exhaustive list of actions that may be taken to address social risks identified in data.

Development Phase: Each development phase affords different actions to address data concerns. Table 2 shows common actions by development phase. For example, during data collection, toxic content biased across social identity groups might be addressed by modifying the dataset or by adjusting model evaluation planning. Alternatively, documentation can flag concerns for public consumption. Moreover, concerns may be addressed through explicit dataset release decisions.

The decision to pursue an action such as the ones listed above will depend on analysis goals, available resource play a key role in determining the mitigation actions that are available. Direct action on a dataset may not be possible, for example if there are cost constraints or if data sources and filtering techniques used to develop a third-party dataset are not clearly defined or known.

5 Applying the Framework

In order to showcase how the framework can guide evaluations, we provide two toy examples applying it to C4 [62] and LAION-400M [73]– two large, unstructured datasets available under a CC-BY 4.0 license. We apply the analyses from the standpoint of an individual or team seeking to repurpose the dataset for their own use. That is, our planned output is a training dataset and our goal is to develop derivative datasets from C4 and LAION-400M while assessing representational biases that have been previously identified in text and image datasets. We do not present every possible analysis; instead, we focus on a few key results. We do this for two reasons. First, not all analyses are relevant for understanding a specific social group or modality. Thus, in practice, only a selection of analyses will be conducted. Second, some analyses are not yet technically feasible and techniques to achieve them are themselves the subject of research (e.g., detecting hateful symbols and memes in images [55]). Finally, because we primarily focus on how to use results across analyses to inform risk mitigation, we do not delve into technical details or performance metrics for the classifiers we use.

Dataset Analysis Output: C4

Age Terms X Top Tokens:

Analysis Object:

Text

Task:

For social identity terms in text, calculate top word co-occurrences for each.

Effort:

Low

Dependencies:

None

Action:

Flag associations. Consider rebalancing harmful associations that emerge in model evaluations

Output:

Ageing

anti-aging, wrinkles, collagen, elasticity, age-related

Old Age

age-related, middle-aged, degeneration, dementia, ripe

Elderly

dementia, ageing, older, alzheimer, frail

Age Terms X Topic:

Task:

Calculate the distribution of topics within text, disaggregated by social identity terms or inferred social identity signals

Analysis Object:

Inferred Text Signals + Text

Effort:

Low

Dependencies:

Topic Classifier

Action:

Flag problematic associations

Output:

Ageing

Medical Lit, Skin Care, Face Care, Anti-Ageing, Nutrition

Old Age

Medical Lit, Books/Lit, Health Cond., History, Retirement

Elderly

Medical Lit, Retirement, Assisted Living, Geriartics, Health Cond.

Figure 2: Sample dataset analyses output for C4.

5.1 Evaluating Age Representation in C4

We motivate our analyses based on prior research that identifies age bias as an issue for ML and AI development [25, 31] and calls for increased representation of older adults in AI datasets [59]. With this in mind, we turn to assessing how older adults are depicted in C4 using Association analyses. We assess the results of age depictions via top-associated tokens and topics with age-related terms.

Output: We see in Figure 2 the tokens most associated with old age terms, which occur 110,000 times in the dataset. These include dementia, and degeneration, both of which can render negative sentiment. We see related associations in the topics disproportionately associated with old age terms. These include health topics, including medical conditions and assisted living, as well as skin and face care beauty products, which likely point to content covering anti-ageing products and discussion.

Action: In line with prior work, we find both low and skewed representation of older age in text. Some work has been conducted on decoupling adjective associations from select identity terms [24], however broader sentential context surrounding age-related terms may still carry negative or stigmatized sentiment. Filtering or removing data stands to worsen older adult underrepresentation in ML datasets, however it is among the lowest cost options. Toward the goal of developing training data, other actions may be taken. If a data collection or generation pipeline is feasible to pursue, targeted data collection or synthetic data can be used to rebalance the data, ideally through ablation studies to understand how model performance changes. Analysis results can also be flagged to evaluate the downstream model and the data it generates for similar biases to assess model limitations. Contingent on these evaluations, documentation warnings or non-use in certain cases may also be necessary.

Dataset Analysis Output: LAION-400M

Queer Identity Terms X Sexual Imagery:

Analysis Object:

Text | Inferred Image Signals

Task:

Calculate co-occurrences between social identity terms and sexual imagery.

Effort:

Low

Dependencies:

Visual Content Classifier

Action:

Data filtering or tagging. Possible rebalancing

Output:

Queer Identity Terms X Topic:

Task:

Calculate the distribution of topics by queer identity terms

Analysis Object:

Inferred Text Signals, Text

Effort:

Low

Dependencies:

Topic Classifier

Action:

Follow up analyses of sexual content

Output:

Lesbian

Adult Videos, LGBT Porn, Porn

Gay

LGBT, LGBT Porn, Online Images

Bisexual

LGBT, Porn, Online Images

Transgender

LGBT, Social Issues, Discrimination

Queer

LGBT, Events/Listings, Online Images

Heterosexuality

Porn, Adult Videos, Porn

Figure 3: Sample dataset analyses output for LAION-400M Dataset.

5.2 Evaluating Queer Representation in LAION-400M

LAION-400M is a multimodal text and image dataset that features over 400 million image-text pairs extracted from Common Crawl, prominently used in text-to-image generation. The dataset is unstructured and uncurated, though it does feature NSFW tags, which were used to identify and filter a number of illicit images.
Text-image datasets are shown to reflect various societal biases; for example, DALL-E and its variants reproduce gender and skin tone biases from neutral prompts [21]. We apply the framework to LAION-400M to analyze such multimodal and text associations.

Researchers have found unintended and undesirable associations with queer identity terms in text datasets [27] and sexually explicit depictions of women in LAION-400M [11, 12]. While explicit content can be used for specific applications, unintentional inclusion risks accidental generation of explicit content by downstream models. We assess a combination of these biases by evaluating queer representation, again considering mitigations for adapting a third-party training dataset. We look to the results of Association analyses in text using the same topic classifier from our prior analyses, and we run multimodal Association analyses using a classifier similar to [35], which identifies sexual content in images. We analyze a collection of social identity terms via webref entities.

Output: The top-associated tokens with each queer identity term, which can be seen in Figure 3, largely reference other queer identity terms. Figure 3 shows the top topics associated with a variety of sexual identities. Prominent among these are those that seemingly refer to various sexual topics and activities. This includes for "heterosexuality". Interestingly, the most frequent topic for "heterosexuality" is "LGBT Porn" which suggests that the term is connected to a subgenre of pornographic videos. Though not a sexual identity, the generally derogatory term "transsexual" is also strongly associated with sexual terms and topics.

Action: Considering dataset usage for T2I training in particular, there is likely a very limited set of use cases in which the generation of sexual content would be appropriate. Such use cases would likely entail very specialized dataset curation and model development. Therefore, one could consider filtering sexual content in order to both limit its downstream production as well as to avoid the inclusion of published sexual content, which has often been made public without sex worker consent [22]. Because filtering may lead to removal of nearly half of the instances of some identity terms, rebalancing may also be needed. Alternatively, sexual content in text data could be augmented with tags to preserve a downstream model’s ability to detect it while limiting its production.

6 Evaluating Representation in Data

In line with Mitchell
et al. [57]’s call to establish practices for measuring data, characterizing how people are represented in data is a necessary part of identifying downstream risks. Yet, RAI lacks systematized guidance to address the myriad ways that traces of social identity emerge in explicitly and implicitly across data modalities. Up to this point, there has been little guidance for triangulating social identity in datasets across different analyses and approaches that use combinations of features to measure a latent representation of social identity have not yet been widely applied in RAI work.

Notably, our framework omits both a canonical list of social identities to analyze as well as an exhaustive list of methods to measure a given social identity. We do this because social identity is unstable nature [37]. Beyond specific data modalities, the characteristics associated with social identities change with context and, in a single context, the same features can be associated with more than one identity group. For example, Hispanic surnames are prevalent in both Latin America and the Philippines, meaning an analysis of cultural diversity cannot rely on these features alone. Scholars in RAI have shown that the axes along which discrimination occurs are culturally specific [68].

Analyzing how people are depicted is additionally challenging because "good" social representation can change with context. As Chasalow and Levy [20] posit, representativeness is both time and place specific. The social categories we attend to are shaped both by normative assumptions about what should be measured as well as the existence of a name for or conception of a social category. For example, Andrews
et al. [5]’s research suggests that a word list generated today to analyze disability representation in a dataset would likely feature different terminology than a list generated 30 years ago. As new language and visual cues emerge to describe existing or newly salient social groups, our approaches to analyzing group representation must be updated. Our framework aims to address these shifts with analyses that are repeatable and updateable with emergent social identity cues.

6.1 The Role of Datasets in Assessing Harm

In developing this framework to support development decisions, we extend the work of others advocating for more attention to data work [69], including the growing focus on data-centric AI (DCAI) [45]. Building from DCAI’s focus on understanding the data used and produced throughout ML development, our framework sets a foundation for systematically analyzing social risks in data. If DCAI is focused broadly on shifting focus from the model to the data, our framework emphasizes human-centered angle within that focus. Some work in DCAI does bring attention to data sources and the sociocultural views they represent, such as those expressed through data annotation [26, 7, 56]. However, this work has limited application to unstructured data.

An important part of dataset evaluation is determining when a data distribution is problematic. Future work in DCAI and RAI should explore the effects of different distributions on model performance and output bias. This work points to opportunities to use the framework to scaffold experiments to study the effects of different data distributions on model performance across contexts. In this way the framework is a concrete aid to what Jarrahi
et al. [45] calls "data benchmarking". Across iterative development of the same model, as well as across developments of distinct models, the framework can act as a consistent measuring stick for relating representations in data to model fairness.

At the same time, dataset evaluations are just one component of RAI evaluation. Much algorithmic fairness work focuses on data sources and dataset pre-processing; however, as Hooker [40] argues, algorithm design choices, such as optimization for privacy guarantees, compression techniques, and even learning rate can contribute to model biases. Hooker also critically points out that dataset evaluations rely on a priori decisions about which features to evaluate and are inherently informed by human biases regarding what should or should not be prioritized. As a result, dataset evaluations must be considered alongside other approaches to mitigating risk and harm. For example, Hooker turns to model compression techniques to isolate data points at risk of exacerbated error rates as a way to guide further auditing. This enables iterative error analysis that DCAI calls for.

6.2 Supporting RAI Goals

RAI development requires dataset analyses that integrate into existing RAI workflows in ways that complement existing tools and frameworks. This includes structured guidance to repeatedly apply analyses of human depictions in data. Our framework supports the development of transparency artifacts by standardizing results and by flagging benchmark tests to prioritize based on problematic data distributions. In this context, the framework stands as a structured auditing aid.

How and when to act on analysis results depends on the nature of the data being used as well as risk mitigation goals. In other words, the same results can warrant different action, depending on context. A primary use motivating the design of this framework is to analyze datasets used for foundation models. The high cost of model training limits opportunities to run comprehensive studies to identify which mitigation strategies improve fairness issues as well as their impacts model performance generally. For RAI, this means making mitigation decisions with limited information about specific impacts. This challenge is exacerbated by data cascades, which can compound to produce out-sized, negative outcomes [69]. Yet, the range of potential downstream risks warrants proactive decision making. While decision making at the training data level is difficult when downstream applications are unclear, the framework provides valuable information about the people represented in data, and can be used to guide later evaluations of fine-tuning data or model-generated data.

7 Conclusion

The open-ended nature of AI risks and harms poses challenges to RAI practitioners seeking to not only identify risks, but also take appropriate action to mitigate them, at times with limited information about how downstream models will be fine-tuned or applied. In response, we propose a standardized framework to evaluate unstructured datasets for downstream risk, with a focus on human representation. Building from critical dataset audits and concurrent work standardizing these efforts (such as by Elazar
et al. [29]), we organize our framework around social representation and provide exemplar uses to demonstrate its application. Our framework is designed as a general-purpose starting point that is extensible to other modalities and application contexts, as needed.

References

• [2]

Abubakar Abid, Maheen
Farooqi, and James Zou.
2021.

Persistent anti-muslim bias in large language
models. In Proceedings of the 2021 AAAI/ACM
Conference on AI, Ethics, and Society. 298–306.

• [3]

Sina Ahmadi and Mariam
Masoud. 2020.

Towards machine translation for the Kurdish
language.

arXiv preprint arXiv:2010.06041
(2020).

• [4]

Miltiadis Allamanis.
2019.

The adverse effects of code duplication in machine
learning models of code. In Proceedings of the
2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and
Reflections on Programming and Software. 143–153.

• [5]

Erin E Andrews, Robyn M
Powell, and Kara Ayers.
2022.

The evolution of disability language: Choosing
terms to describe disability.

Disability and Health Journal
15, 3 (2022),
101328.

• [6]

Rohan Anil, Andrew M.
Dai, Orhan Firat, Melvin Johnson,
Dmitry Lepikhin, Alexandre Passos,
Siamak Shakeri, Emanuel Taropa,
Paige Bailey, Zhifeng Chen,
Eric Chu, Jonathan H. Clark,
Laurent El Shafey, Yanping Huang,
Kathy Meier-Hellstern, Gaurav Mishra,
Erica Moreira, Mark Omernick,
Kevin Robinson, Sebastian Ruder,
Yi Tay, Kefan Xiao,
Yuanzhong Xu, Yujing Zhang,
Gustavo Hernandez Abrego, Junwhan Ahn,
Jacob Austin, Paul Barham,
Jan Botha, James Bradbury,
Siddhartha Brahma, Kevin Brooks,
Michele Catasta, Yong Cheng,
Colin Cherry, Christopher A.
Choquette-Choo, Aakanksha Chowdhery,
Clément Crepy, Shachi Dave,
Mostafa Dehghani, Sunipa Dev,
Jacob Devlin, Mark Díaz,
Nan Du, Ethan Dyer, Vlad
Feinberg, Fangxiaoyu Feng, Vlad Fienber,
Markus Freitag, Xavier Garcia,
Sebastian Gehrmann, Lucas Gonzalez,
Guy Gur-Ari, Steven Hand,
Hadi Hashemi, Le Hou,
Joshua Howland, Andrea Hu,
Jeffrey Hui, Jeremy Hurwitz,
Michael Isard, Abe Ittycheriah,
Matthew Jagielski, Wenhao Jia,
Kathleen Kenealy, Maxim Krikun,
Sneha Kudugunta, Chang Lan,
Katherine Lee, Benjamin Lee,
Eric Li, Music Li, Wei
Li, YaGuang Li, Jian Li,
Hyeontaek Lim, Hanzhao Lin,
Zhongtao Liu, Frederick Liu,
Marcello Maggioni, Aroma Mahendru,
Joshua Maynez, Vedant Misra,
Maysam Moussalem, Zachary Nado,
John Nham, Eric Ni,
Andrew Nystrom, Alicia Parrish,
Marie Pellat, Martin Polacek,
Alex Polozov, Reiner Pope,
Siyuan Qiao, Emily Reif,
Bryan Richter, Parker Riley,
Alex Castro Ros, Aurko Roy,
Brennan Saeta, Rajkumar Samuel,
Renee Shelby, Ambrose Slone,
Daniel Smilkov, David R. So,
Daniel Sohn, Simon Tokumine,
Dasha Valter, Vijay Vasudevan,
Kiran Vodrahalli, Xuezhi Wang,
Pidong Wang, Zirui Wang,
Tao Wang, John Wieting,
Yuhuai Wu, Kelvin Xu,
Yunhan Xu, Linting Xue,
Pengcheng Yin, Jiahui Yu,
Qiao Zhang, Steven Zheng,
Ce Zheng, Weikang Zhou,
Denny Zhou, Slav Petrov, and
Yonghui Wu. 2023.

PaLM 2 Technical Report.

arXiv:cs.CL/2305.10403

• [7]

Kofi Arhin, Ioana
Baldini, Dennis Wei, Karthikeyan Natesan
Ramamurthy, and Moninder Singh.
2021.

Ground-Truth, Whose Truth?–Examining the
Challenges with Annotating Toxic Text Datasets.

arXiv preprint arXiv:2112.03529
(2021).

• [8]

Jack Bandy and Nicholas
Vincent. 2021.

Addressing" documentation debt" in machine learning
research: A retrospective datasheet for bookcorpus.

arXiv preprint arXiv:2105.05241
(2021).

• [9]

Solon Barocas, Kate
Crawford, Aaron Shapiro, and Hanna
Wallach. 2017.

The Problem With Bias: Allocative Versus
Representational Harms in Machine Learning. SIGCIS.

• [10]

Emily M Bender and Batya
Friedman. 2018.

Data statements for natural language processing:
Toward mitigating system bias and enabling better science.

Transactions of the Association for
Computational Linguistics 6 (2018),
587–604.

• [11]

Abeba Birhane and
Vinay Uday Prabhu. 2021.

Large image datasets: A pyrrhic win for computer
vision?. In 2021 IEEE Winter Conference on
Applications of Computer Vision (WACV). IEEE, 1536–1546.

• [12]

Abeba Birhane, Vinay Uday
Prabhu, and Emmanuel Kahembwe.
2021.

Multimodal datasets: misogyny, pornography, and
malignant stereotypes.

arXiv preprint arXiv:2110.01963
(2021).

• [13]

Tolga Bolukbasi, Kai-Wei
Chang, James Y Zou, Venkatesh Saligrama,
and Adam T Kalai. 2016.

Man is to computer programmer as woman is to
homemaker? debiasing word embeddings.

Advances in neural information processing
systems 29 (2016).

• [14]

Rishi Bommasani, Drew A
Hudson, Ehsan Adeli, Russ Altman,
Simran Arora, Sydney von Arx,
Michael S Bernstein, Jeannette Bohg,
Antoine Bosselut, Emma Brunskill,
et al. 2021.

On the opportunities and risks of foundation
models.

arXiv preprint arXiv:2108.07258
(2021).

• [15]

Tom Brown, Benjamin Mann,
Nick Ryder, Melanie Subbiah,
Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam,
Girish Sastry, Amanda Askell,
et al. 2020.

Language models are few-shot learners.

Advances in neural information processing
systems 33 (2020),
1877–1901.

• [16]

Joy Buolamwini and
Timnit Gebru. 2018.

Gender shades: Intersectional accuracy disparities
in commercial gender classification. In Conference
on fairness, accountability and transparency. PMLR,
77–91.

• [17]

Aylin Caliskan, Joanna J
Bryson, and Arvind Narayanan.
2017.

Semantics derived automatically from language
corpora contain human-like biases.

Science 356,
6334 (2017), 183–186.

• [18]

Nicholas Carlini, Florian
Tramer, Eric Wallace, Matthew Jagielski,
Ariel Herbert-Voss, Katherine Lee,
Adam Roberts, Tom B Brown,
Dawn Song, Ulfar Erlingsson,
et al. 2021.

Extracting Training Data from Large Language
Models.. In USENIX Security Symposium,
Vol. 6.

• [19]

Jiyoo Chang and
Christine Custis. 2022.

Understanding Implementation Challenges in Machine
Learning Documentation.

In Equity and Access in Algorithms,
Mechanisms, and Optimization. 1–8.

• [20]

Kyla Chasalow and Karen
Levy. 2021.

Representativeness in statistics, politics, and
machine learning. In Proceedings of the 2021 ACM
Conference on Fairness, Accountability, and Transparency.
77–89.

• [21]

Jaemin Cho, Abhay Zala,
and Mohit Bansal. 2022.

Dall-eval: Probing the reasoning skills and social
biases of text-to-image generative transformers.

arXiv preprint arXiv:2202.04053
(2022).

• [22]

Samantha Cole.
2020.

These Nudes Do Not Exist and I Don’t Know Why This
Startup Does Either. In Vice.

• [23]

Kate Crawford and Trevor
Paglen. 2021.

Excavating AI: The politics of images in machine
learning training sets.

Ai & Society 36,
4 (2021), 1105–1116.

• [24]

Sunipa Dev, Tao Li,
Jeff M Phillips, and Vivek Srikumar.
2021.

OSCaR: Orthogonal Subspace Correction and
Rectification of Biases in Word Embeddings. In
Proceedings of the 2021 Conference on Empirical
Methods in Natural Language Processing. Association for
Computational Linguistics, Online and Punta Cana,
Dominican Republic, 5034–5050.

• [25]

Mark Díaz, Isaac
Johnson, Amanda Lazar, Anne Marie Piper,
and Darren Gergle. 2018.

Addressing age-related bias in sentiment analysis.
In Proceedings of the 2018 chi conference on human
factors in computing systems. 1–14.

• [26]

Mark Díaz, Ian
Kivlichan, Rachel Rosen, Dylan Baker,
Razvan Amironesei, Vinodkumar
Prabhakaran, and Emily Denton.
2022.

Crowdworksheets: Accounting for individual and
collective identities underlying crowdsourced dataset annotation. In
2022 ACM Conference on Fairness, Accountability,
and Transparency. 2342–2351.

• [27]

Lucas Dixon, John Li,
Jeffrey Sorensen, Nithum Thain, and
Lucy Vasserman. 2018.

Measuring and mitigating unintended bias in text
classification. In Proceedings of the 2018
AAAI/ACM Conference on AI, Ethics, and Society. 67–73.

• [28]

Jesse Dodge, Maarten Sap,
Ana Marasović, William Agnew,
Gabriel Ilharco, Dirk Groeneveld,
Margaret Mitchell, and Matt Gardner.
2021.

Documenting large webtext corpora: A case study on
the colossal clean crawled corpus.

arXiv preprint arXiv:2104.08758
(2021).

• [29]

Yanai Elazar, Akshita
Bhagia, Ian Magnusson, Abhilasha
Ravichander, Dustin Schwenk, Alane Suhr,
Pete Walsh, Dirk Groeneveld,
Luca Soldaini, Sameer Singh,
et al. 2023.

What’s In My Big Data?

arXiv preprint arXiv:2310.20707
(2023).

• [30]

Sorelle A Friedler, Carlos
Scheidegger, Suresh Venkatasubramanian,
Sonam Choudhary, Evan P Hamilton, and
Derek Roth. 2019.

A comparative study of fairness-enhancing
interventions in machine learning. In Proceedings
of the conference on fairness, accountability, and transparency.
329–338.

• [31]

Adriana Solange Garcia de Alford,
Steven K Hayden, Nicole Wittlin, and
Amy Atwood. 2020.

Reducing Age Bias in Machine Learning: An
Algorithmic Approach.

SMU Data Science Review
3, 2 (2020),
11.

• [32]

Nikhil Garg, Londa
Schiebinger, Dan Jurafsky, and James
Zou. 2018.

Word embeddings quantify 100 years of gender and
ethnic stereotypes.

Proceedings of the National Academy of
Sciences 115, 16
(2018), E3635–E3644.

• [33]

Timnit Gebru, Jamie
Morgenstern, Briana Vecchione,
Jennifer Wortman Vaughan, Hanna Wallach,
Hal Daumé Iii, and Kate Crawford.
2021.

Datasheets for datasets.

Commun. ACM 64,
12 (2021), 86–92.

• [34]

Samuel Gehman, Suchin
Gururangan, Maarten Sap, Yejin Choi,
and Noah A Smith. 2020.

Realtoxicityprompts: Evaluating neural toxic
degeneration in language models.

arXiv preprint arXiv:2009.11462
(2020).

• [35]

Google Cloud Vision.
[n.d.].

Image Annotation.

• [36]

Xu Han, Zhengyan Zhang,
Ning Ding, Yuxian Gu,
Xiao Liu, Yuqi Huo,
Jiezhong Qiu, Yuan Yao,
Ao Zhang, Liang Zhang, et al.
2021.

Pre-trained models: Past, present and future.

AI Open 2
(2021), 225–250.

• [37]

Alex Hanna, Emily Denton,
Andrew Smart, and Jamila Smith-Loud.
2020.

Towards a critical race methodology in algorithmic
fairness. In Proceedings of the 2020 conference on
fairness, accountability, and transparency. 501–512.

• [38]

Amy K Heger, Liz B
Marquis, Mihaela Vorvoreanu, Hanna
Wallach, and Jennifer Wortman Vaughan.
2022.

Understanding Machine Learning Practitioners’ Data
Documentation Perceptions, Needs, Challenges, and Desiderata.

Proceedings of the ACM on Human-Computer
Interaction 6, CSCW2
(2022), 1–29.

• [39]

Lisa Anne Hendricks,
Kaylee Burns, Kate Saenko,
Trevor Darrell, and Anna Rohrbach.
2018.

Women also snowboard: Overcoming bias in captioning
models. In Proceedings of the European conference
on computer vision (ECCV). 771–787.

• [40]

Sara Hooker.
2021.

Moving beyond “algorithmic bias is a data
problem”.

Patterns 2,
4 (2021), 100241.

• [41]

Dirk Hovy and Shrimai
Prabhumoye. 2021.

Five sources of bias in natural language
processing.

Language and Linguistics Compass
15, 8 (2021),
e12432.

• [42]

Po-Sen Huang, Huan Zhang,
Ray Jiang, Robert Stanforth,
Johannes Welbl, Jack Rae,
Vishal Maini, Dani Yogatama, and
Pushmeet Kohli. 2019.

Reducing sentiment bias in language models via
counterfactual evaluation.

arXiv preprint arXiv:1911.03064
(2019).

• [43]

Ben Hutchinson, Vinodkumar
Prabhakaran, Emily Denton, Kellie
Webster, Yu Zhong, and Stephen
Denuyl. 2020.

Social biases in NLP models as barriers for persons
with disabilities.

arXiv preprint arXiv:2005.00813
(2020).

• [44]

Ben Hutchinson, Andrew
Smart, Alex Hanna, Emily Denton,
Christina Greer, Oddur Kjartansson,
Parker Barnes, and Margaret Mitchell.
2021.

Towards Accountability for Machine Learning
Datasets: Practices from Software Engineering and Infrastructure. In
Proceedings of the 2021 ACM Conference on Fairness,
Accountability, and Transparency (FAccT ’21).
Association for Computing Machinery,
New York, NY, USA, 560–575.

• [45]

Mohammad Hossein Jarrahi,
Ali Memariani, and Shion Guha.
2022.

The Principles of Data-Centric AI (DCAI).

arXiv preprint arXiv:2211.14611
(2022).

• [46]

Justin M Johnson and
Taghi M Khoshgoftaar. 2019.

Survey on deep learning with class imbalance.

Journal of Big Data 6,
1 (2019), 1–54.

• [47]

Michael Kearns, Seth
Neel, Aaron Roth, and Zhiwei Steven
Wu. 2018.

Preventing fairness gerrymandering: Auditing and
learning for subgroup fairness. In International
conference on machine learning. PMLR, 2564–2572.

• [48]

Ira Kemelmacher-Shlizerman,
Steven M Seitz, Daniel Miller, and
Evan Brossard. 2016.

The megaface benchmark: 1 million faces for
recognition at scale. In Proceedings of the IEEE
conference on computer vision and pattern recognition.
4873–4882.

• [49]

Mahrukh Khan,
Muhammad Atif Tahir, and Zeeshan
Ahmed. 2018.

Detection of violent content in cartoon videos
using multimedia content detection techniques. In
2018 IEEE 21st International Multi-Topic Conference
(INMIC). IEEE, 1–5.

• [50]

Jon Kleinberg, Sendhil
Mullainathan, and Manish Raghavan.
2016.

Inherent trade-offs in the fair determination of
risk scores.

arXiv preprint arXiv:1609.05807
(2016).

• [51]

Andreas Lanitis, Chrisina
Draganova, and Chris Christodoulou.
2004.

Comparing different classifiers for automatic age
estimation.

IEEE Transactions on Systems, Man, and
Cybernetics, Part B (Cybernetics) 34, 1
(2004), 621–628.

• [52]

Katherine Lee, Daphne
Ippolito, Andrew Nystrom, Chiyuan Zhang,
Douglas Eck, Chris Callison-Burch, and
Nicholas Carlini. 2021.

Deduplicating training data makes language models
better.

arXiv preprint arXiv:2107.06499
(2021).

• [53]

Gabriel Lima, Nina
Grgić-Hlača, Jin Keun Jeong, and
Meeyoung Cha. 2022.

The Conflict Between Explainable and Accountable
Decision-Making Algorithms. In 2022 ACM Conference
on Fairness, Accountability, and Transparency. 2103–2113.

• [54]

Alexandra Sasha Luccioni and
Joseph D Viviano. 2021.

What’s in the Box? A Preliminary Analysis of
Undesirable Content in the Common Crawl Corpus.

arXiv preprint arXiv:2105.02732
(2021).

• [55]

Lambert Mathias, Shaoliang
Nie, Aida Mostafazadeh Davani, Douwe
Kiela, Vinodkumar Prabhakaran, Bertie
Vidgen, and Zeerak Waseem.
2021.

Findings of the WOAH 5 shared task on fine grained
hateful memes detection. In Proceedings of the 5th
Workshop on Online Abuse and Harms (WOAH 2021). 201–206.

• [56]

Abhilash Mishra and Yash
Gorana. 2021.

Who Decides if AI is Fair? The Labels Problem in
Algorithmic Auditing.

arXiv preprint arXiv:2111.08723
(2021).

• [57]

Margaret Mitchell,
Alexandra Sasha Luccioni, Nathan Lambert,
Marissa Gerchick, Angelina
McMillan-Major, Ezinwanne Ozoani, Nazneen
Rajani, Tristan Thrush, Yacine Jernite,
and Douwe Kiela. 2022.

Measuring Data.

arXiv preprint arXiv:2212.05129
(2022).

• [58]

Margaret Mitchell, Simone
Wu, Andrew Zaldivar, Parker Barnes,
Lucy Vasserman, Ben Hutchinson,
Elena Spitzer, Inioluwa Deborah Raji,
and Timnit Gebru. 2019.

Model Cards for Model Reporting. In
Proceedings of the Conference on Fairness,
Accountability, and Transparency (FAT* ’19).
Association for Computing Machinery,
New York, NY, USA, 220–229.

• [59]

Joon Sung Park, Michael S
Bernstein, Robin N Brewer, Ece Kamar,
and Meredith Ringel Morris.
2021.

Understanding the Representation and
Representativeness of Age in AI Data Sets. In
Proceedings of the 2021 AAAI/ACM Conference on AI,
Ethics, and Society. 834–842.

• [60]

Amandalynne Paullada,
Inioluwa Deborah Raji, Emily M. Bender,
Emily Denton, and Alex Hanna.
2021.

Data and its (dis)contents: A survey of dataset
development and use in machine learning research.

Patterns 2,
11 (2021), 100336.

• [61]

Mahima Pushkarna, Andrew
Zaldivar, and Oddur Kjartansson.
2022.

Data Cards: Purposeful and Transparent Dataset
Documentation for Responsible AI. In Proceedings
of the 2022 ACM Conference on Fairness, Accountability, and Transparency.

• [62]

Colin Raffel, Noam
Shazeer, Adam Roberts, Katherine Lee,
Sharan Narang, Michael Matena,
Yanqi Zhou, Wei Li, and
Peter J. Liu. 2020.

Exploring the Limits of Transfer Learning with a
Unified Text-to-Text Transformer.

Journal of Machine Learning Research
21, 140 (2020),
1–67.

• [63]

Inioluwa Deborah Raji,
Timnit Gebru, Margaret Mitchell,
Joy Buolamwini, Joonseok Lee, and
Emily Denton. 2020a.

Saving face: Investigating the ethical concerns of
facial recognition auditing. In Proceedings of the
AAAI/ACM Conference on AI, Ethics, and Society. 145–151.

• [64]

Inioluwa Deborah Raji,
Andrew Smart, Rebecca N White,
Margaret Mitchell, Timnit Gebru,
Ben Hutchinson, Jamila Smith-Loud,
Daniel Theron, and Parker Barnes.
2020b.

Closing the AI accountability gap: Defining an
end-to-end framework for internal algorithmic auditing. In
Proceedings of the 2020 conference on fairness,
accountability, and transparency. 33–44.

• [65]

Negar Rostamzadeh, Diana
Mincu, Subhrajit Roy, Andrew Smart,
Lauren Wilcox, Mahima Pushkarna,
Jessica Schrouff, Razvan Amironesei,
Nyalleng Moorosi, and Katherine
Heller. 2022.

Healthsheet: development of a transparency artifact
for health datasets. In 2022 ACM Conference on
Fairness, Accountability, and Transparency. 1943–1961.

• [66]

Olga Russakovsky, Jia
Deng, Hao Su, Jonathan Krause,
Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy,
Aditya Khosla, Michael S. Bernstein,
Alexander C. Berg, and Li Fei-Fei.
2014.

ImageNet Large Scale Visual Recognition Challenge.

CoRR abs/1409.0575
(2014).

arXiv:1409.0575

• [67]

Pedro Saleiro, Benedict
Kuester, Loren Hinkson, Jesse London,
Abby Stevens, Ari Anisfeld,
Kit T Rodolfa, and Rayid Ghani.
2018.

Aequitas: A bias and fairness audit toolkit.

arXiv preprint arXiv:1811.05577
(2018).

• [68]

Nithya Sambasivan, Erin
Arnesen, Ben Hutchinson, Tulsee Doshi,
and Vinodkumar Prabhakaran.
2021a.

Re-imagining algorithmic fairness in india and
beyond. In Proceedings of the 2021 ACM conference
on fairness, accountability, and transparency. 315–328.

• [69]

Nithya Sambasivan, Shivani
Kapania, Hannah Highfill, Diana Akrong,
Praveen Paritosh, and Lora M Aroyo.
2021b.

“Everyone wants to do the model work, not the
data work”: Data Cascades in High-Stakes AI. In
proceedings of the 2021 CHI Conference on Human
Factors in Computing Systems. 1–15.

• [70]

Maarten Sap, Dallas Card,
Saadia Gabriel, Yejin Choi, and
Noah A. Smith. 2019.

The Risk of Racial Bias in Hate Speech Detection.
In Proceedings of the 57th Annual Meeting of the
Association for Computational Linguistics. Association
for Computational Linguistics, Florence, Italy,
1668–1678.

• [71]

Morgan Klaus Scheuerman,
Jacob M Paul, and Jed R Brubaker.
2019.

How computers see gender: An evaluation of gender
classification in commercial facial analysis services.

Proceedings of the ACM on Human-Computer
Interaction 3, CSCW
(2019), 1–33.

• [72]

Daniel Schiff, Bogdana
Rakova, Aladdin Ayesh, Anat Fanti, and
Michael Lennon. 2020.

Principles to practices for responsible AI: Closing
the gap.

arXiv preprint arXiv:2006.04707
(2020).

• [73]

Christoph Schuhmann,
Richard Vencu, Romain Beaumont,
Robert Kaczmarczyk, Clayton Mullis,
Aarush Katta, Theo Coombes,
Jenia Jitsev, and Aran Komatsuzaki.
2021.

Laion-400m: Open dataset of clip-filtered 400
million image-text pairs.

arXiv preprint arXiv:2111.02114
(2021).

• [74]

Shreya Shankar, Yoni
Halpern, Eric Breck, James Atwood,
Jimbo Wilson, and D Sculley.
2017.

No classification without representation: Assessing
geodiversity issues in open data sets for the developing world.

arXiv preprint arXiv:1711.08536
(2017).

• [75]

Ramya Srinivasan, Emily
Denton, Jordan Famularo, Negar
Rostamzadeh, Fernando Diaz, and Beth
Coleman. 2021.

Artsheets for Art Datasets.

(2021).

• [76]

Antonio Torralba, Rob
Fergus, and William T Freeman.
2008.

80 million tiny images: A large data set for
nonparametric object and scene recognition.

IEEE transactions on pattern analysis and
machine intelligence 30, 11
(2008), 1958–1970.

• [77]

Trieu H Trinh and Quoc V
Le. 2018.

A simple method for commonsense reasoning.

arXiv preprint arXiv:1806.02847
(2018).

• [78]

Ben Wagner, Krisztina
Rozgonyi, Marie-Therese Sekwenz, Jennifer
Cobbe, and Jatinder Singh.
2020.

Regulating transparency? Facebook, twitter and the
German network enforcement act. In Proceedings of
the 2020 Conference on Fairness, Accountability, and Transparency.
261–271.

• [79]

Laura Weidinger, John
Mellor, Maribeth Rauh, Conor Griffin,
Jonathan Uesato, Po-Sen Huang,
Myra Cheng, Mia Glaese,
Borja Balle, Atoosa Kasirzadeh,
et al. 2021.

Ethical and social risks of harm from language
models.

arXiv preprint arXiv:2112.04359
(2021).

• [80]

Benjamin Wilson, Judy
Hoffman, and Jamie Morgenstern.
2019.

Predictive inequity in object detection.

arXiv preprint arXiv:1902.11097
(2019).

• [81]

Jieyu Zhao, Tianlu Wang,
Mark Yatskar, Vicente Ordonez, and
Kai-Wei Chang. 2018.

Gender bias in coreference resolution: Evaluation
and debiasing methods.

arXiv preprint arXiv:1804.06876
(2018).

Appendix A Framework Introduction

Here we present a framework overview, detailed for text, image, and text-image application. Rather than an exhaustive list of every possible analysis, the Human-Content Associations section shows only social identity analyses for each modality. At the end of the overview we provide a list of example tools that can be used to meet the dependency requirements of the analyses. The full framework features recommended risk mitigation actions, however, in practice, mitigation steps should depend on development context and goals. The below tables list common analysis goals and mitigation actions and the following section on Guiding Considerations lists questions and prompts to guide framework use.

Table 3: A non-exhaustive list of data analysis goals.

Table 4: A non-exhaustive list of actions that may be taken to address social risks identified in data.

A.1 Guiding Considerations

• •

What are the planned deliverables of the data being analyzed?
The planned outputs help to inform decisions regarding resource investment in analyses and potential mitigations. For example, a novel benchmark dataset may warrant more extensive analysis than a pre-training dataset because it will be repeatedly use to judge the performance of many models. Similarly, a fine-tuning dataset may be the subject of more modification than a pre-training dataset due to its smaller size and direct role in shaping model compliance with a range of fairness and policy constraints.

• •

What are the primary goals the analyses will support?
Compliance audits and use decisions may only warrant documentation, or packaging and release actions; model understanding may simply be used to help diagnose model behavior (i.e., no mitigation action on data); whereas the development of a new dataset can involve many mitigation actions.

• •

To what extent can completed or planned development steps be revisited or modified
Additional analyses and mitigations require time and resources, as does experimentation to compare which actions are most effective for a given project. For example, budget may be limited for additional data collection, while data filtering or tagging may still be possible.

• •

To what extent is the dataset mutable? (i.e., can data be added, removed, or changed?)
The mutability of a dataset narrows the available mitigation actions and the degree to which data can be filtered or rebalanced.

• •

Can new or additional model benchmarks be run?
When developing a new model, an ultimate consideration is whether biases identified in data will persist through training and finetuning steps. This is known to be true for a number of biases discovered in certain datasets. For novel or recently discovered biases, relevant benchmarks may not exist or are not feasible to create to test downstream models for the bias. This may place emphasis on other mitigation actions, depending on the risk associated with the identified bias.

• •

Will the data or a model trained on the data be released or made available for external use? If so, can the release terms (e.g., terms of use, licensing, policies) be modified?
The use of data and models released externally cannot be as carefully controlled as data and models used strictly within a single institution. Safeguards against risks may be handled through terms of use or licensing agreements in order to mitigate data or model limitations.

• •

Are there institutional or legal requirements regarding what should be analyzed in the data?
Legal and institutional compliance are a first step for planning analyses and reporting results. These can be supplemented with parallel fairness analyses of additional social groups, or analyzed more in depth to make mitigation decisions (e.g., analyzing which data sources feature the strongest identified biases in order to make filtering decisions)

• •

Are there additional fairness or bias concerns a priori?
Beyond compliance concerns, consider analyses that involve groups or data subjects that have been previously impacted by similar systems or within which the context the data or model will be released.

• •

What signals are available for analyzing social characteristics in the data?
The feasibility of running a given analysis depends on the available data features. Aside from straightforward considerations, such as whether a dataset contains faces that can be detected, varied social characteristics may map to a single data feature (e.g., common surnames across disparate racial and ethnic groups) or may not easily map to any data features (e.g., religion and disability are two examples of social characteristics that are not always ’visible’ in images).

Appendix B Analysis Overview

B.1.1 Presence of People

• •

Personally-Identifiable Information

• •

Faces

B.1.2 Social Characteristics

• •

Social Identity Terms

• •

Pronouns

• •

Hateful Terms in Text

• •

Dialect

• •

Perceived Social Identity in Images

• •

Hateful Symbols in Images

B.2.1 Content

• •

Offensive Speech

• •

Topics

• •

Sexual Imagery

• •

Violent Imagery

B.2.2 Provenance

• •

Top Sources

• •

Source Geography

• •

Source Publication Time

• •

Language

• •

Data Duplication

• •

Dataset Overlap

B.3 Human X Content Associations

Text

• •

Social Identity Terms X Top Tokens

• •

Social Identity Terms X Offensive Speech

• •

Social Identity Terms X Topic

Image

• •

Perceived Social Identity Features X Top Image Features

• •

Perceived Social Identity Features X Sexual Imagery

• •

Perceived Social Identity Features X Violent Imagery

Text-Image

• •

Social Identity Terms X Sexual Imagery

• •

Social Identity Terms X Violent Imagery

• •

Perceived Social Identity Features X Text-Image

• •

Perceived Social Identity Features X Sexual Imagery

• •

Perceived Social Identity Features X Violent Imagery

• •

Perceived Social Identity Features X Offensive Speech

• •

Perceived Social Identity Features X Topic

B.4 Analysis Dependencies

Below is a list of technical dependencies required for framework analyses along with example classifiers that can be used.

PII Detection:

Google Data Loss Prevention API (InfoType Detector)

Face Detection:

Google Cloud Vision API (Image Annotation)

Sexual Imagery:

Google Cloud Vision API (Safe Search Annotation)

Violent Imagery:

Google Cloud Vision API (Safe Search Annotation)

Object Classifier:

Google Cloud Vision API (Object Localization)

Topic Classifier:

Google Cloud Natural Language API (Content Categories)

Offensive Speech:

Perspective API (Note: Not offensive speech, but Toxicity)

Language Classifier:

Google Cloud Natural Language API (Text Annotation

Dialect Classifier:

No comprehensive classification approaches.

Hateful Symbols:

No currently reliable method

Social Identity Features:

A number of published audits point to important limitations of social identity classification systems (e.g., [16] and [71] audits of gender classification for image, and [63] audit of gender and age classification for image), therefore we encourage research and results interpretation with individual usage contexts in mind.

Appendix C List of Framework Analyses

C.1 Human Factors

Analyses focused on identifying individuals, social identity groups, and cultural groups in data.

Presence of People

C.1.1 Personally Identifiable Information (PII)

Personally identifiable information allows for an individual’s identity to be potentially inferred through direct or indirect means. It includes, but is not limited to, names, email addresses, addresses, and images or video with identifying characteristics. The presence of personally identifiable information in a dataset is not inherently problematic. However, even if a dataset is not intended for public use, publishing a model trained on a dataset containing personally identifiable information risks “leaking” that information through attack methods designed to recover specific training instances from a model [18]. Generative models that have overfit to the training distribution may also risk generating outputs that constitute a breach in privacy.

Task:

Identify data containing PII

Analysis Object:

Text | Inferred Text Signals

Effort:

Medium

Dependencies:

PII Detection

Output:

Proportion and list of data points containing PII

Action:

Flag PII to assess potential privacy violations that arise for model tasks or as a result of dataset publication

C.1.2 People in Images

As a subcategory of personally-identifiable information, faces in datasets are at risk of “leaking” through attack methods designed to recover specific training instances from a model [18]. The presence of PII is not inherently problematic, however privacy risks must be evaluated in the context of specific tasks.

Task:

Calculate the proportion of images that depict people (e.g., via face detection)

Analysis Object:

Inferred Image Signals

Effort:

Low

Dependencies:

Face Detection

Output:

Proportion of images that depict people

Action:

Flag images of people

Social Characteristics

C.1.3 Social Identity Terms

Under-representation in datasets can contribute to representational harms [9], such as strong under-representation of darker-skinned individuals in datasets linked to under-performance in pedestrian detection [80] and data imbalances generally causing issues for class prediction [46]. Attention to social identity representation in datasets also helps to support intersectional analyses which inherently focus on relatively smaller intersections of data but require sufficient data points to conduct.

Task:

Calculate proportion of text referencing different social identity groups, considering unitary and intersectional groups

Analysis Object:

Text

Effort:

Low

Dependencies:

Social Identity Term List

Output:

Frequency of text depicting unitary and intersectional groups

Action:

Flag social identity representation

C.1.4 Pronoun Distribution

Pronoun distributions help provide a high-level snapshot of gender representation in datasets to evaluate their connection to model performance on gender bias benchmarks. Gender stereotypic associations have been well documented in NLP [13, 17] and mitigating imbalances in gendered terms in datasets has been shown to mitigate gender bias in benchmark performance [81].

Task:

Count number of gender pronouns that occur

Analysis Object:

Text

Effort:

Low

Dependencies:

Pronoun List

Output:

Distribution of pronouns

Action:

Flag pronoun distribution

C.1.5 Hateful Terms in Text

When training datasets contain derogatory language, slurs, or offensive language, there is a risk models trained on the data will apply these terms in an inappropriate manner, e.g. describing people with such terms. Several recent audits of image classification datasets and image-to-text datasets have uncovered the presence racial and ethnic slurs and other derogatory and hateful language [12, 11, 23].

Task:

Count number of hateful terms that occur

Analysis Object:

Text

Effort:

Low

Dependencies:

Hateful Term List

Output:

Distribution of hateful terms, the groups they reference, and their occurrences

Action:

Flag hateful terms

C.1.6 Dialect

Dialects vary across regions and cultures. Data that lacks manners of speaking or dialects prominently used by subgroups can lead to erasure of those dialects in resultant models, or categorization of non-dominant dialects as incorrect, low quality, or even offensive [70].

Task:

Calculate proportion of documents representing languages and dialects aimed to be supported in downstream applications

Analysis Object:

Inferred Text Signals

Effort:

High

Dependencies:

Dialect Classifier

Output:

Proportion of utterances representing different languages and dialects

Action:

Flag dialect representation. Consider rebalancing strongly underrepresented dialects or qualifying downstream model capabilities.

C.1.7 Social Identity in Images

Training datasets that under-represent certain social identity groups can produce models that exhibit poor performance when presented with novel depictions of those groups. For example, image classifiers trained on datasets sourced predominantly from western countries have lower accuracy when applied to images from non-western countries [74], and facial analysis systems trained on datasets that skew heavily towards lighter skinned subjects have higher error rates when applied to images depicting faces with darker skin tones [16].

Task:

Calculate proportion of images depicting social identity groups, considering unitary and intersectional groups

Analysis Object:

Inferred Image Signals

Effort:

Low

Dependencies:

Perceived Social Identity Classifiers

Output:

Proportion of images depicting unitary and intersectional groups

Action:

Flag social identity representation

C.1.8 Hateful Symbols in Images

Visual imagery associated with hate groups (e.g., swastikas) may go undetected or undocumented in datasets. Identifying hateful content across modalities can help mitigate the risk of unintentionally generating hateful or offensive content. It is important to note that offensiveness of imagery is often dependent on cultural context, and hence any action to address this issue should account for this.

Task:

Calculate proportion of images that depict known hateful symbols or text

Analysis Object:

Inferred Image Signals

Effort:

Not yet possible by automated methods

Dependencies:

Hateful Symbol Classifier

Output:

Proportion of images depicting known hateful symbols or text

Action:

Flag images with hateful symbols

C.2 Content Factors

Analyses focused on identifying content that may heighten the sensitivity of human depictions.

Content

C.2.1 Offensive Speech Distribution

Identifying the distribution of offensive content in training data can help to understand the relationship between offensive speech in training data and model outputs as well as help mitigate the risk of unintentionally generating toxic content. Analyses of OWTC and OpenAI-WT suggest that toxic language generation has links to toxic language in web text data and toxic training examples that may be harder for a model to “forget” [34].

Task:

Calculate the distribution of toxicity

Analysis Object:

Inferred Text Signals

Effort:

Low

Dependencies:

Offensive Speech Classifier or similar (e.g., Perspective API)

Output:

Histogram of offensiveness across documents

Action:

Flag toxic content

C.2.2 Topic Distribution

Topic distribution can provide a birds-eye view of the composition of the dataset across various topics such as sports, games, finance, etc. More importantly, it can also give an indication of sexually explicit or sensitive topics they contain, and potential biases they bring. For example, models trained primarily on news data have been shown to exhibit biases against particular country names and professions [42]. Assessments of data distributions across languages may highlight potential weaknesses across languages in generative tasks.

Task:

Calculate the distribution of topics

Analysis Object:

Inferred Text Signals

Effort:

Low

Dependencies:

Topic Classifier (e.g., Google Cloud Content Categories)

Output:

Histogram of topic distribution

Action:

Flag dominant topics

C.2.3 Sexual Imagery

Many datasets have been identified as unintentionally containing sexually explicit content. For example, audits of image datasets such as Imagenet, TinyImage, and LAION-400M found pornographic and explicit imagery, predominantly depicting women [12, 11]. While some datasets may be explicitly curated to contain sexually explicit content, unintentional inclusion of such content can risk accidental generation of sexually explicit content by models trained on the data. Sexually explicit image content also raises potential ethical concerns regarding data sourcing since much publically available pornographic content has been stolen from sex workers or captured in a non-consentual manner [22].

Task:

Calculate the proportion of images that depict sexual content

Analysis Object:

Inferred Image Signals

Effort:

Low

Dependencies:

Visual Content Classifier (e.g., Cloud Vision API)

Output:

Proportion of images depicting sexual content

Action:

Flag sexual imagery. Consider filtering sexual imagery depending on downstream tasks and to avoid problematically-sourced dataset content

C.2.4 Violent Imagery

Like sexual imagery, violent content may feature in datasets as a part of efforts to moderate content, such as in the detection of cartoon content that may be inappropriate for children [49]. For general purposes, inadvertent generation of violent content may be undesirable or counter to institutional or platform policies.

Task:

Calculate the proportion of images that depict violent content

Analysis Object:

Inferred Image Signals

Effort:

Low

Dependencies:

Visual Content Classifier (e.g., Cloud Vision API)

Output:

Proportion of images depicting violent content

Action:

Flag violent imagery. Consider filtering violent imagery depending on downstream tasks

Provenance

C.2.5 Top Sources

The top source URL domains provide a high-level indication of the range of content most represented in web-scraped data. This, in turn, provides insight into document content, such as language and cultural content represented, as well as the prevalence of machine-generated text [28].

Task:

Count the top websites from which tokens were collected

Analysis Object:

Metadata

Effort:

Low

Dependencies:

None

Output:

List of the top websites ordered by tokens collected

Action:

Flag top URLs

C.2.6 Source Geographic Spread

Source URL domains provide an indication of geographic, cultural, and social representation. When training datasets have strong cultural skews, a resulting model can exhibit poor performance when presented with culturally or regionally specific depictions. For example, image classifiers trained on datasets sourced predominantly from western countries have lower rates of accuracy when applied to images from non-western countries [74].

Task:

Calculate the known proportions of total tokens from different geographic regions (e.g., as determined through country-specific top-level domains such as, .uk, .de, .gh, etc.)

Analysis Object:

Metadata

Effort:

Low

Dependencies:

None

Output:

List of the top country-level domains with the proportion of data sourced from each

Action:

Flag geographic spread. Consider rebalancing if regions relatively underrepresented in the dataset are prioritized for downstream uses.

C.2.7 Source Data Publication Time

The relevance of some data can change over time. For example, assessments of truth or offensiveness of a statement can change in relation to when it is evaluated. Data recency may have impacts on models supporting low-resource languages, which can disproportionately rely on religious or historical texts due to data scarcity [3].

Task:

Identify the dates (years) of publication for data obtained

Analysis Object:

Metadata

Effort:

Low

Dependencies:

None

Output:

Histogram of data published per year

Action:

Flag publication time. Consider rebalancing if prioritizing tasks likely to be relatively sensitive to data publication time (e.g., Q&A factuality; use of contemporary slang in generative language tasks)

C.2.8 Language

Languages vary across regions and cultures, however work in NLP frequently focuses on English, even when there is data available for other languages [41]. As with under-representation of data and classes in other contexts, poorly represented languages in a language dataset may lead to worsened performance of a downstream model when applied to that langauge.

Task:

Calculate proportion of documents representing languages aimed to be supported in downstream applications

Analysis Object:

Text | Inferred Text Signals

Effort:

Low

Dependencies:

Language Classifier (e.g., Google Cloud Natural Language API)

Output:

Proportion of utterances representing different languages

Action:

Flag language representation. Consider rebalancing strongly underrepresented languages or qualifying downstream model capabilities.

C.2.9 Data Duplication

Duplicate text has been identified in a number of datasets (e.g., [8]) and has been shown to worsen model performance on some tasks [4] as well as induce model memorization [52], which can carry privacy risk [18].

Task:

Calculate the proportion of duplicated data points

Analysis Object:

Text | Image

Effort:

Low

Dependencies:

None

Output:

Proportions of duplicated data points

Action:

Filter duplicate data to reduce memorization and privacy risks and improve model performance

C.2.10 Dataset Overlap

There is growing awareness and concern of test set contamination, i.e. presence of training instances in the test set, especially with large internet sourced datasets. Explicitly examining and mitigating any overlap in training and evaluation splits is increasingly commonplace (e.g., [77, 15].

Task:

Identify overlaps between the present dataset and relevant datasets and benchmarks

Analysis Object:

Text | Image

Effort:

Medium

Dependencies:

None

Output:

For benchmark datasets identified, list of percent overlap with the current dataset based on exact matches of target text

Action:

Filter dataset overlaps to preserve the validity of benchmark results

C.3 Human X Content Associations

Disaggregated analyses of how content is associated with human factors. Listed below are a selection of all possible associations between Human Factors and Content Factors. Disaggregated analyses enable evaluation of stereotypical or otherwise harmful associations between depictions of people and dataset content. For example, prior research has identified stereotype-aligned associations between gender and activities in image datasets [81, 39], associations between muslims and violence in language models [2], and associations between women of color and explicit content in text datasets [54] and image-text datasets [12].

Text

C.3.1 Social Identity Terms X Top Tokens

Analysis Object:

Text

Task:

For social identity terms in text, calculate top word co-occurrences for each

Effort:

Low

Dependencies:

None

Output:

List of most frequent social identity terms and their respective top co-occurrences, demarcating identity terms (e.g., woman, Muslim, trans)

Action:

Flag associations. Consider rebalancing harmful associations that emerge in model evaluations

C.3.2 Social Identity Terms X Topic

Task:

Calculate the distribution of topics within text, disaggregated by social identity terms or inferred social identity signals

Analysis Object:

Inferred Text Signals + Social Identity Term List

Effort:

Low

Dependencies:

Topic Classifier (e.g., Google Cloud Content Categories)

Output:

Distribution of topics disaggregated by social identity

Action:

Flag disproportionate associations between social identity terms and stereotypical or sensitive topics

C.3.3 Social Identity Terms X Offensive Speech

Task:

Calculate the distribution of toxicity within text, disaggregated by social identity terms or inferred social identity signals

Analysis Object:

Inferred Text Signals + Social Identity Term List

Effort:

Low

Dependencies:

Offensive Speech Classifier or similar (e.g., Perspective API

Output:

Distribution of toxicity, disaggregated by social identity

Action:

Flag high toxicity. Consider rebalancing harmful associations that emerge in model evaluations

Image

C.3.4 Perceived Social Identity Features X Top Image Features

Task:

Calculate co-occurrences between perceived social identity features and image features, such as objects and other people.

Analysis Object:

Inferred Image Signals

Effort:

Low

Dependencies:

Object Classifier (e.g., Google Cloud Vision API)

Output:

List of perceived social identity features and the top co-occurrences for each.

Action:

Flag stereotypical associations. Consider rebalancing harmful associations that emerge in model evaluations.

C.3.5 Perceived Social Identity Features X Sexual Imagery

Task:

Calculate co-occurrences between perceived social identity features and sexual imagery.

Analysis Object:

Inferred Image Signals

Effort:

Low

Dependencies:

Visual Content Classifier (e.g., Google Cloud Vision API

Output:

Proportion of images depicting sexual content disaggregated by social identity features

Action:

Flag sexual content. Consider filtering sexual content depending on downstream tasks and to avoid problematically-sourced dataset content.

C.3.6 Perceived Social Identity Features X Violent Imagery

Task:

Calculate co-occurrences between perceived social identity features and violent imagery.

Analysis Object:

Inferred Image Signals

Effort:

Low

Dependencies:

Visual Content Classifier (e.g., Cloud Vision API

Output:

Proportion of images depicting violent content disaggregated by social identity

Action:

Flag violent content. Consider filtering sexual content depending on downstream tasks and to avoid problematically-sourced dataset content

C.3.7 Perceived Social Identity Features X Hateful Symbols

Task:

Calculate co-occurrences between perceived social identity features and hateful symbols detected in images.

Analysis Object:

Inferred Image Signals

Effort:

Not currently possible

Dependencies:

Hateful Symbol Classifier

Output:

Proportion of images depicting sexual content disaggregated by social identity

Action:

Flag hateful content. Consider filtering hateful content depending on downstream tasks.

Text-Image

C.3.8 Social Identity Terms X Sexual Imagery

Analysis Object:

Social Identity Term List + Inferred Image Signals

Task:

Calculate co-occurrences between social identity terms and sexual imagery.

Effort:

Low

Dependencies:

Visual Content Classifier (e.g., Cloud Vision API

Output:

Proportion of images depicting sexual content disaggregated by social identity term

Action:

Flag associations. Consider rebalancing harmful associations that emerge in downstream model evaluations

C.3.9 Social Identity Terms X Violent Imagery

Analysis Object:

Social Identity Term List + Inferred Image Signals

Task:

Calculate co-occurrences between social identity terms and violent imagery.

Dependencies:

Visual Content Classifier (e.g., Cloud Vision API

Output:

Proportion of images depicting violent content disaggregated by social identity term

Action:

Flag associations. Consider rebalancing harmful associations that emerge in downstream model evaluations

Effort:

Low

C.3.10 Perceived Social Identity Features X Top Text Tokens

Analysis Object:

Inferred Image Signals + Text

Task:

Calculate co-occurrences between perceived social identity signals and tokens in associated text.

Effort:

Low

Dependencies:

Perceived Social Identity Classifiers

Output:

List of most frequent co-occurrences for each social identity signal

Action:

Flag associations. Consider rebalancing harmful associations that emerge in downstream model evaluations

C.3.11 Perceived Social Identity Features X Offensive Speech

Analysis Object:

Inferred Image Signals + Inferred Text Signals

Task:

Calculate co-occurrences between perceived social identity signals and tokens in associated text.

Effort:

Low

Dependencies:

Offensive Speech Classifier or similar (e.g., Perspective API

Output:

Distribution of toxicity, disaggregated by social identity

Action:

Flag high toxicity. Consider rebalancing harmful associations that emerge in model evaluations

C.3.12 Perceived Social Identity Features X Topic

Analysis Object:

Inferred Image Signals + Inferred Text Signals

Task:

Calculate co-occurrences between perceived social identity signals and tokens in associated text.

Effort:

Low

Dependencies:

Topic Classifier (e.g., Google Cloud Content Categories)

Output:

Distribution of toxicity, disaggregated by social identity

Action:

Flag high toxicity. Consider rebalancing harmful associations that emerge in model evaluations

Where this page came from

This page was imported from arXiv. “SoUnD Framework: Analyzing (So)cial Representation in (Un)structured (D)ata” by Mark Díaz, Sunipa Dev, Emily Reif et al., arXiv:2311.17259 (2023), published under CC BY 4.0. Changed here: set as a page from arXiv's HTML version; footnotes and author notes left out; figures the paper marks as under other terms left out.

Nobody has written it yet — it is the source material at a new address, which is why search engines are asked to skip it and why no one earns from it. It is up for grabs: take it on, and it is yours to rewrite and to earn from.