Nlp

Reducing Toxicity in Language Models

Large pretrained language models are trained on extensive collections of online data. As a result, they inevitably absorb certain toxic behaviors and biases present on the Internet. These pretrained models are highly capable and have demonstrated strong performance across many NLP tasks. However, deploying them safely in practical, real-world applications requires robust safety controls over the model’s generation process.

2021 · 22 min read · Curated and presented by Arthur Sedek

Toxicity prevents the safe deployment of powerful pretrained language models in real-world applications. To mitigate toxicity in language models, this post examines three dimensions of the problem: training dataset collection, toxic content detection, and model detoxification.

Large pretrained language models are trained on substantial collections of online data. As a result, they inevitably absorb toxic behaviors and biases present on the Internet. These pretrained language models are highly capable and have demonstrated strong performance across many NLP tasks. However, deploying them safely in practical applications requires robust safety controls over the model’s generation process.

Efforts to reduce unsafe content face several challenges:

First, unsafe content spans many forms, including toxicity, abusiveness, hate speech, biases, stereotypes, cyberbullying, identity attacks, and others, which may or may not require different handling strategies.
Second, there is no single, broadly accepted categorization or definition of unsafe behavior in pretrained language models. Perceptions can differ significantly across individuals due to varying social backgrounds.

This post focuses specifically on toxicity in language models. Because I am still working toward a concrete definition of toxic content, I include a few definitions from the literature below.

[Perspective API] A rude, disrespectful, or unreasonable comment; likely to make people leave a discussion.

[Kurita et al. 2019] Content that can offend or harm its recipients, including hate speech, racism, and offensive language.

[Pavlopoulos et al. 2020] We use the term ’toxic’ as an umbrella term, but we note that the literature uses several terms for different kinds of toxic language or related phenomena: ‘offensive’, ‘abusive’, ‘hateful’, etc.

In general, toxicity is an umbrella term covering multiple categories of unsafe content. The methods discussed in this post can be applied once some operational definition of toxicity is specified, for example, via annotator instructions. Establishing an appropriate definition of toxicity and collecting accurate labels accordingly are out of scope for this post.

Categorization of Toxic Content

Classifying toxic content is not straightforward. Determining what qualifies as toxic and identifying distinct types of toxicity can be highly subjective. Language that appears unobjectionable to one community may be perceived as inappropriate by another.

A widely used categorization of offensive language was introduced by Zampieri et al. (2019). It proposes a three-level hierarchical taxonomy that considers both the type of offense and its target. The Offensive Language Identification Dataset (OLID) is collected using this taxonomy.

The three-level hierarchical taxonomy for categorizing offensive language, proposed by Zampieri et al. (2019).

Level A: “Is it offensive?”
- [OFF] Offensive: Inappropriate language, insults, or threats.
- [NOT] Not offensive: No offense or profanity.
Level B: “Is the offensive text targeted?”
- [TIN] Targeted Insult: A targeted insult or threat aimed at an individual, a group, or other entity.
- [UNT] Untargeted: Non-targeted profanity and swearing.
Level C: What is the target?
- [IND] The offense targets an individual, often described as “cyberbullying”.
- [GRP] The offense targets a group of people based on ethnicity, gender, sexual orientation, religion, or another shared characteristic, often described as “hate speech”.
- [OTH] The target falls into another category, such as an organization, an event, an issue, etc.

Data Collection

Building a dataset of examples labeled as “safe” versus “unsafe” is the basis for training a toxic language classifier and for providing signals used in downstream detoxification methods.

Human Annotations

Vidgen & Derczynski (2020) summarized that, at a high level, toxicity detection annotations can be collected through the following approaches:

Expert coding: An expert has sufficient knowledge or training to perform annotation with strong quality, for example, a researcher studying prejudice, a moderately trained student, or an NLP practitioner. This option is more expensive, but it typically yields higher-quality data.
Crowdsourcing: A crowdsourcing platform assigns tasks to large numbers of non-expert annotators. This approach scales more easily, but it requires careful quality control.
Professional moderators: Professional moderators have task experience and training, but their objectives are likely to be optimized for platform-specific outcomes.
Synthetic data: A training dataset can also be manually authored by relevant content creators to cover a broad range of toxic content types.

Among these, crowdsourcing is the most common approach (Davidson et al. 2017, Zampieri et al. 2019). Several practices can improve the resulting data quality:

Test data: A small set of expert-produced annotations can be used as test questions (Zampieri et al. 2019) to screen annotators on the crowdsourcing platform and filter out those who do not meet a required performance threshold.
Clear guidelines: Detailed instructions help annotators produce consistent, aligned labels. Without guidance, annotators tend to rely on personal judgment, which can be problematic because (1) subjective interpretations of toxic content vary widely across individuals and (2) without guidance, it is difficult to consistently label certain types of noise such as sarcasm and irony.
Majority vote: It is common to collect labels from multiple annotators per sample and use a majority vote to determine the final label.
Understanding annotators’ identities: Demographic background strongly influences how annotators interpret the task. The goal should be to recruit annotators who are both diverse and qualified.

Semi-supervised Dataset

Khatri et al. (2018) proposed a straightforward bootstrapping method to create a large semi-supervised dataset for training toxic content classifiers. The method assumes access to a small labeled dataset and a large unlabeled dataset.

First, they compile a blacklist of more than 800 words spanning profanity, hate, sexual content, and insults. A profanity blacklist can offer high precision but low recall; nonetheless, it can provide weak supervision signals.
Subreddits are ranked by the fraction of blacklisted words. Sensitive examples are sampled from the top-ranked subreddits, while non-sensitive examples are sampled from the bottom-ranked subreddits.
A weak binary classifier is trained and then used to select additional samples from the ranked subreddits:
- Sensitive: contains blacklisted words or toxic classifier confidence > 0.8;
- Non-sensitive: does not contain blacklisted words and toxic classifier confidence < 0.3
Using this expanded dataset, they train a new classifier referred to as “Two-stage bootstrap” (TS bootstrap).

Their experiments showed that the TS bootstrap classifier achieved strong results on F1 score, accuracy, and recall, and it also transferred to out-of-domain test data.

The two-stage bootstrap classifier is trained on a dataset bootstrapped by a weak toxic binary classifier on Reddit data. (Image source: Khatri et al. 2018)

SOLID (Semi-Supervised Offensive Language Identification Dataset; Rosenthal et al. 2020) contains more than 9 M tweets annotated using the same taxonomy as OLID. SOLID uses OLID as a seed and expands it via a semi-supervised method called democratic co-training. Democratic co-training (Zhou & Goldman, 2004) constructs a large dataset from noisy labels produced by a diverse collection of models trained on a small supervised dataset. SOLID is created as follows:

First, train a diverse group of supervised models on the labeled OLID dataset. The paper experiments with PMI (n-gram-based similarity), FastText (a shallow neural model similar to a BoW model), LSTM, and BERT.
For each example in the unlabeled dataset, each model outputs a confidence score for the target class. These scores are aggregated using avg() or min(). High-confidence examples are then added to the dataset.

BERT performance does not improve when the supervised dataset is sufficiently large for a simple task, but it can benefit from a large semi-supervised dataset when the original supervised dataset is too small for the task.

Toxicity Detection

With a supervised dataset, we can train a text classifier from scratch or fine-tune a pretrained language model for classification. However, what if the training samples are low quality or too limited? What if no such supervised dataset is available?

Adversarial Attacks

To build toxicity detection models that are robust to adversarial attacks, Dinan et al. (2019) proposed an iterative build it, break it, fix it framework for improving dialogue system safety with humans in the loop.

Build it: Train a BERT classifier to detect toxic comments using the Jigsaw dataset.
Break it: Ask crowdsourced workers to write toxic messages that the model incorrectly labels as “safe”.
Fix it: Retrain the model on a combined dataset consisting of the original data plus the newly collected adversarial samples.
Repeat: Redeploy the strengthened model and run another iteration starting again from step 1.

The illustration of iteratively improving a toxic content detection model via the "build it, break it, fix it" process. (Image source: Dinan et al. 2019)

One baseline in their experiments replaces the adversarial data collection in the “break it” step with a standard collection procedure, where workers are simply asked to submit “offensive” messages directly. Relative to standard collection, adversarial collection includes less explicit profanity and more negations intended to fool the model. The task also becomes more difficult in later rounds.

Models trained with adversarial collection are more robust to adversarial attacks than baseline models trained on the standard collection. The third-round adversarial model performs worse on the standard task than the standard model, likely because of overfitting. I am curious how performance would change if the model were trained using both adversarial and standard collections, but I did not find such an experiment reported in the paper.

The comparison of performance on standard and adversarial tasks of models trained on standard ($S\_i$) and adversarial data collection ($A\_i$). The subscript $i$ indicates the number of training rounds. (Image source: Dinan et al. 2019)

Another form of adversarial attack attempts to cause a detection model to misclassify a toxic sentence as safe by replacing or scrambling a subset of characters. Kurita et al. (2019) introduced a model-agnostic method for generating such adversarial attacks using multiple types of character-level perturbations:

Character scrambling: Randomly permute character positions.
Homoglyph substitution: Replace one or more letters with visually similar international characters.
Dictionary-based near-neighbor replacement: Replace a token with the closest but distinct token according to Levenshtein distance.
Distractor injection: Inject distractor tokens by repeating randomly selected sequences of non-toxic tokens.

Adversarial noise that combines token obfuscation with distractor tokens causes substantial degradation in toxic classifier performance. Character-level perturbations reduce performance more than distractors do.

The paper proposed two approaches for mitigating adversarial attacks:

Adversarial training: Train the model on a dataset that includes noise. This requires knowing details of the incoming attacks in advance, and there is no guarantee that training on arbitrary noise will generalize to test-time attacks.
CDAE (contextual denoising autoencoder): Use character-level and contextual signals to denoise obfuscated tokens. CDAE takes a noisy sample and predicts a denoised version. This still requires knowing which character-level perturbations may be used to construct the noisy samples. CDAE performs comparably to BERT, but not substantially better.

Perspective API

perspective API (www.perspectiveapi.com) is the most widely used commercial API for toxic content detection. Perspective trains machine learning models that produce scores for multiple attributes: toxicity, severe toxicity, insult, profanity, identity attack, threat, and sexually explicit. Each score is a value in [0, 1] that reflects how likely the message contains the attribute (that is, the confidence of a binary classifier), and it does not represent the attribute’s severity.

Gehman et al. (2020) measured Perspective API toxicity scores for unprompted generations sampled from several pretrained language models. Here, “unprompted” means generation conditioned only on the start-of-sentence tokens, without adding any additional context. Notably, for all evaluated models, the expected maximum toxicity exceeds 0.5 after 100 generations. They also noted that training datasets for large language models contain a non-negligible amount of toxic content.

Perspective API toxicity scores of unprompted generations. Each model generates a pool of 10k samples and the expected maximum toxicity score is estimated via bootstrapping. (Image source: Gehman et al. 2020)

They collected the RealToxicityPrompt dataset to study toxicity in conditional language model generation. The dataset contains 100k naturally occurring prompts, each paired with a toxicity score from Perspective API. Some prompts that contain no toxic language can still elicit highly offensive completions.

Despite its widespread use, Perspective API has known biases, summarized by Gehman et al. (2020):

… exhibit biases against minorities and suffer from low agreement in annotations, partially due to annotator identity influencing their perception of hate speech and differences in annotation task setup.

Notably, recent work has found that systems are overestimating the prevalence of toxicity in text that contains a minority identity mention (e.g., “I’m a gay man”) or text by racial minorities (e.g., text in African American English). This is partially due to detectors’ over-reliance on lexical cues of toxicity (including swearwords, slurs, and other “bad” words).

Prompt-based Detection

Self-diagnosis (Schick, et al. 2021) exploits a pretrained language model’s ability to detect socially undesired attributes in its own outputs. The diagnosis uses a predefined prompt template that describes the attributes in brief text, then measures the normalized probability that the model answers “yes” versus “no”. Notably, self-diagnosis does not require access to a labeled training dataset.

# Self-diagnosis template
"{sample text}"
Question: Does the above text contain {undesired attribute}?
Answer: {yes/no}

They use RealToxicityPrompts dataset and Perspective API for evaluation in the experiments. The self-diagnosis performance is positively correlated with the model size.

Self-diagnosis abilities for identifying undesired attributes. The ground truth is provided by Perspective API. (Image source: Schick, et al. 2021)

Detoxification

Blacklisting

Bad word filtering is an intuitive and often effective method for preventing explicit profane words from appearing in language model outputs. During decoding, we can manually lower the probabilities of blocked words to reduce the chance of sampling them. However, this approach is not sufficient on its own, since unsafe content can still be produced using only otherwise safe tokens.

Vocabulary shifting (Gehman et al. 2020) learns a two-dimensional representation for each token in the pretrained model’s vocabulary, capturing toxicity versus non-toxicity. The component encoding non-toxicity is then used to increase the likelihood of non-toxic tokens during decoding.

Prompt-based Detox

Self-debiasing (Schick et al. 2021) builds on a concept similar to self-diagnosis. It uses the internal knowledge of a pretrained language model to reduce the probability of undesired attributes in generation.

# Self-debiasing template, denoted as sdb(.)
The following text contains {undesired attribute s}:
{sample text x}

Given an input prompt $\mathbf{x}$, a textual description of undesired attributes $s$, and the language model $M$, self-debiasing computes the difference between the probability of next words without and with the self-debiasing template $\text{sdb}(.)$:

$ \Delta(w, \mathbf{x}, s) = p_M(w\vert\mathbf{x}) - p_M(w\vert\text{sdb}(\mathbf{x}, s)) $

Because $\text{sdb}(.)$ is expected to increase the probabilities of undesired words, $\Delta(w, \mathbf{x}, s)$ should be negative for undesirable words.

In self-diasing decoding, a scaling function of the probability difference $\alpha(\Delta(w, \mathbf{x}, s)): \mathbb{R}\to[0,1]$ is used to modify the true sampling distribution,

$ \tilde{p}_M(w\vert\mathbf{x}) \propto \alpha(\Delta(w, \mathbf{x}, s)) p_M(w\vert\mathbf{x}) $

In the paper, they use a soft variant in which the probabilities of words with negative $\Delta$ are reduced in proportion to the magnitude of $\Delta(w, \mathbf{x}, s)$:

$ \alpha(x)=\begin{cases} 1 & \text{ if } x\geq 0 \\ e^{\lambda\cdot x} & \text{ otherwise} \end{cases} $

Self-diasing decoding can reduce the probabilities of undesirable attributes. The scores are provided by Perspective API. (Image source: Schick et al. 2021)

Self-debiasing detoxification has several important limitations:

Evaluation relies exclusively on Perspective API, so it cannot capture bias and toxicity attributes not covered by Perspective API, such as gender bias. Human evaluation is an alternative, but it does not scale well.
Self-debiasing can be overly aggressive, filtering out benign words, and it does not preserve the original model’s perplexity level.
The method is limited by the model’s internal capacity. For example, if the model is not aware of certain biases, it cannot correct them.

Text Style Transfer

Unsupervised style transfer can be applied to convert offensive sentences into innocuous ones (Santos et al. 2018). The method is designed for non-parallel datasets, meaning we only have two separate corpora (offensive and non-offensive), rather than paired examples. To preserve content while transferring style, the approach adopts a cycle consistency loss (Zhu et al. 2017).

Let $s_i$ be the desired style ($i=0$ for offensive and $i=1$ for non-offensive), and let $\mathbf{x}^i_k$ be the $k$-th sample with style $s_i$, $k = 1, \dots, n$. Both the encoder $E$ and decoder $G$ take a sample (or hidden state) together with a style label. The classifier $C$ predicts a probability distribution over style labels given an input sample.

Following the illustration in Fig. 9:

The top branch of forward transfer is an autoencoder: $E(\mathbf{x}^i_k, s_i) \to H^i_k \to G(H^i_k, s_i) \to \hat{\mathbf{x}}^{i\to i}_k$. Two losses are computed:
- Reconstruction loss measures how well the decoder can reconstruct the original sample:

$ \mathcal{L}_\text{self} = \mathbb{E}_{\mathbf{x}^i_k \sim \mathcal{X}} [-\log p_G(\mathbf{x}_k^i \mid E(\mathbf{x}^i_k, s_i), s_i)] $

The bottom branch of forward transfer: $E(\mathbf{x}^i_k, s_i) \to H^i_k \to G(H^i_k, s_j) \to \hat{\mathbf{x}}^{i\to j}_k$
- Classification loss measures how effectively the style is transferred:

$ \mathcal{L}_\text{style_fwd} = \mathbb{E}_{\hat{\mathbf{x}}^{i\to j}_k \sim \hat{\mathcal{X}}} [-\log p_C(s_j \mid \hat{\mathbf{x}}^{i\to j}_k)] $

The back transfer uses the cycle consistency loss: $E(\hat{\mathbf{x}}^{i\to j}_k, s_j) \to H^{i\to j}_k \to G(H^{i\to j}_k, s_i) \to \hat{\mathbf{x}}^{i\to j \to i}_k$
- The cycle consistency loss controls how accurately the transferred sample can be converted back to its original form, encouraging content preservation:

$ \mathcal{L}_\text{cycle} = \mathbb{E}_{\mathbf{x}^i_k \sim \mathcal{X}} [-\log p_G(\mathbf{x}_k^i \mid E(\hat{\mathbf{x}}^{i \to j}_k, s_j), s_i)] $

- The classification loss ensures that the back-transferred sample has the correct label:

$ \mathcal{L}_\text{style_back} = \mathbb{E}_{\hat{\mathbf{x}}^{i\to j}_k \sim \hat{\mathcal{X}}} [-\log p_C(s_i \mid G(E(\hat{\mathbf{x}}^{i\to j}_k, s_j), s_i))] $

An additional supervised classification loss is included to train an accurate classifier:

$ \mathcal{L}_\text{class} = \mathbb{E}_{\hat{\mathbf{x}}^{i\to j}_k \sim \hat{\mathcal{X}}} [-\log p_C(s_i \mid \hat{\mathbf{x}}^i_k)] $

The final training objective is defined as follows, and the encoder, decoder, and classifier are trained jointly:

$ \mathcal{L}(\theta_E, \theta_G, \theta_C) = \min_{E, G, C} \mathcal{L}_\text{self} + \mathcal{L}_\text{style_fwd} + \mathcal{L}_\text{cycle} + \mathcal{L}_\text{style_back}+ \mathcal{L}_\text{class} $

Style Transformer (Dai et al. 2019) also targets unsupervised text style transfer. Unlike the encoder-decoder approach in Santos et al. 2018, it learns a Transformer-based style transfer function $f_\theta(\mathbf{x}, s)$ for an input sample $\mathbf{x}$ and a desired style control variable $s$.

Comparison between Style Transformer and earlier models that rely on disentangled latent representations. (Image source: Dai et al. 2019)

When a parallel corpus is unavailable, Style Transformer introduces a discriminator to derive supervision from non-parallel data.

Let $s$ and $\hat{s}$ denote two mutually exclusive style variables, and let $\mathbf{x}$ be a sample with style $s$. Style Transformer optimizes several loss terms:

Self-reconstruction loss: $\mathcal{L}_\text{self} = - p_\theta (\mathbf{x} \vert \mathbf{x}, s)$
Cycle-consistency loss: $\mathcal{L}_\text{cycle} = - p_\theta (\mathbf{x} \vert f_\theta(\mathbf{x}, \hat{s}), s)$
Style-controlling loss: This term is required because, without it, the model could degenerate into simply copying the input.

$ \mathcal{L}_\text{style} = - p_\phi(\text{class} = 1 \vert f_\theta(\mathbf{x}, \hat{s}), \hat{s}) $

where the discriminator is a simple binary classifier trained to optimize the negative log-likelihood of the correct style. The discriminator is trained by labeling:

$\{(\mathbf{x}, s), (f_\theta(\mathbf{x}, s), s), (f_\theta(\mathbf{x}, \hat{s}), \hat{s})\}$ as positive class 1
$\{(\mathbf{x}, \hat{s}), (f_\theta(\mathbf{x}, s), \hat{s}), (f_\theta(\mathbf{x}, \hat{s}), s)\}$ as negative class 0.

Training process for Style Transformer. (Image source: Dai et al. 2019)

Motivated by the research question, “Can we fine-tune a pre-trained language model to suggest civil rephrasings of rude comments using a dataset solely annotated in toxicity?”, Laugier et al. (2021) fine-tuned a pretrained text-to-text transformer using a denoising objective combined with a cyclic auto-encoder loss.

Let $s$ be the attribute of $\mathbf{x}$ (for example, “civil”), and let $\bar{s}$ be the opposite attribute (for example, “toxic”). These two attributes are mutually exclusive. The objective is to learn a mapping function $f_\theta$ such that it transforms $x$ into a new fluent sequence $y$ with target attribute $a$, while preserving the content of $x$.

The encoder-decoder model is trained with the following loss:

$ \mathcal{L} = \lambda_\text{DAE} \mathcal{L}_\text{DAE} + \lambda_\text{cycle} \mathcal{L}_\text{cycle} $

The denoising auto-encoder loss follows the standard denoising auto-encoder objective, where $\eta$ is a masking function, the same as in BERT training:

$ \mathcal{L}_\text{DAE} = \mathbb{E}_{\mathbf{x} \sim \mathcal{X}} [−\log p_\theta(\mathbf{x} \mid \eta(\mathbf{x}), s)] $

The cycle-consistency loss (Zhu et al. 2017) uses $\tilde{\theta}$ to generate a non-differentiable pseudo-prediction $\hat{\mathbf{y}}$, and it does not backpropagate gradients.

$ \mathcal{L}_\text{cycle} = \mathbb{E}_{\mathbf{x} \sim \mathcal{X}} [−\log p_\theta(\mathbf{x} \mid f_{\tilde{\theta}}(\mathbf{x}, \bar{s}), s)] $

They applied the loss above to fine-tune a T5 model, producing a model named CAE-T5. Conditioning is implemented in the same spirit as CTRL, using a control code (“civil” or “toxic”) prepended to the beginning of the sequence.

Automatic evaluation of style-transfer outputs typically relies on three metrics:

Accuracy: Classification accuracy, indicating how successfully the style has been transferred.
Fluency: Commonly measured using perplexity from a separately trained language model on non-toxic samples.
Content preservation: Content similarity between the transferred and original sentences, measured via BLEU or embedding-based similarity.

Human evaluation is also important, but it is more expensive to run.

Relative to the baseline (Shen et al. 2017), the style-transfer approach of Santos et al. 2018 achieves higher classification accuracy and stronger content preservation, but worse perplexity. Compared with a set of baselines that includes Style Transformer, CAE-T5 shows lower classification accuracy, competitive content preservation, and improved perplexity.

Controllable Generation

One way to reduce toxic outputs is to use controllable text generation. Several widely used approaches can steer a pretrained language model toward desired styles, topics, or safety constraints:

Use guided decoding strategies and select preferred outputs at inference time.
Optimize for the desired behavior through effective prompt design.
Fine-tune the base model, or steerable components, to enable conditioned generation.

For additional background, see my last post on controllable neural text generation, which covers methods such as AutoPrompt, CTRL, PPLM, GeDi, and many others.

Gehman et al. (2020) evaluated both data-based methods (supervised fine-tuning, CTRL training) and decoding-based methods (vocabulary shifting, blocked word filtering, PPLM) for detoxifying language models. They observed that toxicity control tokens (CTRL) and swear-word filters are less successful than approaches that require more computation or more data, such as fine-tuning on non-toxic corpora and PPLM.

Table listing the expected maximum toxicity score over 25 generations (left) and the empirical probability of generating toxic text over 25 generations (right) across multiple detoxification methods. Scores are produced by the Perspective API. (Image source: Gehman et al., 2020)

System-level Safety Solution

Xu et al. (2020) proposed a comprehensive system-level design for building safer chatbots.

Illustration of a safe chatbot system. (Image source: Xu et al. 2020)

They describe four broad strategies in their recipes for improving bot safety:

Detect unsafe content: Use a classifier to identify unsafe language on both the input and output sides, adding an extra safety layer on top of the language model.
- The classifier is trained on an enhanced version of the Jigsaw toxic comment dataset (binary safe vs unsafe labels), augmented with adversarial human attacks (Dinan et al. 2019) and semi-supervision (Khatri et al. 2018).
- The safety classifier can be applied to both user inputs and model outputs. If unsafe content is detected, the system can be configured to return a canned, predefined response (for example, “I’m sorry I’m not sure what to say.”) or to switch topics. It is worth noting that this approach depends on a high-quality classifier. Excessive false positives would severely disrupt the conversation experience.
- Bot adversarial dialogue (BAD) safety: The goal is to gather data from humans who adversarially probe the system to induce failures, and then use those data for additional training. During annotation, human labelers can rate the bot’s response on an unsafe-safe scale based on the fraction of the population that may consider it unsafe. These probing data are then used to train a multi-turn safety classifier that predicts whether a response is offensive given the dialogue context.
Safe generation: Train a model that is less likely to produce unsafe responses.
- A predefined list of unsafe words or n-grams can be blocked during decoding.
- Pretraining data can be filtered using the safety classifier above, or filtered based on known authors.
- A limitation of pretraining exclusively on safe datasets is that, if the model has never encountered toxic language during training, it may not know how to respond at test time (OOD). For example, it may simply copy offensive content. Instead, they construct training samples in which the last utterance is labeled “unsafe” and then append a safe response after that unsafe attack. The model is then fine-tuned on this “baked-in” safety data.
- Perform CTRL-style training by assigning “safe” vs “unsafe” labels using the safety classifier.
Avoid sensitive topics:
- To avoid sensitive topics (politics, religion, drug use, medical advice, and NSFW and relationships/dating), they trained a multi-class classifier to detect those topics using crowdsourced lists of subreddits. The classifier can be retrained periodically to track how topics evolve over time.
- They also collect a small validation set by recruiting crowdsourced workers to discuss one of the targeted topics.
Gender bias mitigation:
- They apply CTRL-style training to mitigate gender bias.
- Specifically, given a gendered word list, they tag training samples with $F^0 M^0$, $F^0 M^+$, $F^+ M^+$, and $F^+ M^0$ labels, indicating whether the response contains female or male words ($+$ contains, $-$ does not contain). At test time, the system runs with a control label $F^0 M^0$ to avoid generating gender-specific words.

Appendix: Datasets

(*Only datasets in English are listed here.)

Hate Speech and Offensive Language Dataset (2017): contains about 25k tweets, each manually labeled as one of three categories: hate speech, offensive but not hate speech, or neither offensive nor hate speech. [Download]

Jigsaw Toxic Comments Classification Dataset (2018): contains about 160k examples extracted from Wikipedia discussion pages, each annotated into 7 classes: toxic, severe toxic, obscene, threat, insult, identity hate, and non-toxic. The labeling process involved 5000 crowdsourced annotators. [Download]

Jigsaw Unintended Bias in Toxicity Classification Dataset (2019): contains about 2 Millions comments from the Civil Comments platform, which shut down in 2017. The data are annotated for toxicity, toxicity sub-types, and identity mentions, enabling evaluation of unintended bias with respect to identity mentions. [Download]

OLID (Offensive Language Identification Dataset; 2019): contains 14,100 English tweets, annotated according to the three-level taxonomy described here. [Download]

SOLID (Semi-Supervised Offensive Language Identification Dataset; 2020): contains 9+ Millions tweets annotated following OLID’s three level taxonomy. [Download]

RealToxicityPrompts dataset (2020): contains 100k sentence snippets from the web with Perspective API toxicity scores for studying the risk of neural toxic degeneration in language models. [Download]

Citation

Cited as:

Weng, Lilian. (Mar 2021). Reducing toxicity in language models. Lil’Log. https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/.

@article{weng2021toxic,
  title   = "Reducing Toxicity in Language Models.",
  author  = "Weng, Lilian",
  journal = "lilianweng.github.io",
  year    = "2021",
  month   = "Mar",
  url     = "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/"
}