Construct a strong text-based toxicity predictor

With the expansion and recognition of on-line social platforms, folks can keep extra related than ever by way of instruments like on the spot messaging. Nonetheless, this raises an extra concern about poisonous speech, in addition to cyber bullying, verbal harassment, or humiliation. Content material moderation is essential for selling wholesome on-line discussions and creating wholesome on-line environments. To detect poisonous language content material, researchers have been creating deep learning-based pure language processing (NLP) approaches. Most up-to-date strategies make use of transformer-based pre-trained language fashions and obtain excessive toxicity detection accuracy.

In real-world toxicity detection functions, toxicity filtering is generally utilized in security-relevant industries like gaming platforms, the place fashions are continually being challenged by social engineering and adversarial assaults. Consequently, straight deploying text-based NLP toxicity detection fashions may very well be problematic, and preventive measures are essential.

Analysis has proven that deep neural community fashions don’t make correct predictions when confronted with adversarial examples. There was a rising curiosity in investigating the adversarial robustness of NLP fashions. This has been completed with a physique of newly developed adversarial assaults designed to idiot machine translation, query answering, and textual content classification programs.

On this publish, we practice a transformer-based toxicity language classifier utilizing Hugging Face, take a look at the educated mannequin on adversarial examples, after which carry out adversarial coaching and analyze its impact on the educated toxicity classifier.

Resolution overview

Adversarial examples are deliberately perturbed inputs, aiming to mislead machine studying (ML) fashions in direction of incorrect outputs. Within the following instance (supply: https://aclanthology.org/2020.emnlp-demos.16.pdf), by altering simply the phrase “Excellent” to “Spotless,” the NLP mannequin gave a totally reverse prediction.

Social engineers can use any such attribute of NLP fashions to bypass toxicity filtering programs. To make text-based toxicity prediction fashions extra strong in opposition to deliberate adversarial assaults, the literature has developed a number of strategies. On this publish, we showcase one in all them—adversarial coaching, and the way it improves textual content toxicity prediction fashions’ adversarial robustness.

Adversarial coaching

Profitable adversarial examples reveal the weak point of the goal sufferer ML mannequin, as a result of the mannequin couldn’t precisely predict the label of those adversarial examples. By retraining the mannequin with a mixture of authentic coaching knowledge and profitable adversarial examples, the retrained mannequin shall be extra strong in opposition to future assaults. This course of is known as adversarial coaching.

TextAttack Python library

TextAttack is a Python library for producing adversarial examples and performing adversarial coaching to enhance NLP fashions’ robustness. This library gives implementations of a number of state-of-the-art textual content adversarial assaults from the literature and helps a wide range of fashions and datasets. Its code and tutorials can be found on GitHub.

Dataset

The Poisonous Remark Classification Problem on Kaggle gives a lot of Wikipedia feedback which have been labeled by human raters for poisonous habits. The varieties of toxicity are:

• poisonous
• severe_toxic
• obscene
• menace
• insult
• identity_hate

On this publish, we solely predict the poisonous column. The practice set accommodates 159,571 situations with 144,277 non-toxic and 15,294 poisonous examples, and the take a look at set accommodates 63,978 situations with 57,888 non-toxic and 6,090 poisonous examples. We break up the take a look at set into validation and take a look at units, which include 31,989 situations every with 29,028 non-toxic and a pair of,961 poisonous examples. The next charts illustrate our knowledge distribution.

For the aim of demonstration, this publish randomly samples 10,000 situations for coaching, and 1,000 for validation and testing every, with every dataset balanced on each lessons. For particulars, discuss with our pocket book.

Practice a transformer-based poisonous language classifier

Step one is to coach a transformer-based poisonous language classifier. We use the pre-trained DistilBERT language mannequin as a base and fine-tune the mannequin on the Jigsaw poisonous remark classification coaching dataset.

Tokenization

Tokens are the constructing blocks of pure language inputs. Tokenization is a approach of separating a chunk of textual content into tokens. Tokens can take a number of kinds, both phrases, characters, or subwords. To ensure that the fashions to grasp the enter textual content, a tokenizer is used to arrange the inputs for an NLP mannequin. A number of examples of tokenizing embody splitting strings into subword token strings, changing token strings to IDs, and including new tokens to the vocabulary.

Within the following code, we use the pre-trained DistilBERT tokenizer to course of the practice and take a look at datasets:

pretrained_model_name_or_path = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path)

def preprocess_function(examples):
    end result = tokenizer(
        examples["text"], padding="max_length", max_length=128, truncation=True
    )
    return end result

train_dataset = train_dataset.map(
    preprocess_function, batched=True, load_from_cache_file=False, num_proc=num_proc
)

valid_dataset = valid_dataset.map(
    preprocess_function, batched=True, load_from_cache_file=False, num_proc=num_proc
)

test_dataset = test_dataset.map(
    preprocess_function, batched=True, load_from_cache_file=False, num_proc=num_proc
)

For every enter textual content, the DistilBERT tokenizer outputs 4 options:

• textual content – Enter textual content.
• labels – Output labels.
• input_ids – Indexes of enter sequence tokens in a vocabulary.
• attention_mask – Masks to keep away from performing consideration on padding token indexes. Masks values chosen are [0, 1]:
  • 1 for tokens that aren’t masked.
  • 0 for tokens which might be masked.

Now that we’ve the tokenized dataset, the subsequent step is to coach the binary poisonous language classifier.

Modeling

Step one is to load the bottom mannequin, which is a pre-trained DistilBERT language mannequin. The mannequin is loaded with the Hugging Face Transformers class AutoModelForSequenceClassification:

base_model = AutoModelForSequenceClassification.from_pretrained(
    pretrained_model_name_or_path, num_labels=1
)

Then we customise the hyperparameters utilizing class TrainingArguments. The mannequin is educated with batch dimension 32 on 10 epochs with studying fee of 5e-6 and warmup steps of 500. The educated mannequin is saved in model_dir, which was outlined to start with of the pocket book.

training_args = TrainingArguments(
    output_dir=model_dir,
    num_train_epochs=10,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=5,
    logging_dir=os.path.be part of(model_dir, "logs"),
    learning_rate=5e-6,
    load_best_model_at_end=True,
    metric_for_best_model="loss",
    disable_tqdm=True,
)

To judge the mannequin’s efficiency throughout coaching, we have to present the Coach with an analysis operate. Right here we’re report accuracy, F1 scores, common precision, and AUC scores.

# compute metrics operate
def compute_metrics(pred):
    targets = 1 * (pred.label_ids >= 0.5)
    outputs = 1 * (pred.predictions >= 0.5)
    accuracy = metrics.accuracy_score(targets, outputs)
    f1_score_micro = metrics.f1_score(targets, outputs, common="micro")
    f1_score_macro = metrics.f1_score(targets, outputs, common="macro")
    f1_score_weighted = metrics.f1_score(targets, outputs, common="weighted")
    ap_score_micro = metrics.average_precision_score(
        targets, pred.predictions, common="micro"
    )
    ap_score_macro = metrics.average_precision_score(
        targets, pred.predictions, common="macro"
    )
    ap_score_weighted = metrics.average_precision_score(
        targets, pred.predictions, common="weighted"
    )
    auc_score_micro = metrics.roc_auc_score(targets, pred.predictions, common="micro")
    auc_score_macro = metrics.roc_auc_score(targets, pred.predictions, common="macro")
    auc_score_weighted = metrics.roc_auc_score(
        targets, pred.predictions, common="weighted"
    )
    return {
        "accuracy": accuracy,
        "f1_score_micro": f1_score_micro,
        "f1_score_macro": f1_score_macro,
        "f1_score_weighted": f1_score_weighted,
        "ap_score_micro": ap_score_micro,
        "ap_score_macro": ap_score_macro,
        "ap_score_weighted": ap_score_weighted,
        "auc_score_micro": auc_score_micro,
        "auc_score_macro": auc_score_macro,
        "auc_score_weighted": auc_score_weighted,
    }

The Coach class gives an API for feature-complete coaching in PyTorch. Let’s instantiate the Coach by offering the bottom mannequin, coaching arguments, coaching and analysis dataset, in addition to the analysis operate:

coach = Coach(
    mannequin=base_model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=valid_dataset,
    compute_metrics=compute_metrics,
)

After the Coach is instantiated, we are able to kick off the coaching course of:

train_result = coach.practice()

When the coaching course of is completed, we save the tokenizer and mannequin artifacts regionally:

tokenizer.save_pretrained(model_dir)
coach.save_model(model_dir)

Consider the mannequin robustness

On this part, we attempt to reply one query: how strong is our toxicity filtering mannequin in opposition to text-based adversarial assaults? To reply this query, we choose an assault recipe from the TextAttack library and use it to assemble perturbed adversarial examples to idiot our goal toxicity filtering mannequin. Every assault recipe generates textual content adversarial examples by remodeling seed textual content inputs into barely modified textual content samples, whereas ensuring the seed and its perturbed textual content comply with sure language constraints (for instance, semantic preserved). If these newly generated examples trick a goal mannequin into mistaken classifications, the assault is profitable; in any other case, the assault fails for that seed enter.

A goal mannequin’s adversarial robustness is evaluated by way of the Assault Success Fee (ASR) metric. ASR is outlined because the ratio of profitable assaults in opposition to all of the assaults. The decrease the ASR, the extra strong a mannequin is in opposition to adversarial assaults.

First, we outline a customized mannequin wrapper to wrap the tokenization and mannequin prediction collectively. This step additionally makes positive the prediction outputs meet the required output codecs by the TextAttack library.

class CustomModelWrapper(ModelWrapper):
    def __init__(self, mannequin):
        self.mannequin = mannequin

    def __call__(self, text_input_list):
        gadget = self.mannequin.gadget
        encoded_input = tokenizer(
            text_input_list,
            truncation=True,
            padding="max_length",
            max_length=128,
            return_tensors="pt",
        ).to(gadget)     
        # print(encoded_input.gadget)

        with torch.no_grad():
            output = self.mannequin(**encoded_input)
        logits = output.logits
        preds = torch.sigmoid(logits)
        preds = preds.squeeze(dim=-1)
        final_preds = torch.stack((1 - preds, preds), dim=1)
        return final_preds

Now we load the educated mannequin and create a customized mannequin wrapper utilizing the educated mannequin:

trained_model = AutoModelForSequenceClassification.from_pretrained(model_dir)
trained_model = trained_model.to("cuda:0")

model_wrapper = CustomModelWrapper(trained_model)

Generate assaults

Now we have to put together the dataset as seed for an assault recipe. Right here we solely use these poisonous examples as seeds, as a result of in a real-world situation, the social engineer will principally attempt to perturb poisonous examples to idiot a goal filtering mannequin as benign. Assaults might take time to generate; for the aim of this publish, we randomly pattern 1,000 poisonous coaching samples to assault.

We generate the adversarial examples for each take a look at and practice datasets. We use take a look at adversarial examples for robustness analysis and the practice adversarial examples for adversarial coaching.

threshold = 0.5
sub_sample_to_attack = 1000
df_train_to_attack = df_train[df_train['labels']==1].pattern(sub_sample_to_attack)

## We assault the poisonous samples
## Objective is to perturbe poisonous samples sufficient that the mannequin classifies them as Non-toxic
test_dataset_to_attack = textattack.datasets.Dataset(
    [
        (x, 1)
        for x, y in zip(
            test_dataset["text"], 
            test_dataset["labels"], 
        )
        if y > threshold
    ]
)

train_dataset_to_attack = textattack.datasets.Dataset(
    [
        (x, 1)
        for x, y in zip(
            df_train_to_attack["text"],
            df_train_to_attack["labels"],
        )
        if y > threshold
    ]
)

Then we outline the operate to generate the assaults:

def generate_attacks(
    recipe, model_wrapper, dataset_to_attack, num_examples=-1, parallel=False
):
    print(f"The Assault Recipe is: {recipe}")
    if recipe == "textfoolerjin2019":
        assault = TextFoolerJin2019.construct(model_wrapper)
    elif recipe == "a2t_yoo_2021":
        assault = A2TYoo2021.construct(model_wrapper)
    elif recipe == "Pruthi2019":
        assault = Pruthi2019.construct(model_wrapper)
    elif recipe == "TextBuggerLi2018":
        assault = TextBuggerLi2018.construct(model_wrapper)
    elif recipe == "DeepWordBugGao2018":
        assault = DeepWordBugGao2018.construct(model_wrapper)

    attack_args = textattack.AttackArgs(
        num_examples=num_examples, parallel=parallel, num_workers_per_device=5
    )  
    ## num_examples = -1 means all the dataset
    attacker = Attacker(assault, dataset_to_attack, attack_args)
    attack_results = attacker.attack_dataset()
    return attack_results

Select an assault recipe and generate assaults:

%%time
recipe="textfoolerjin2019"
test_attack_results = generate_attacks(recipe, model_wrapper, test_dataset_to_attack, num_examples=-1)
train_attack_results = generate_attacks(recipe, model_wrapper, train_dataset_to_attack, num_examples=-1)

Log the assault outcomes right into a Pandas knowledge body:

def log_attack_results(attack_results):
    exception_ids = []
    logger = CSVLogger(color_method="html")
    
    for i in vary(len(attack_results)):
        strive:
            end result = attack_results[i]
            logger.log_attack_result(end result)
        besides:
            exception_ids.append(i)
    df_attacks = logger.df
    return df_attacks, exception_ids


df_attacks_test, test_exception_ids = log_attack_results(test_attack_results)
df_attacks_train, train_exception_ids = log_attack_results(train_attack_results)

The assault outcomes include original_text, perturbed_text, original_output, and perturbed_output. When the perturbed_output is the alternative of the original_output, the assault is profitable.

show(
    HTML(df_attacks_test[["original_text", "perturbed_text"]].head().to_html(escape=False))
)

The purple textual content represents a profitable assault, and the inexperienced represents a failed assault.

Consider the mannequin robustness by way of ASR

Use the next code to guage the mannequin robustness:

ASR_test = (
    df_attacks_test.result_type.value_counts()["Successful"]
    / df_attacks_test.result_type.value_counts().sum()
)

ASR_train = (
    df_attacks_train.result_type.value_counts()["Successful"]
    / df_attacks_train.result_type.value_counts().sum()
)

print(f"The Assault Success Fee of the mannequin towards take a look at dataset is {ASR_test*100}%")

print(f"The Assault Success Fee of the mannequin towards practice dataset is {ASR_train*100}%")

This returns the next:

The Assault Success Fee of the mannequin towards take a look at dataset is 52.400000000000006%
The Assault Success Fee of the mannequin towards practice dataset is 51.1%

Put together profitable assaults

With all of the assault outcomes accessible, we take the profitable assault from the practice adversarial examples and use them to retrain the mannequin:

# Provide the unique labels to the profitable assaults
# Right here the unique labels are all 1, there are additionally some datasets with fractional labels between 0-1

df_attacks_train = df_attacks_train[["perturbed_text", "result_type"]].copy()
df_attacks_train["labels"] = df_train_to_attack["labels"].reset_index(drop=True)

# Clear the textual content
df_attacks_train["text"] = df_attacks_train["perturbed_text"].substitute(
    "<font shade = .{1,6}>|</font>", "", regex=True
)
df_attacks_train["text"] = df_attacks_train["text"].substitute("<SPLIT>", "n", regex=True)

# Put together knowledge so as to add to the coaching dataset
df_succ_attacks_train = df_attacks_train.loc[
    df_attacks_train.result_type == "Successful", ["text", "labels"]
]
df_succ_attacks_train.form, df_succ_attacks_train.head(2)

Adversarial coaching

On this part, we mix the profitable adversarial assaults from the coaching knowledge with the unique coaching knowledge, then practice a brand new mannequin on this mixed dataset. This mannequin is known as the adversarial educated mannequin.

# New Practice: Unique Practice + Profitable Assaults on Unique Practice

df_train_attacked = pd.concat([df_train, df_succ_attacks_train], ignore_index=True)
data_train_attacked = Dataset.from_pandas(df_train_attacked)
data_train_attacked = data_train_attacked.map(
    preprocess_function, batched=True, load_from_cache_file=False, num_proc=num_proc
)

training_args_AT = TrainingArguments(
    output_dir=model_dir_AT,
    num_train_epochs=10,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=5,
    logging_dir=os.path.be part of(model_dir, "logs"),
    learning_rate=5e-6,
    load_best_model_at_end=True,
    metric_for_best_model="loss",
    disable_tqdm=True,
)

trainer_AT = Coach(
    mannequin=base_model,
    args=training_args_AT,
    train_dataset=data_train_attacked,
    eval_dataset=valid_dataset,
    compute_metrics=compute_metrics
)

trainer_AT.practice()

Save the adversarial educated mannequin to native listing model_dir_AT:

tokenizer.save_pretrained(model_dir_AT)
trainer_AT.save_model(model_dir_AT)

Consider the robustness of the adversarial educated mannequin

Now the mannequin is adversarially educated, we wish to see how the mannequin robustness modifications accordingly:

trained_model_AT = AutoModelForSequenceClassification.from_pretrained(model_dir_AT)
trained_model_AT = trained_model_AT.to("cuda:0")
trained_model_AT.gadget

model_wrapper_AT = CustomModelWrapper(trained_model_AT)
test_attack_results_AT = generate_attacks(recipe, model_wrapper_AT, test_dataset_to_attack, num_examples=-1)
df_attacks_AT_test, AT_test_exception_ids = log_attack_results(test_attack_results_AT)

ASR_AT_test = (
    df_attacks_AT_test.result_type.value_counts()["Successful"]
    / df_attacks_AT_test.result_type.value_counts().sum()
)

print(f"The Assault Success Fee of the mannequin is {ASR_AT_test*100}%")

The previous code returns the next outcomes:

The Assault Success Fee of the mannequin is nineteen.8%

Examine the robustness of the unique mannequin and the adversarial educated mannequin:

print(
    f"The ASR of the Adversarial Educated mannequin has a {(ASR_test - ASR_AT_test)/ASR_test*100}% lower evaluate with the unique mannequin. This proves that the Adversarial Coaching improves the mannequin's robustness in opposition to the assaults."
)

This returns the next:

The ASR of the Adversarial Educated mannequin has a 62.213740458015266% lower
evaluate with the unique mannequin. This proves that the Adversarial Coaching
improves the mannequin's robustness in opposition to the assaults.

Thus far, we’ve educated a DistilBERT-based binary toxicity language classifier, examined its robustness in opposition to adversarial textual content assaults, carried out adversarial coaching to acquire a brand new toxicity language classifier, and examined the brand new mannequin’s robustness in opposition to adversarial textual content assaults.

We observe that the adversarial educated mannequin has a decrease ASR, with an 62.21% lower utilizing the unique mannequin ASR because the benchmark. This means that the mannequin is extra strong in opposition to sure adversarial assaults.

Mannequin efficiency analysis

Apart from mannequin robustness, we’re additionally eager about studying how a mannequin predicts on clear samples after it’s adversarially educated. Within the following code, we use batch prediction mode to hurry up the analysis course of:

def batch_predict(model_wrapper, text_list, batch_size=64):
    """This operate performs batch prediction for given mannequin nad textual content checklist"""
    predictions = []
    for i in tqdm(vary(0, len(text_list), batch_size)):
       batch = text_list[i : i + batch_size]
       model_predictions = model_wrapper(batch)[:, 1]
       model_predictions = model_predictions.cpu().numpy()
       predictions.append(model_predictions)
       predictions = np.concatenate(predictions, axis=0)
    return predictions

Consider the unique mannequin

We use the next code to guage the unique mannequin:

test_text_list = df_test.textual content.to_list()

model_predictions = batch_predict(model_wrapper, test_text_list, batch_size=64)

y_true_prob = np.array(df_test["labels"])
y_true = [0 if x < 0.5 else 1 for x in y_true_prob]

threshold = 0.5
y_pred_prob = model_predictions.flatten()
y_pred = [0 if x < threshold else 1 for x in y_pred_prob]

fig, ax = plt.subplots(figsize=(10, 10))
conf_matrix = confusion_matrix(y_true, y_pred)
ConfusionMatrixDisplay(conf_matrix).plot(ax=ax)
print(classification_report(y_true, y_pred))

The next figures summarize our findings.

Consider the adversarial educated mannequin

Use the next code to guage the adversarial educated mannequin:

model_predictions_AT = batch_predict(model_wrapper_AT, test_text_list, batch_size=64)

y_pred_prob_AT = model_predictions_AT.flatten()
y_pred_AT = [0 if x < threshold else 1 for x in y_pred_prob_AT]

fig, ax = plt.subplots(figsize=(10, 10))
conf_matrix = confusion_matrix(y_true, y_pred_AT)
ConfusionMatrixDisplay(conf_matrix).plot(ax=ax)
print(classification_report(y_true, y_pred_AT))

The next figures summarize our findings.

We observe that the adversarial educated mannequin tended to foretell extra examples as poisonous (801 predicted as 1) in contrast with the unique mannequin (763 predicted as 1), which results in a rise in recall of the poisonous class and precision of the non-toxic class, and a drop in precision of the poisonous class and recall of the non-toxic class. This may because of the truth that extra of the poisonous class is seen within the adversarial coaching course of.

Abstract

As a part of content material moderation, toxicity language classifiers are used to filter poisonous content material and create more healthy on-line environments. Actual-world deployment of toxicity filtering fashions requires not solely excessive prediction efficiency, but in addition for being strong in opposition to social engineering, like adversarial assaults. This publish gives a step-by-step course of from coaching a toxicity language classifier to enhance its robustness with adversarial coaching. We present that adversarial coaching can assist a mannequin grow to be extra strong in opposition to assaults whereas sustaining excessive mannequin efficiency. For extra details about this up-and-coming subject, we encourage you to discover and take a look at our script by yourself. You’ll be able to entry the pocket book on this publish from the AWS Examples GitHub repo.

Hugging Face and AWS introduced a partnership earlier in 2022 that makes it even simpler to coach Hugging Face fashions on SageMaker. This performance is on the market by way of the event of Hugging Face AWS DLCs. These containers embody the Hugging Face Transformers, Tokenizers, and Datasets libraries, which permit us to make use of these sources for coaching and inference jobs. For an inventory of the accessible DLC pictures, see Out there Deep Studying Containers Pictures. They’re maintained and often up to date with safety patches.

You’ll find many examples of tips on how to practice Hugging Face fashions with these DLCs within the following GitHub repo.

AWS affords pre-trained AWS AI companies that may be built-in into functions utilizing API calls and require no ML expertise. For instance, Amazon Comprehend can carry out NLP duties equivalent to customized entity recognition, sentiment evaluation, key phrase extraction, subject modeling, and extra to assemble insights from textual content. It may carry out textual content evaluation on all kinds of languages for its varied options.

References

Concerning the Authors

Yi Xiang is a Knowledge Scientist II on the Amazon Machine Studying Options Lab, the place she helps AWS clients throughout completely different industries speed up their AI and cloud adoption.

Yanjun Qi is a Principal Utilized Scientist on the Amazon Machine Studying Resolution Lab. She innovates and applies machine studying to assist AWS clients pace up their AI and cloud adoption.