📰 Building a news classifier with weak supervision

In this tutorial, we will build a news classifier using rules and weak supervision:

  • 📰 For this example, we use the AG News dataset but you can follow this process to programmatically label any dataset.

  • 🤿 The train split without labels is used to build a training set with rules, Rubrix and Snorkel’s Label model.

  • 🔧 The test set is used for evaluating our weak labels, label model and downstream news classifier.

  • 🤯 We achieve 0.82 macro avg. f1-score without using a single example from the original dataset and using a pretty lightweight model (scikit-learn’s MultinomialNB).

The following diagram shows the overall process for using Weak supervision with Rubrix:

Labeling workflow

Introduction

Weak supervision is a branch of machine learning where noisy, limited, or imprecise sources are used to provide supervision signal for labeling large amounts of training data in a supervised learning setting. This approach alleviates the burden of obtaining hand-labeled data sets, which can be costly or impractical. Instead, inexpensive weak labels are employed with the understanding that they are imperfect, but can nonetheless be used to create a strong predictive model. [Wikipedia]

For a broader introduction to weak supervision, as well as further references, we recommend the excellent overview by Alex Ratner et al..

This tutorial aims to be a practical introduction to weak supervision and will walk you through its entire process. First we will generate weak labels with Rubrix, combine these labels with Snorkel, and finally train a classifier with Scikit Learn.

Setup

Rubrix, is a free and open-source tool to explore, annotate, and monitor data for NLP projects.

If you are new to Rubrix, check out the ⭐ Github repository.

If you have not installed and launched Rubrix yet, check the Setup and Installation guide.

For this tutorial we also need some third party libraries that can be installed via pip:

[ ]:
%pip install snorkel datasets sklearn -qqq

Note

If you want to skip the first three sections of this tutorial, and only prepare the training set and train a downstream model, you can load the records directly from the Hugging Face Hub:

import rubrix as rb
from datasets import load_dataset

# this replaces the `records = label_model.predict()` line of section 4
records = rb.read_datasets(
    load_dataset("rubrix/news", split="train"),
    task="TextClassification",
)

1. Load test and unlabelled datasets into Rubrix

First, let’s download the ag_news data set and have a quick look at it.

[ ]:
from datasets import load_dataset

# load our data
dataset = load_dataset("ag_news")

# get the index to label mapping
labels = dataset["test"].features["label"].names
[10]:
import pandas as pd

# quick look at our data
with pd.option_context('display.max_colwidth', None):
    display(dataset["test"].to_pandas().head())
text label
0 Fears for T N pension after talks Unions representing workers at Turner Newall say they are 'disappointed' after talks with stricken parent firm Federal Mogul. 2
1 The Race is On: Second Private Team Sets Launch Date for Human Spaceflight (SPACE.com) SPACE.com - TORONTO, Canada -- A second\team of rocketeers competing for the #36;10 million Ansari X Prize, a contest for\privately funded suborbital space flight, has officially announced the first\launch date for its manned rocket. 3
2 Ky. Company Wins Grant to Study Peptides (AP) AP - A company founded by a chemistry researcher at the University of Louisville won a grant to develop a method of producing better peptides, which are short chains of amino acids, the building blocks of proteins. 3
3 Prediction Unit Helps Forecast Wildfires (AP) AP - It's barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar. 3
4 Calif. Aims to Limit Farm-Related Smog (AP) AP - Southern California's smog-fighting agency went after emissions of the bovine variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure. 3

Now we will log the test split of our data set to Rubrix, which we will be using for testing our label and downstream models.

[ ]:
import rubrix as rb

# build our test records
records = [
    rb.TextClassificationRecord(
        text=record["text"],
        metadata={"split": "test"},
        annotation=labels[record["label"]]
    )
    for record in dataset["test"]
]

# log the records to Rubrix
rb.log(records, name="news")

In a second step we log the train split without labels. Remember, our goal is to programmatically build a training set using rules and weak supervision.

[ ]:
# build our training records without labels
records = [
    rb.TextClassificationRecord(
        text=record["text"],
        metadata={"split": "unlabelled"},
    )
    for record in dataset["train"]
]

# log the records to Rubrix
rb.log(records, name="news")

The result of the above is the following dataset in Rubrix, with 127,600 records (120,000 unlabelled and 7,600 for testing).

You can use the web app to find good rules for programmatic labeling!

2. Interactive weak labeling: Finding and defining rules

After logging the dataset, you can find and save rules directly with the UI. Then, you can read the rules with Python to train a label or downstream model, as we’ll see in the next step.

3. Denoise weak labels with Snorkel’s Label Model

The goal at this step is to denoise the weak labels we’ve just created using rules. There are several approaches to this problem using different statistical methods.

In this tutorial, we’re going to use Snorkel but you can actually use any other Label model or weak supervision method, such as FlyingSquid for example (see the Weak supervision guide for more details). For convenience, Rubrix defines a simple wrapper over Snorkel’s Label Model so it’s easier to use with Rubrix weak labels and datasets

Let’s first read the rules defined in our dataset and create our weak labels:

[ ]:
from rubrix.labeling.text_classification import WeakLabels

weak_labels = WeakLabels(dataset="news")
[22]:
weak_labels.summary()
[22]:
label coverage annotated_coverage overlaps conflicts correct incorrect precision
money {Business} 0.008276 0.008816 0.002437 0.001936 30 37 0.447761
financ* {Business} 0.019655 0.017763 0.005893 0.005188 80 55 0.592593
dollar* {Business} 0.016591 0.016316 0.003542 0.002908 87 37 0.701613
war {World} 0.011779 0.013289 0.003213 0.001348 75 26 0.742574
gov* {World} 0.045078 0.045263 0.010878 0.006270 170 174 0.494186
minister* {World} 0.030031 0.028289 0.007531 0.002821 193 22 0.897674
conflict {World} 0.003041 0.002895 0.001003 0.000102 18 4 0.818182
footbal* {Sports} 0.013166 0.015000 0.004945 0.000439 107 7 0.938596
sport* {Sports} 0.021191 0.021316 0.007045 0.001223 139 23 0.858025
game {Sports} 0.038879 0.037763 0.014083 0.002375 216 71 0.752613
play* {Sports} 0.052453 0.050000 0.016889 0.005063 268 112 0.705263
sci* {Sci/Tech} 0.016552 0.018421 0.002735 0.001309 114 26 0.814286
techno* {Sci/Tech} 0.027218 0.028289 0.008433 0.003174 155 60 0.720930
computer* {Sci/Tech} 0.027320 0.028026 0.011058 0.004459 159 54 0.746479
software {Sci/Tech} 0.030243 0.029605 0.009655 0.003346 184 41 0.817778
web {Sci/Tech} 0.015376 0.013289 0.004067 0.001607 76 25 0.752475
total {World, Sports, Business, Sci/Tech} 0.317022 0.311447 0.053582 0.019561 2071 774 0.727944
[ ]:
from rubrix.labeling.text_classification import Snorkel

# create the label model
label_model = Snorkel(weak_labels)

# fit the model
label_model.fit()
[28]:
print(label_model.score(output_str=True))
              precision    recall  f1-score   support

      Sports       0.79      0.96      0.87       632
    Sci/Tech       0.77      0.77      0.77       773
       World       0.70      0.80      0.74       509
    Business       0.65      0.36      0.46       453

    accuracy                           0.75      2367
   macro avg       0.73      0.72      0.71      2367
weighted avg       0.74      0.75      0.73      2367

4. Prepare our training set

Now, we already have a “denoised” training set, which we can prepare for training a downstream model. The label model predict returns TextClassificationRecord objects with the predictions from the label model.

We can either refine and review these records using the Rubrix web app, use them as is, or filter them by score, for example.

In this case, we assume the predictions are precise enough and use them without any revision. Our training set has ~38,000 records, which corresponds to all records where the label model has not abstained.

[ ]:
import pandas as pd

# get records with the predictions from the label model
records = label_model.predict()
# you can replace this line with
# records = rb.read_datasets(
#    load_dataset("rubrix/news", split="train"),
#    task="TextClassification",
# )

# we could also use the `weak_labels.label2int` dict
label2int = {'Sports': 0, 'Sci/Tech': 1, 'World': 2, 'Business': 3}

# extract training data
X_train = [rec.text for rec in records]
y_train = [label2int[rec.prediction[0][0]] for rec in records]
[16]:
# quick look at our training data with the weak labels from our label model
with pd.option_context('display.max_colwidth', None):
    display(pd.DataFrame({"text": X_train, "label": y_train}))
text label
0 FA Cup: Third round draw - joy for Yeading and Exeter The great big fat bullies from across the playground enter the FA Cup arena at the third round stage, for which the draw was made yesterday (December 5). 0
1 Rats May Help Unravel Human Drug Addiction Mysteries By LAURAN NEERGAARD WASHINGTON (AP) -- Rats can become drug addicts. That's important to know, scientists say, and has taken a long time to prove... 1
2 Palmer Passes Test Bengals quarterback Carson Palmer enjoyed his breakthrough game at the expense of the Super Bowl champion Patriots, racking up 179 yards on 12-of-19 passing in a 31-3 triumph on Saturday night. 0
3 Compromises urged amid deadlock in Darfur talks ABUJA, Nigeria -- Peace talks on Sudan's violence-torn Darfur region are deadlocked, a mediator said yesterday, as the chief of the African Union appealed to the Sudanese government and rebels to compromise. 2
4 CAPELLO FED UP WITH FEIGNING Juventus coach Fabio Capello has ordered his players not to kick the ball out of play when an opponent falls to the ground apparently hurt because he believes some players fake injury to stop the match. 0
... ... ...
38080 Apple ships Mac OS X update The 26MB upgrade to version 10.3.6 is available now via the OS #39; Software Update control panel and from Apple #39;s support web site. 1
38081 Bob Evans, mainframe pioneer, dies at 77 Bob Evans, an IBM computer scientist who helped to develop the modern mainframe computer, died Thursday. He was 77. Evans died of heart failure at his at his home in the San Francisco suburb of Hillsborough, his son Evan told the Associated Press. 1
38082 For Brazil's Economy, the Doctor Is In Antonio Palocci, a doctor from Brazil's farm belt, has found himself presiding as the country's finance minister during the most robust economic expansion in a decade. 2
38083 UbiSoft Get Ready With The Next Rainbox Six Installment Ubisoft today announced its plans to launch the next installment in the Tom Clancys Rainbow Six franchise in Spring 2005. The next Rainbow Six game follows Team Rainbow, the worlds most 0
38084 PM #39;s visit to focus on reconstruction of Kashmir New Delhi: The two-day visit of Prime Minister Manmohan Singh to Jammu and Kashmir, starting on Wednesday will focus more on reconstruction and development of the state, Parliamentary Affairs Minister Ghulam Nabi Azad has said. 2

38085 rows × 2 columns

5. Train a downstream model with scikit-learn

Now, let’s train our final model using scikit-learn:

[ ]:
from sklearn.feature_extraction.text import TfidfTransformer, CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

# define our final classifier
classifier = Pipeline([
    ('vect', CountVectorizer()),
    ('clf', MultinomialNB())
])

# fit the classifier
classifier.fit(
    X=X_train,
    y=y_train,
)

To test our trained model, we use the records with validated annotations, that is the original ag_news test set.

[ ]:
# retrieve records with annotations
test_ds = weak_labels.records(has_annotation=True)
# you can replace this line with
# test_ds = rb.read_datasets(
#    load_dataset("rubrix/news_test", split="train"),
#    task="TextClassification",
# )

# extract text and labels
X_test = [rec.text for rec in test_ds]
y_test = [label2int[rec.annotation] for rec in test_ds]
[77]:
# compute the test accuracy
accuracy = classifier.score(
    X=X_test,
    y=y_test,
)

print(f"Test accuracy: {accuracy}")
Test accuracy: 0.8182894736842106

Not too bad! 🥳

We have achieved around 0.82 accuracy without even using a single example from the original ag_news train set and with a small set of 16 rules. Also, we’ve improved over the 0.75 accuracy of our Label Model.

Finally, let’s take a look at more detailed metrics:

[76]:
from sklearn import metrics

# get predictions for the test set
predicted = classifier.predict(X_test)

print(metrics.classification_report(y_test, predicted, target_names=label2int.keys()))
              precision    recall  f1-score   support

      Sports       0.86      0.98      0.91      1900
    Sci/Tech       0.76      0.84      0.80      1900
       World       0.80      0.89      0.84      1900
    Business       0.88      0.57      0.69      1900

    accuracy                           0.82      7600
   macro avg       0.82      0.82      0.81      7600
weighted avg       0.82      0.82      0.81      7600

At this point, we could go back to the UI to define more rules for those labels with less performance. Looking at the above table, we might want to add some more rules for increasing the recall of the Business label.

Summary

In this tutorial, we saw how you can leverage weak supervision to quickly build up a large training data set, and use it for the training of a first lightweight model.

Rubrix is a very handy tool to start the weak supervision process by making it easy to find a good set of starting rules, and to reiterate on them dynamically. Since Rubrix also provides built-in support for the most common label models, you can get from rules to weak labels in a few straight forward steps. For more suggestions on how to leverage weak labels, you can checkout our weak supervision guide where we describe an interesting approach to jointly train the label and a transformers downstream model.

Next steps

If you are interested in the topic of weak supervision check our weak supervision guide.

⭐ Rubrix Github repo to stay updated.

📚 Rubrix documentation for more guides and tutorials.

🙋‍♀️ Join the Rubrix community on Slack

Appendix I: Create rules and weak labels from Python

For some use cases, you might want to use Python for defining labeling rules and generating weak labels. Rubrix provides you with the ability to define and test rules and labeling functions directly using Python. This might be useful for combining it with rules defined in the UI, and for leveraging structured resources such as lexicons and gazeteers which are easier to use directly a programmatic environment.

In this section, we define the rules we’ve defined in the UI, this time directly using Python:

[ ]:
from rubrix.labeling.text_classification import Rule

# define queries and patterns for each category (using ES DSL)
queries = [
  (["money", "financ*", "dollar*"], "Business"),
  (["war", "gov*", "minister*", "conflict"], "World"),
  (["footbal*", "sport*", "game", "play*"], "Sports"),
  (["sci*", "techno*", "computer*", "software", "web"], "Sci/Tech")
]

# define rules
rules = [
    Rule(query=term, label=label)
    for terms,label in queries
    for term in terms
]
[ ]:
from rubrix.labeling.text_classification import WeakLabels

# generate the weak labels
weak_labels = WeakLabels(
    rules=rules,
    dataset="news"
)

On our machine it took around 24 seconds to apply the rules and to generate weak labels for the 127,600 examples.

Typically, you want to iterate on the rules and check their statistics. For this, you can use weak_labels.summary method:

[78]:
weak_labels.summary()
[78]:
label coverage annotated_coverage overlaps conflicts correct incorrect precision
money {Business} 0.008276 0.008816 0.002437 0.001936 30 37 0.447761
financ* {Business} 0.019655 0.017763 0.005893 0.005188 80 55 0.592593
dollar* {Business} 0.016591 0.016316 0.003542 0.002908 87 37 0.701613
war {World} 0.011779 0.013289 0.003213 0.001348 75 26 0.742574
gov* {World} 0.045078 0.045263 0.010878 0.006270 170 174 0.494186
minister* {World} 0.030031 0.028289 0.007531 0.002821 193 22 0.897674
conflict {World} 0.003041 0.002895 0.001003 0.000102 18 4 0.818182
footbal* {Sports} 0.013166 0.015000 0.004945 0.000439 107 7 0.938596
sport* {Sports} 0.021191 0.021316 0.007045 0.001223 139 23 0.858025
game {Sports} 0.038879 0.037763 0.014083 0.002375 216 71 0.752613
play* {Sports} 0.052453 0.050000 0.016889 0.005063 268 112 0.705263
sci* {Sci/Tech} 0.016552 0.018421 0.002735 0.001309 114 26 0.814286
techno* {Sci/Tech} 0.027218 0.028289 0.008433 0.003174 155 60 0.720930
computer* {Sci/Tech} 0.027320 0.028026 0.011058 0.004459 159 54 0.746479
software {Sci/Tech} 0.030243 0.029605 0.009655 0.003346 184 41 0.817778
web {Sci/Tech} 0.015376 0.013289 0.004067 0.001607 76 25 0.752475
total {World, Sports, Business, Sci/Tech} 0.317022 0.311447 0.053582 0.019561 2071 774 0.727944

From the above, we see that our rules cover around 30% of the original training set with an average precision of 0.73. Our hope is that the label and downstream models will improve both the recall and the precision of the final classifier.

Appendix II: Log datasets to the Hugging Face Hub

Here we will show you how we pushed our Rubrix datasets (records) to the Hugging Face Hub. In this way you can effectively version any of your Rubrix datasets.

[ ]:
train_rb = rb.DatasetForTextClassification(label_model.predict())
train_rb.to_datasets().push_to_hub("rubrix/news")
[ ]:
test_rb = rb.load("news", query="status:Validated")
test_rb.to_datasets().push_to_hub("rubrix/news_test")