NLP Project

NLP Disaster Tweets Classification

Download public disaster tweet CSVs, clean text lightly, train a TF-IDF baseline, evaluate F1, inspect errors, and create a submission-style file.

Final deliverable: A text-classification notebook with validation F1 and a submission.csv file.

Dataset: Disaster Tweets public mirror

The notebook downloads public train/test tweet CSVs. The target label indicates whether a tweet describes a real disaster.

text: tweet text
keyword: optional keyword extracted from the tweet
location: optional user-provided location
target: 1 for real disaster, 0 otherwise

Step 1

Download Disaster Tweet Data

Download public train.csv and test.csv files.

Your task

Run the public download cell and verify files are present.

Try writing this cell

# Run this cell in Colab, Kaggle, or a local notebook.
# It downloads public dataset files directly, so you do not need Kaggle credentials.

from pathlib import Path
from urllib.request import urlretrieve
import zipfile

DATA_DIR = Path("data/nlp-getting-started")
DATA_DIR.mkdir(parents=True, exist_ok=True)

def find_file(name):
    possible_roots = [DATA_DIR, Path("/kaggle/input")]
    for root in possible_roots:
        if root.exists():
            matches = list(root.rglob(name))
            if matches:
                return matches[0]
    raise FileNotFoundError(f"Could not find {name}. Run the dataset download cell first.")

files = {
    "train.csv": "https://huggingface.co/datasets/VuduVations/disaster_tweets/resolve/main/data/train.csv",
    "test.csv": "https://huggingface.co/datasets/VuduVations/disaster_tweets/resolve/main/data/test.csv"
}

for filename, url in files.items():
    target = DATA_DIR / filename
    if target.exists() and target.stat().st_size > 0:
        print(f"already exists: {target}")
        continue
    print(f"downloading {filename} ...")
    urlretrieve(url, target)
    print(f"saved: {target}")

for zip_path in DATA_DIR.glob("*.zip"):
    extract_dir = DATA_DIR / zip_path.stem
    extract_dir.mkdir(exist_ok=True)
    if any(extract_dir.iterdir()):
        print(f"already extracted: {extract_dir}")
        continue
    with zipfile.ZipFile(zip_path, "r") as zf:
        zf.extractall(extract_dir)
    print(f"extracted: {zip_path} -> {extract_dir}")

print("Files:")
all_files = [path for path in sorted(DATA_DIR.rglob("*")) if path.is_file()]
for path in all_files[:40]:
    print(path)
if len(all_files) > 40:
    print(f"... {len(all_files) - 40} more files not shown")

Expected output

A file list containing train.csv and test.csv.

Show reference solution

# Run this cell in Colab, Kaggle, or a local notebook.
# It downloads public dataset files directly, so you do not need Kaggle credentials.

from pathlib import Path
from urllib.request import urlretrieve
import zipfile

DATA_DIR = Path("data/nlp-getting-started")
DATA_DIR.mkdir(parents=True, exist_ok=True)

def find_file(name):
    possible_roots = [DATA_DIR, Path("/kaggle/input")]
    for root in possible_roots:
        if root.exists():
            matches = list(root.rglob(name))
            if matches:
                return matches[0]
    raise FileNotFoundError(f"Could not find {name}. Run the dataset download cell first.")

files = {
    "train.csv": "https://huggingface.co/datasets/VuduVations/disaster_tweets/resolve/main/data/train.csv",
    "test.csv": "https://huggingface.co/datasets/VuduVations/disaster_tweets/resolve/main/data/test.csv"
}

for filename, url in files.items():
    target = DATA_DIR / filename
    if target.exists() and target.stat().st_size > 0:
        print(f"already exists: {target}")
        continue
    print(f"downloading {filename} ...")
    urlretrieve(url, target)
    print(f"saved: {target}")

for zip_path in DATA_DIR.glob("*.zip"):
    extract_dir = DATA_DIR / zip_path.stem
    extract_dir.mkdir(exist_ok=True)
    if any(extract_dir.iterdir()):
        print(f"already extracted: {extract_dir}")
        continue
    with zipfile.ZipFile(zip_path, "r") as zf:
        zf.extractall(extract_dir)
    print(f"extracted: {zip_path} -> {extract_dir}")

print("Files:")
all_files = [path for path in sorted(DATA_DIR.rglob("*")) if path.is_file()]
for path in all_files[:40]:
    print(path)
if len(all_files) > 40:
    print(f"... {len(all_files) - 40} more files not shown")

Why this step matters

The dataset setup is part of the notebook so learners can reproduce the exact workflow.

Common mistake

Assuming the tweet text is already clean enough for every model without inspecting examples.

Step 2

Load and Inspect Text

Read training tweets and understand target balance.

Your task

Load train.csv, print shape, target distribution, and a few text examples.

Try writing this cell

from pathlib import Path
import pandas as pd

possible_roots = [
    Path("data/nlp-getting-started"),
    Path("/kaggle/input"),
]

def find_file(name):
    for root in possible_roots:
        if root.exists():
            matches = list(root.rglob(name))
            if matches:
                return matches[0]
    raise FileNotFoundError(f"Could not find {name}. Run the dataset download/load cell first.")

train_path = find_file("train.csv")
df = pd.read_csv(train_path)

print(train_path)
print(df.shape)
df.head()

# Check target balance

# Display sample tweets

Expected output

Target distribution and sample disaster/non-disaster tweets.

Show reference solution

from pathlib import Path
import pandas as pd

possible_roots = [
    Path("data/nlp-getting-started"),
    Path("/kaggle/input"),
]

def find_file(name):
    for root in possible_roots:
        if root.exists():
            matches = list(root.rglob(name))
            if matches:
                return matches[0]
    raise FileNotFoundError(f"Could not find {name}. Run the dataset download/load cell first.")

train_path = find_file("train.csv")
df = pd.read_csv(train_path)

print(train_path)
print(df.shape)
df.head()

display(df["target"].value_counts(normalize=True))
display(df[["text", "keyword", "location", "target"]].sample(8, random_state=42))

Why this step matters

Text projects need qualitative inspection. Many examples contain URLs, punctuation, abbreviations, and ambiguous wording.

Common mistake

Cleaning too aggressively and deleting useful words like fire, flood, crash, or evacuation.

Step 3

Train a TF-IDF Baseline

Create a strong simple baseline with TF-IDF and logistic regression.

Your task

Split the data, vectorize text, train LogisticRegression, and evaluate F1.

Try writing this cell

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

# Split text and labels

# Train pipeline

# Evaluate F1

Expected output

Validation F1 and precision/recall by class.

Show reference solution

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

X_train, X_val, y_train, y_val = train_test_split(
    df["text"], df["target"], test_size=0.2, random_state=42, stratify=df["target"]
)

model = Pipeline([
    ("tfidf", TfidfVectorizer(min_df=2, ngram_range=(1, 2), stop_words="english")),
    ("clf", LogisticRegression(max_iter=1000)),
])

model.fit(X_train, y_train)
val_preds = model.predict(X_val)

print("F1:", f1_score(y_val, val_preds))
print(classification_report(y_val, val_preds))

Why this step matters

TF-IDF is a strong baseline for many text tasks. It is fast, explainable, and a useful benchmark before transformers.

Common mistake

Using accuracy only; this competition is evaluated with F1.

Step 4

Inspect False Positives and False Negatives

Read model mistakes to understand ambiguity.

Your task

Create a validation table and show examples where prediction differs from target.

Try writing this cell

# Build validation error table

# Show false positives

# Show false negatives

Expected output

Tables of tweets the model over-called and missed.

Show reference solution

error_df = pd.DataFrame({
    "text": X_val,
    "actual": y_val,
    "predicted": val_preds,
})

false_pos = error_df[(error_df["actual"] == 0) & (error_df["predicted"] == 1)]
false_neg = error_df[(error_df["actual"] == 1) & (error_df["predicted"] == 0)]

print("False positives")
display(false_pos.head(8))
print("False negatives")
display(false_neg.head(8))

Why this step matters

NLP errors are often semantic. Reading mistakes shows sarcasm, metaphor, missing context, and keyword traps.

Common mistake

Only tuning vectorizer parameters without reading actual failed tweets.

Step 5

Try Keyword Plus Text

Experiment with adding keyword as extra text signal.

Your task

Create a combined text field that includes keyword when available and retrain the same baseline.

Try writing this cell

# Create combined_text from keyword and text

# Retrain and compare F1

Expected output

An F1 score that can be compared with text-only baseline.

Show reference solution

df["keyword_clean"] = df["keyword"].fillna("").str.replace("%20", " ", regex=False)
df["combined_text"] = (df["keyword_clean"] + " " + df["text"]).str.strip()

X_train, X_val, y_train, y_val = train_test_split(
    df["combined_text"], df["target"], test_size=0.2, random_state=42, stratify=df["target"]
)

model.fit(X_train, y_train)
val_preds = model.predict(X_val)
print("F1 with keyword:", f1_score(y_val, val_preds))

Why this step matters

Feature experiments should be compared against the baseline. Extra metadata helps only if it improves validation behavior.

Common mistake

Adding features and assuming they help without comparing the metric.

Step 6

Create Submission-Style CSV

Train on full data, predict test tweets, and save submission.csv.

Your task

Prepare test combined_text, predict target, and save id/target submission.

Try writing this cell

# Load test.csv

# Create same combined_text field

# Fit on full train and predict test

# Save submission.csv

Expected output

submission.csv with id and target columns.

Show reference solution

test_df = pd.read_csv(find_file("test.csv"))
test_df["keyword_clean"] = test_df["keyword"].fillna("").str.replace("%20", " ", regex=False)
test_df["combined_text"] = (test_df["keyword_clean"] + " " + test_df["text"]).str.strip()

model.fit(df["combined_text"], df["target"])
test_preds = model.predict(test_df["combined_text"])

submission = pd.DataFrame({
    "id": test_df["id"],
    "target": test_preds,
})
submission.to_csv("submission.csv", index=False)
display(submission.head())

Why this step matters

The final submission should use the same preprocessing as validation, then train on all labeled data.

Common mistake

Creating combined_text in train but not in test.