SentimentLM: A Minimal Language Model for Text Classification

SentimentLM is a lightweight neural language modeling project for sentence-level sentiment classification.
The project explores how different tokenization strategies, embedding choices, and model architectures affect performance in natural language processing.

This work was developed as part of CSE 256 / LIGN 256 – Statistical Natural Language Processing, but is structured as a modular, end-to-end NLP system suitable for controlled experimentation and extension.

---

Project Objectives

• Compare classical and neural approaches to text classification
• Study the effects of word-level vs. subword-level tokenization
• Understand the role of pretrained vs. trainable embeddings
• Build a clean, extensible pipeline for NLP experimentation

---

Models Overview

1) Bag-of-Words (BOW)

A sparse, non-neural baseline using word frequency features.

• Input: Bag-of-words vectors from CountVectorizer
• Features: Sparse word count representations
• Classifier: Feedforward neural network (MLP)
• Embeddings: None
• Purpose: Provides a strong classical baseline for comparison

---

2) Word-Level Deep Averaging Network (DAN)

A neural language model operating on word-level tokenization with pretrained embeddings.

• Tokenization: Word-level
• Input: Every word has been converted to list of word indices based on Indexer.
• Embeddings: Pretrained GloVe (50-dimensional or 300-dimensional)
• Architecture:word tokens → embedding lookup → average pooling → MLP → sentiment label

3) Subword Deep Averaging Network (BPE-DAN)

A subword-aware variant of DAN using Byte Pair Encoding (BPE).

• Tokenization: Subword-level via BPE
• Input: Every word has been converted to list of word indices based on Indexer in BPE.
• Embeddings: Randomly initialized, fully trainable
• Architecture: subword tokens → embedding lookup → masked average pooling → MLP → sentiment label

Dataset and Task

This project uses the movie review dataset of Socher et al., consisting of short snippets from Rotten Tomatoes.

• Task: Binary sentiment classification
• Labels:
• 0 — Negative
• 1 — Positive
• Data format: The task is a positive/negative binary sentiment classification of sentences. The data files given to you have newline-separated sentiment examples, consisting of a label ( 0 or 1) followed by a tab, followed by the sentence, which has been tokenized but not lowercased. The data has been split into a train, and development (dev) set.

Features

Three model variants

• BOW baseline with sparse word counts
• Word-level DAN with pretrained GloVe embeddings
• BPE-based subword DAN with learned embeddings

Tokenization strategies

• Word-level tokenization
• Subword tokenization via Byte Pair Encoding (BPE)

Embedding options

• Pretrained GloVe embeddings (50d or 300d)
• Randomly initialized trainable embeddings

End-to-end NLP pipeline

• Dataset loading and preprocessing
• Vocabulary and indexer construction
• Padded batching via custom collate_fn
• Model training and evaluation loops

Modular and configurable

• Switch models via command-line flags
• Tune BPE merge operations
• Easily extendable to deeper architectures or new tokenizers

Code Structure

├── sentiment_data.py     # Data loading and SentimentExample definitions
├── utils.py              # Indexer and vocabulary utilities
├── BOWmodels.py          # Bag-of-Words baseline models
├── DANmodels.py          # Word-level and subword DAN models
├── BPE_tokenizer.py      # Byte Pair Encoding (BPE) implementation
├── main.py               # Training and evaluation entry point
├── data/
│   ├── train.txt
│   ├── dev.txt
│   └── glove.6B.50d-relativized.txt
└── README.md

Implementing the code:

1) Bag-of-Words Baseline:

python main.py --model BOW

• Trains and evaluates the BOW model
• Expected dev accuracy: low 70%
• Vocabulary size truncated to 512
• No pretrained embeddings
• Both the results for two-layer and three-layer models will be shown

2) Word-Level DAN:

python main.py --model DAN

• Uses both the pretrained GloVe embeddings and randomly-initialized and trainable embeddings
• Word-level tokenization
• Neural averaging-based language model

3) Subword DAN (BPE)

python main.py --model SUBWORDDAN

• Trains a BPE tokenizer on the training-set, for development set, use the same indexer as the training set.
• Uses subword-level tokenization
• Randomly initialized, trainable embeddings
• Shares the same BPE tokenizer between training and dev sets

---