🧠 Machine Learning in Robotics

MSc Robotics and Artificial Intelligence — Course Repository

"A robot that cannot perceive its world cannot act within it. Teaching machines to see road signs, segment lane markings, and cluster scenes without labels is not merely computer vision — it is the foundation of autonomous robotic intelligence."

Author: Umer Ahmed Baig Mughal
Programme: MSc Robotics and Artificial Intelligence
Specialization: Machine Learning · Computer Vision · Human-Robot Interaction · Autonomous Systems · Robotic Motion Control
Institution: ITMO University — Faculty of Control Systems and Robotics

---

📋 Table of Contents

• 📖 About This Repository
• 🗂️ Repository Structure
• 🔬 Course Overview
• 🧪 Lab Summaries
  • 🔵 Lab 1 — Road Sign Classification with Custom CNN and Transfer Learning
  • 🟠 Lab 2 — Road Sign Segmentation with SegNet and Data Augmentation
  • 🟣 Lab 3 — Road Scene Reconstruction and Clustering with Convolutional Autoencoder
  • 🟢 Lab 4 — Autoencoder Comparison: Latent Space Interpolation, Vector Arithmetic and Denoising
• 🔗 Progressive Learning Pathway
• ⚙️ Common Platform — Shared Dataset and Training Infrastructure
• 🚀 Quick Start
• 📊 Results at a Glance
• 🧰 Tech Stack
• 👤 Author
• 📄 License

---

📖 About This Repository

This repository contains the complete implementation of all four laboratory assignments from the Machine Learning in Robotics (MLR) course, part of the MSc in Robotics and Artificial Intelligence at ITMO University. The labs form a carefully sequenced progression — from supervised image-level classification to pixel-level segmentation to fully unsupervised latent space exploration — covering the core machine learning paradigms applied directly to autonomous vehicle perception data.

The course progresses in a deliberate two-phase structure: Phase 1 (Labs 1 and 2) establishes the supervised learning foundation by training models that require labelled ground truth — binary image classification and pixel-wise segmentation of road photographs. Phase 2 (Labs 3 and 4) makes a fundamental paradigm shift to unsupervised learning, where no labels are used at all — the model learns entirely from the structure of the images themselves via a convolutional autoencoder, then uses its learned representations for clustering, interpolation, semantic vector arithmetic, and zero-shot denoising.

All architectures are implemented from first principles in PyTorch. Every component — the SimpleCnn classifier, the SegNet encoder-decoder, the SimpleAutoencoder, the WeightedRandomSampler balancing pipeline, the IoU metric, and the latent interpolation experiments — is coded explicitly without relying on any pre-built segmentation or clustering framework, ensuring deep architectural understanding at every level of the machine learning stack.

🎯 What You Will Find Here

📁 Lab	🏷️ Topic	🧠 Core Concept	🛠️ Key Tools
Lab 1	Binary Image Classification	Custom CNN · WeightedSampler · Transfer Learning	PyTorch · torchvision · scikit-learn
Lab 2	Pixel-Wise Segmentation	SegNet · MaxPool Indices · IoU Metric	PyTorch · torchsummary
Lab 3	Unsupervised Reconstruction	Convolutional Autoencoder · t-SNE · KMeans	PyTorch · scikit-learn
Lab 4	Latent Space Exploration	Interpolation · Vector Arithmetic · Denoising	PyTorch · scikit-learn

---

🗂️ Repository Structure

📦 Machine-Learning-in-Robotics/
│
├── 📁 Lab_1/
│   ├── 📄 Readme.md                                          # Lab 1 full documentation
│   ├── 📁 src/
│   │   └── 📓 Road_Sign_Classification.ipynb                 # Custom CNN + Augmentation + ResNet50
│   └── 📁 results/
│       ├── 🖼️  Road_Sign.png                                 # Dataset sample visualisation
│       ├── 🖼️  Loss_Curve_CustomCNN.png                      # Training loss — Model 1
│       ├── 🖼️  Loss_Curve_AugmentedCNN.png                   # Training loss — Model 2
│       └── 🖼️  Loss_Curve_ResNet50.png                       # Training loss — Model 3
│
├── 📁 Lab_2/
│   ├── 📄 Readme.md                                          # Lab 2 full documentation
│   ├── 📁 src/
│   │   └── 📓 Road_Sign_Segmentation.ipynb                   # SegNet baseline + augmented
│   └── 📁 results/
│       ├── 🖼️  Road_Sign_Dataset.png                         # Dataset + mask visualisation
│       ├── 🖼️  Loss_IoU_Curve_Baseline.png                   # Loss + IoU vs epoch — Model 1
│       ├── 🖼️  Loss_IoU_Curve_Augmented.png                  # Loss + IoU vs epoch — Model 2
│       ├── 🖼️  Prediction_Samples_Baseline.png               # Test predictions — Model 1
│       └── 🖼️  Prediction_Samples_Augmented.png              # Test predictions — Model 2
│
├── 📁 Lab_3/
│   ├── 📄 Readme.md                                          # Lab 3 full documentation
│   ├── 📁 src/
│   │   └── 📓 Road_Scene_Autoencoder.ipynb                   # Autoencoder + clustering pipeline
│   └── 📁 results/
│       ├── 🖼️  Road_Sign_Data.png                            # Dataset sample visualisation
│       ├── 🖼️  Epoch_10_from_100.png                         # Reconstruction snapshot — epoch 10
│       ├── 🖼️  Epoch_50_from_100.png                         # Reconstruction snapshot — epoch 50
│       ├── 🖼️  Epoch_100_from_100.png                        # Reconstruction snapshot — epoch 100
│       ├── 🖼️  Loss_Curve_Phase1.png                         # MSE loss — Phase 1 (100 epochs)
│       ├── 🖼️  Loss_Curve_Phase2.png                         # MSE loss — Phase 2 (100 epochs)
│       ├── 🖼️  tSNE_Clusters.png                             # t-SNE scatter plot, KMeans labels
│       └── 🖼️  Cluster_Image_Grid.png                        # 6 clusters × 5 images grid
│
├── 📁 Lab_4/
│   ├── 📄 Readme.md                                          # Lab 4 full documentation
│   ├── 📁 src/
│   │   └── 📓 Autoencoder_Comparison_and_Latent_Space.ipynb  # model64 vs model32 — all experiments
│   └── 📁 results/
│       ├── 🖼️  Loss_Model64.png                              # MSE loss vs epoch — model64
│       ├── 🖼️  Loss_Model32.png                              # MSE loss vs epoch — model32
│       ├── 🖼️  tSNE_Clusters_Model64.png                     # t-SNE scatter — model64
│       ├── 🖼️  tSNE_Clusters_Model32.png                     # t-SNE scatter — model32
│       ├── 🖼️  Cluster_Grid_Model64.png                      # 6×5 cluster image grid — model64
│       ├── 🖼️  Cluster_Grid_Model32.png                      # 6×5 cluster image grid — model32
│       ├── 🖼️  Interpolation_Model64.png                     # 1×7 interpolation figure — model64
│       ├── 🖼️  Interpolation_Model32.png                     # 1×7 interpolation figure — model32
│       ├── 🖼️  VectorArithmetic_Model64.png                  # 3×4 arithmetic grid — model64
│       ├── 🖼️  VectorArithmetic_Model32.png                  # 3×4 arithmetic grid — model32
│       ├── 🖼️  Denoising_Model64.png                         # 5×3 denoising grid — model64
│       └── 🖼️  Denoising_Model32.png                         # 5×3 denoising grid — model32
│
└── 📄 README.md                                              # ← You are here

---

🔬 Course Overview

The Machine Learning in Robotics course develops both the theoretical foundations and the practical engineering skills required to apply modern machine learning techniques to robotic perception tasks. The curriculum spans supervised classification and segmentation, unsupervised representation learning, and advanced latent space analysis — all applied to a real-world road-scene dataset that connects directly to autonomous vehicle and mobile robotics applications.

The four labs advance in a deliberate two-phase sequence:

Phase 1 — Supervised Learning (Labs 1 and 2): The course begins with labelled learning. Lab 1 establishes the supervised classification pipeline — deriving labels from raw segmentation masks, handling class imbalance with WeightedRandomSampler, training a custom CNN from scratch, and comparing it against ResNet50 transfer learning. Lab 2 scales the output space from a single scalar per image to a full spatial mask, introducing pixel-level segmentation with the SegNet architecture and the IoU metric. Together these two labs build the complete supervised perception pipeline from data preparation through evaluation.

Phase 2 — Unsupervised Learning (Labs 3 and 4): With the supervised foundation in place, the course makes a fundamental paradigm shift. Lab 3 trains a convolutional autoencoder without a single label — the model learns by reconstructing its own inputs, producing a compact 16,384-dimensional latent representation of each road scene that is then used for t-SNE projection and KMeans clustering. Lab 4 probes the geometry of this learned latent space through four systematic experiments — cluster comparison, latent interpolation, vector arithmetic, and zero-shot denoising — comparing two autoencoder variants of different capacity across all experiments.

  Supervised              Supervised           Unsupervised         Unsupervised
  Classification          Segmentation         Reconstruction       Latent Exploration
  ──────────────          ────────────         ─────────────        ──────────────────
   ┌─────────┐            ┌─────────┐          ┌─────────┐          ┌─────────┐
   │  LAB 1  │──────────► │  LAB 2  │────────► │  LAB 3  │────────► │  LAB 4  │
   └─────────┘            └─────────┘          └─────────┘          └─────────┘
   Custom CNN             SegNet               SimpleAutoencoder    model64 vs model32
   WeightedSampler        MaxPool Indices      Latent Bottleneck    Interpolation
   Balanced Accuracy      IoU Metric           t-SNE + KMeans       Vector Arithmetic
   ResNet50 Transfer      pos_weight=2.0       MSE Reconstruction   Zero-Shot Denoising

---

🧪 Lab Summaries

---

🔵 Lab 1 — Road Sign Classification with Custom CNN and Transfer Learning

📌 Task Description

Given a dataset of 256×256 RGB road photographs paired with segmentation masks, derive binary classification labels from the mask pixel sums, handle the resulting class imbalance, and train three progressively sophisticated models to classify whether a road sign is present or absent — evaluating all using balanced accuracy.

This lab establishes the complete supervised image classification pipeline, answering the foundational question: "Given a raw unlabelled dataset and its associated segmentation masks, how do you construct a reliable binary classifier that is robust to class imbalance and generalises well to unseen data?"

What the task requires:

• Derive binary labels from segmentation masks by computing per-image pixel sums and applying a threshold of 1205 — converting a segmentation dataset into a classification one without manual labelling.
• Diagnose the class imbalance between road-sign-present and road-sign-absent images, and apply WeightedRandomSampler with inverse-frequency weights to draw balanced mini-batches during training.
• Design and train a SimpleCnn classifier from scratch: four convolutional blocks (3→8→16→32→64 channels, Conv2d + ReLU + MaxPool2d + BatchNorm2d) followed by a three-layer fully connected head (16384→480→384→1) with Dropout regularization.
• Train a second SimpleCnn variant using an OurOwnDataset class that applies random rotation (±10°) and horizontal flipping on-the-fly — isolating the effect of data augmentation on overfitting reduction.
• Load a ResNet50 backbone with ImageNet weights, freeze all parameters, replace the final fc layer with Linear(2048→1), and fine-tune only the new head for 5 epochs — demonstrating transfer learning efficiency.
• Use balanced_accuracy_score from scikit-learn as the primary metric, ensuring that errors on the minority class are penalized equally to errors on the majority class.

🔑 Key Concepts

Concept	Description
🏷️ Label Derivation	np.sum(Y, axes=(1,2,3)) > 1205 — pixel-sum threshold converts mask dataset to classification labels
⚖️ WeightedRandomSampler	Per-sample inverse-frequency weights; draws balanced batches without discarding any data
🧱 SimpleCnn	4 conv blocks → flatten 16,384 → FC(480) → FC(384) → FC(1) — all implemented from scratch
📐 BCEWithLogitsLoss	Fused sigmoid + BCE — numerically stable binary cross-entropy for logit outputs
🔄 On-the-Fly Augmentation	OurOwnDataset.__getitem__ applies fresh RandomRotation(±10°) + RandomHorizontalFlip per access
🏛️ ResNet50 Transfer	Frozen backbone (23M params) + new Linear(2048→1) head — only 2,049 parameters trained
📊 Balanced Accuracy	= 0.5 × (recall_class_0 + recall_class_1) — corrects for imbalance in evaluation
🎯 ImageNet Normalization	mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225) — applied in-place per image

📤 Key Classification Results

Dataset:  x_train.npy (256×256×3 RGB) + y_train.npy (masks → pixel-sum threshold labels)
          Labels: sum > 1205 → sign present (1) | sum ≤ 1205 → absent (0)
          Split: 2700 train / 200 val / ~100+ test  |  SEED=60  |  Batch=128

                            Balanced Accuracy Scores
┌──────────────────────────────┬──────────┬───────────┬───────────┬──────────────────┐
│ Model                        │  Train   │    Val    │   Test    │ Pass Requirement │
├──────────────────────────────┼──────────┼───────────┼───────────┼──────────────────┤
│ Custom CNN (no augmentation) │  99.60%  │  97.20%   │  92.30%   │ ≥ 95.0% test ✅  │
│ Augmented CNN                │  97.16%  │  97.45%   │    —      │ ≥ 96.0% val  ✅  │
│ ResNet50 Transfer Learning   │  99.41%  │  95.01%   │    —      │ ≥ 95.0% val  ✅  │
└──────────────────────────────┴──────────┴───────────┴───────────┴──────────────────┘
All three pass requirements satisfied  |  Augmented model: train-val gap reduced to 0.29%

📂 → View Lab 1 Full Documentation

---

🟠 Lab 2 — Road Sign Segmentation with SegNet and Data Augmentation

📌 Task Description

Using the same road photograph dataset as Lab 1, subsample 1000 images, binarize their multi-class segmentation masks, and train a SegNet encoder-decoder to predict a pixel-wise binary foreground mask for each image — with the final augmented model achieving IoU ≥ 0.65 on the test set.

This lab makes the critical architectural leap from image-level classification to pixel-level segmentation — the model must now predict a label for every one of the 65,536 pixels in a 256×256 image, not just a single scalar. The defining architectural innovation is SegNet's pooling index propagation, which routes the exact location of each MaxPool operation back to the corresponding unpooling step in the decoder.

What the task requires:

• Subsample the full dataset at stride 3 to obtain 1000 images, and binarize the multi-class masks: collapse road signs (class 1) and crosswalk markings (class 2) into a single foreground class 1, leaving background as class 0.
• Build SegNet entirely from scratch: five encoder blocks (3→64→128→256→512→512 channels), each applying Conv2d + BatchNorm + ReLU followed by MaxPool2d(return_indices=True); a symmetric five-block decoder using MaxUnpool2d to restore spatial resolution using saved pool indices.
• Apply BCEWithLogitsLoss(pos_weight=2.0) to compensate for the severe pixel-level imbalance between background (majority) and foreground (minority) pixels.
• Implement the IoU metric in PyTorch: IoU = (|target ∩ pred| + ε) / (|target ∪ pred| + ε) using bitwise AND/OR operations, with SMOOTH=1e-8 to prevent zero division on all-background batches.
• Train a unified loop across {'train': ..., 'valid': ...} stages for 65 epochs, visualising six validation image/mask/prediction triplets after every epoch, and computing final IoU on all three splits after training.
• Extend to a second model trained with an OurOwnDataset that concatenates image and mask into a 4-channel tensor for joint geometric transforms — ensuring the augmentation is applied identically to both image and its corresponding mask.

🔑 Key Concepts

Concept	Description
🖼️ Pixel-Wise Output	Model outputs (B, 1, 256, 256) logit mask — one prediction per pixel per image
🔗 MaxPool Index Propagation	MaxPool2d(return_indices=True) saves argmax → MaxUnpool2d restores exact spatial locations
📦 5-Block Encoder	3→64→128→256→512→512 channels; 5× MaxPool2d(2,2) → 8×8 spatial bottleneck
🔁 Symmetric Decoder	MaxUnpool2d mirrors each encoder block; channels reversed 512→512→256→128→64→1
⚖️ pos_weight=2.0	Upweights foreground BCE loss 2× to counter pixel-level class imbalance
📐 IoU Metric	intersection / union per image — penalises both false positives and false negatives equally
🎨 Joint Augmentation	4-channel tensor (R,G,B,mask) allows single transform to affect image and mask identically
📊 score_model()	Evaluates mean IoU over full DataLoader under torch.no_grad() — one scalar per split

📤 Key Segmentation Results

Dataset:  x_train.npy[0:3000:3] → 1000 images  |  masks binarized {0,1,2} → {0,1}
          Split: 700 train / 150 val / 150 test  |  SEED=44  |  Batch=32
          SegNet: 5-block Enc-Dec, MaxPool index propagation, Adam lr=0.0001

                          Mean IoU Scores (pixel-level)
┌─────────────────────────┬───────────┬──────────┬───────────┬──────────────────┐
│ Model                   │ Train IoU │  Val IoU │ Test IoU  │ Pass Requirement │
├─────────────────────────┼───────────┼──────────┼───────────┼──────────────────┤
│ Baseline SegNet         │   0.939   │   0.613  │   0.619   │ ≥ 0.50 test  ✅  │
│ Augmented SegNet        │   0.929   │   0.656  │   0.669   │ ≥ 0.65 test  ✅  │
└─────────────────────────┴───────────┴──────────┴───────────┴──────────────────┘
Both pass requirements satisfied  |  Augmentation improves test IoU by +0.050 (8.1% gain)

📂 → View Lab 2 Full Documentation

---

🟣 Lab 3 — Road Scene Reconstruction and Clustering with Convolutional Autoencoder

📌 Task Description

Using all 3,000 road photographs downscaled to 128×128, train a convolutional autoencoder entirely without labels to reconstruct its own inputs through a 16,384-dimensional bottleneck — then use the learned latent representations to discover six natural scene clusters via t-SNE projection and KMeans clustering.

This lab marks the fundamental paradigm shift from supervised to unsupervised learning. No ground-truth labels, masks, or thresholds are used anywhere. The TensorDataset(X_t, X_t) self-supervised pairing makes each image both the input and the reconstruction target; the model learns by minimising its own pixel-wise reconstruction error. The bottleneck forces it to encode only the most essential visual structure, producing a compact latent space rich enough to reflect semantic groupings.

What the task requires:

• Process all 3000 images (X[0:3000:1]), downscaling from 256×256 to 128×128 via TVF.resize() and applying ImageNet normalization — creating a TensorDataset(X_t, X_t) where input equals target.
• Design SimpleAutoencoder(latent_channels=64) from scratch: a three-layer strided-convolution encoder reducing spatial resolution from 128×128 to 16×16 — producing a (B, 64, 16, 16) = 16,384-element bottleneck (3.0× compression) — followed by a symmetric three-layer transposed-convolution decoder with no final activation.
• Train in two phases using MSELoss and Adam(lr=0.001): Phase 1 for 100 epochs targeting val loss ≤ 0.05; Phase 2 continuing the same model for a further 100 epochs targeting val loss ≤ 0.01. Save a reconstruction snapshot every 2 epochs.
• Extract flattened latent vectors (2100, 16384) from the encoder for all validation images, apply TSNE(n_components=2, perplexity=30) to reduce to 2D, and fit KMeans(n_clusters=6) to assign cluster labels.
• Visually interpret each of the six clusters by sampling five representative images per cluster — connecting the unsupervised groupings to semantic scene categories based on lighting conditions, road environment, and time of day.

🔑 Key Concepts

Concept	Description
🔄 Self-Supervised Pairing	TensorDataset(X_t, X_t) — input equals target; no external labels needed
🏗️ Strided Conv Encoder	Conv2d(stride=2) × 3 reduces 128→64→32→16; learnable downsampling
📦 Latent Bottleneck	(B, 64, 16, 16) = 16,384 elements; 3.0× compression of 49,152-element input
🔁 ConvTranspose2d Decoder	ConvTranspose2d(output_padding=1) × 3 restores exact 128×128 resolution
📉 MSELoss	Pixel-wise L² regression — appropriate for continuous normalized pixel targets
🚫 No Final Activation	Decoder outputs raw logits — ImageNet normalization range extends beyond [0,1]
🗺️ t-SNE Projection	TSNE(n_components=2, perplexity=30) maps 16,384-dim → 2D for visualisation
🔵 KMeans Clustering	Fitted on 2D t-SNE features; n_clusters=6 reveals scene-type groupings

📤 Key Reconstruction and Clustering Results

Dataset:  x_train.npy  X[0:3000:1] → 3000 images, downscaled 256→128 via TVF.resize()
          Split: 900 train / 2100 val (no test set)  |  SEED=44  |  Batch=256
          SimpleAutoencoder(latent_channels=64)  |  Bottleneck: 16,384 elements  |  3.0× compression

                     MSE Reconstruction Loss
┌──────────────────────────┬────────┬──────────────┬──────────────┬────────────────┐
│ Training Phase           │ Epochs │  Start Loss  │  Final Loss  │ Pass Target    │
├──────────────────────────┼────────┼──────────────┼──────────────┼────────────────┤
│ Phase 1 — initial train  │  100   │    ~1.000    │    ~0.044    │ ≤ 0.05    ✅   │
│ Phase 2 — continued      │  100   │    ~0.044    │    ≤ 0.010   │ ≤ 0.01    ✅   │
└──────────────────────────┴────────┴──────────────┴──────────────┴────────────────┘

6 Semantic Clusters Discovered (KMeans on 2D t-SNE of validation latent features):
  Cluster 0 — Night / High-Illumination scenes
  Cluster 1 — Overcast Daytime roads
  Cluster 2 — Bright Urban Crosswalks
  Cluster 3 — Dusk / Evening Urban environments
  Cluster 4 — Mixed-Light Urban scenes
  Cluster 5 — Bright Rural / Suburban roads

📂 → View Lab 3 Full Documentation

---

🟢 Lab 4 — Autoencoder Comparison: Latent Space Interpolation, Vector Arithmetic and Denoising

📌 Task Description

Introduce a second SimpleAutoencoder variant (model32, latent_channels=32) alongside the Lab 3 model (model64, latent_channels=64) and evaluate both across four systematic experiments that probe the geometry, capacity, and semantic structure of their respective latent spaces: clustering comparison, latent interpolation, vector arithmetic, and zero-shot denoising.

This lab transforms the autoencoder from a reconstruction tool into a subject of systematic study. Rather than simply training a model, the student runs controlled experiments that reveal how the learned latent space is structured — whether it is smooth enough to interpolate between images, whether it encodes semantic directions that can be transferred between scenes, and whether the bottleneck compression incidentally filters Gaussian noise. The dual-model design isolates the precise effect of halving the latent capacity from 16,384 to 8,192 elements.

What the task requires:

• Train model64 (latent_channels=64) to val MSE ≤ 0.01 and model32 (latent_channels=32) to val MSE ≤ 0.05 — both using Adam(lr=0.003) for 100 epochs each on the identical Lab 3 dataset and split.
• Experiment 1 — Clustering comparison: Extract and flatten latent features from both encoders across the 2,100-image validation set; apply independent TSNE(n_components=2, perplexity=30) projections and KMeans(n_clusters=6) to each; compare t-SNE scatter plots and 6×5 cluster image grids side-by-side.
• Experiment 2 — Latent interpolation (image fusion): Select two validation images from different clusters; compute z(α) = (1−α)·z₁ + α·z₂ for α ∈ {0, 0.25, 0.5, 0.75, 1.0}; decode each intermediate vector to a 1×7 visualisation; compare transition smoothness between models.
• Experiment 3 — Vector arithmetic: For each model, select images A, B, C from three different clusters and compute D = C + (B − A) in latent space; decode D and interpret whether the semantic transformation from A to B has been transferred to C — repeated for three cyclic cluster permutations per model.
• Experiment 4 — Zero-shot denoising: Add Gaussian noise (noise_level=0.7) to clean validation images, clamp to [-1.0, 1.0], and pass through each autoencoder without any denoising-specific training; compute F.mse_loss(denoised, clean) for both models and visualise the clean / noisy / denoised triplets.

🔑 Key Concepts

Concept	Description
📦 Latent Capacity	model64: 16,384 elements, 3.0× compression · model32: 8,192 elements, 6.0× compression
🗺️ Dual Clustering	Independent t-SNE + KMeans applied per model — reveals how capacity affects feature geometry
🔀 Latent Interpolation	z(α) = (1−α)·z₁ + α·z₂ — smooth manifold → coherent transitions; rough manifold → blurring
➕ Vector Arithmetic	D = C + (B − A) — tests whether semantic directions are linearly separable in latent space
🔇 Zero-Shot Denoising	Bottleneck discards noise dimensions not present in training — no retraining needed
📐 Reshape for Decoder	z.view(1, C, 16, 16) required before decoding flattened interpolated/arithmetic vectors
📊 MSE Denoising Metric	F.mse_loss(denoised, clean) — quantifies noise suppression relative to corrupted baseline
⚖️ Capacity vs. Quality	Higher latent dim → richer geometry, smoother interpolation, better arithmetic, lower denoising MSE

📤 Key Experiment Results

Two-model comparison — identical architecture, same dataset, different bottleneck depth
  model64: latent (B, 64, 16, 16) = 16,384 elements  |  compression: 3.0×  |  target ≤ 0.01 MSE ✅
  model32: latent (B, 32, 16, 16) =  8,192 elements  |  compression: 6.0×  |  target ≤ 0.05 MSE ✅

                         Four-Experiment Comparison Summary
┌────────────────────────────────┬────────────────────────┬────────────────────────┐
│ Experiment                     │  model64 (16,384-dim)  │  model32 (8,192-dim)   │
├────────────────────────────────┼────────────────────────┼────────────────────────┤
│ Clustering (t-SNE + KMeans)    │ Tight, well-separated  │ More overlap between   │
│                                │ cluster boundaries.    │ adjacent clusters.     │
│ Interpolation quality          │ Smooth, semantically   │ Blurring at midpoints; │
│                                │ coherent transitions.  │ abrupt element fading. │
│ Vector arithmetic D=C+(B−A)    │ Global changes (light) │ Degraded fidelity;     │
│                                │ transfer cleanly ✅    │ structural noise ❌   │
│ Denoising MSE (noise=0.7)      │       0.224126   ✅    │      0.231699    ❌   │
└────────────────────────────────┴────────────────────────┴────────────────────────┘
model64 outperforms model32 across all four experiments — performance gap widens with task complexity
Both models reduce MSE by ~53–54% from the noisy baseline — zero-shot denoising is robust at both capacities

📂 → View Lab 4 Full Documentation

---

🔗 Progressive Learning Pathway

The four labs form an interlocking progression where each lab builds directly on the foundations of the previous one. The dependency is not merely thematic — specific architectures, dataset pipelines, training functions, and evaluation methods introduced in each lab are carried forward and extended in the next, building cumulative engineering depth.

╔═════════════════════════════════════════════════════════════════════════════════════════╗
║                     MACHINE LEARNING IN ROBOTICS — PROGRESSION                          ║
╠══════════════════╦═══════════════════╦═══════════════════╦══════════════════════════════╣
║   🔵 LAB 1 🔵   ║   🟠 LAB 2 🟠    ║   🟣 LAB 3 🟣    ║        🟢 LAB 4 🟢          ║
║  Road Sign       ║  Road Sign        ║  Road Scene       ║  Autoencoder                 ║
║  Classification  ║  Segmentation     ║  Autoencoder      ║  Comparison                  ║
╠══════════════════╬═══════════════════╬═══════════════════╬══════════════════════════════╣
║  Supervised      ║  Supervised       ║  Unsupervised     ║  Unsupervised                ║
║  Classification  ║  Pixel-Wise Seg.  ║  Reconstruction   ║  Latent Exploration          ║
║  Balanced Acc.   ║  IoU Metric       ║  MSELoss          ║  4 Experiments               ║
╠══════════════════╬═══════════════════╬═══════════════════╬══════════════════════════════╣
║  SimpleCnn       ║  SegNet           ║  SimpleAutoencoder║  model64 vs model32          ║
║  4 conv blocks   ║  5-block Enc-Dec  ║  3 strided Conv   ║  16,384 vs 8,192 latent      ║
║  + ResNet50 TL   ║  MaxPool indices  ║  + 3 ConvTranspose║  dual model comparison       ║
╠══════════════════╬═══════════════════╬═══════════════════╬══════════════════════════════╣
║  Adam lr=0.0003  ║  Adam lr=0.0001   ║  Adam lr=0.001    ║  Adam lr=0.003               ║
║  20–25 epochs    ║  65 epochs        ║  200 epochs total ║  100 epochs each             ║
║  BCEWithLogits   ║  BCEWithLogits    ║  MSELoss          ║  MSELoss                     ║
║  WeightedSampler ║  pos_weight=2.0   ║  TensorDataset    ║  TensorDataset               ║
╠══════════════════╬═══════════════════╬═══════════════════╬══════════════════════════════╣
║  Establishes     ║  Scales output:   ║  Paradigm shift:  ║  Extends Lab 3:              ║
║  supervised      ║  image-level →    ║  supervised →     ║  adds model32, runs          ║
║  classification  ║  pixel-level mask ║  unsupervised     ║  clustering, interpolation   ║
║  baseline        ║  + IoU metric     ║  + latent space   ║  arithmetic, denoising       ║
╚══════════════════╩═══════════════════╩═══════════════════╩══════════════════════════════╝

Dependency chain:

• 🔵 Lab 1 establishes the full supervised pipeline — dataset loading, ImageNet normalization, DataLoader creation, the train() / predict() loop pattern, and balanced accuracy evaluation. These infrastructure components are reused verbatim in Lab 2.
• 🟠 Lab 2 inherits the Lab 1 dataset and normalization pipeline, scales the output from a scalar to a full spatial mask, and introduces the shared loaders = {'train': ..., 'valid': ...} training loop pattern with per-stage model.train() / model.eval() switching — a pattern carried forward into Labs 3 and 4.
• 🟣 Lab 3 inherits the training loop pattern from Lab 2, reuses the same segmentation_new.zip dataset with stride-1 sampling, and introduces the SimpleAutoencoder architecture and unsupervised TensorDataset(X_t, X_t) self-supervised pairing — both directly reused in Lab 4.
• 🟢 Lab 4 inherits SimpleAutoencoder, the Lab 3 dataset pipeline, and the training loop entirely, adding only the second capacity variant and four new experiment functions that probe the geometry of the learned latent space.

---

⚙️ Common Platform — Shared Dataset and Training Infrastructure

📂 Shared Dataset — segmentation_new.zip

All four labs operate on a single shared source dataset of road photographs and segmentation masks, downloaded once and used across the entire course. The dataset is what connects the machine learning techniques to a genuine robotic perception application — autonomous vehicles must classify, segment, and understand road scenes in real time.

Source:  http://mlr.vedyakov.com/segmentation_new.zip
Files:   x_train.npy  — shape (N, 256, 256, 3), uint8, RGB road photographs
         y_train.npy  — shape (N, 256, 256, 1), uint8, pixel-class segmentation masks

Each lab draws from this single source but uses it differently, with progressively richer exploitation of the data's structure:

📁 Lab	Images Used	Resolution	Labels	Split	Batch	SEED
Lab 1	~3000 (stride=1)	256 × 256	Binary class (mask pixel-sum > 1205)	2700 / 200 / ~100	128	60
Lab 2	1000 (stride=3)	256 × 256	Binary pixel mask ({0,1,2}→{0,1})	700 / 150 / 150	32	44
Lab 3	3000 (stride=1)	128 × 128 ↓	None (self-supervised)	900 / 2100 / —	256	44
Lab 4	3000 (stride=1)	128 × 128 ↓	None (self-supervised)	900 / 2100 / —	256	44

ImageNet normalization is applied identically across all four labs — the only consistent preprocessing step shared from Lab 1 through Lab 4:

Channel	Mean	Standard Deviation
Red	0.485	0.229
Green	0.456	0.224
Blue	0.406	0.225

---

🔁 Shared PyTorch Training Infrastructure

Labs 2, 3, and 4 share the same fundamental training loop architecture. The loaders = {'train': ..., 'valid': ...} dictionary pattern, model.train() / model.eval() stage transitions, per-batch GPU-to-CPU memory cleanup, and torch.cuda.empty_cache() calls are reused across all three labs with only the loss function, metric, and visualisation callback changing.

# Shared dataset preparation pattern (all labs)
X    = np.load('./x_train.npy')                              # (N, H, W, 3)
X_t  = torch.FloatTensor(np.transpose(X / 255, (0,3,1,2)))  # CHW float
for x in X_t:                                                # normalize in-place
    TVF.normalize(x, mean=(0.485,0.456,0.406),
                     std=(0.229,0.224,0.225), inplace=True)

# Shared training loop pattern (Labs 2, 3, 4)
for epoch in range(epochs):
    for stage, dataloader in loaders.items():
        model.train() if stage == 'train' else model.eval()

        for X_batch, Y_batch in dataloader:
            X_batch = X_batch.to(device)
            Y_batch = Y_batch.to(device)

            if stage == 'train':
                optimizer.zero_grad()
                output = model(X_batch)
                loss   = loss_fn(output, Y_batch)      # BCEWithLogitsLoss or MSELoss
                loss.backward()
                optimizer.step()
            else:
                with torch.no_grad():
                    output = model(X_batch)
                    loss   = loss_fn(output, Y_batch)

            # Explicit GPU memory cleanup — required for large batch sizes
            X_batch.cpu(); Y_batch.cpu()
            torch.cuda.empty_cache()

PyTorch API elements introduced progressively across the four labs:

API Element	Introduced	Purpose
BCEWithLogitsLoss, WeightedRandomSampler	Lab 1	Stable binary loss + class-imbalance correction
balanced_accuracy_score, predict()	Lab 1	Imbalance-aware evaluation metric
BCEWithLogitsLoss(pos_weight=2.0)	Lab 2	Pixel-level foreground weight amplification
MaxPool2d(return_indices=True), MaxUnpool2d	Lab 2	SegNet pooling index propagation
iou_pytorch(), score_model()	Lab 2	Per-image and full-split IoU evaluation
TVF.resize(), MSELoss, TensorDataset(X,X)	Lab 3	Downscaling, reconstruction loss, self-supervised pair
TSNE, KMeans, plot_example()	Lab 3	Latent space projection, clustering, snapshot saves
Latent interpolation, vector arithmetic	Lab 4	Explicit latent manifold geometry manipulation
F.mse_loss denoising evaluation	Lab 4	Zero-shot denoising quantification

---

🚀 Quick Start

1️⃣ Clone the Repository

git clone https://github.com/umerahmedbaig7/Machine-Learning-in-Robotics.git
cd Machine-Learning-in-Robotics

2️⃣ Install Dependencies

All labs run on Google Colab where packages are pre-installed. For a local environment:

# All labs
pip install torch torchvision numpy scikit-learn matplotlib seaborn tqdm torchsummary

📌 For a clean local virtual environment:

python -m venv mlr_env
source mlr_env/bin/activate        # Windows: mlr_env\Scripts\activate
pip install torch torchvision numpy scikit-learn matplotlib seaborn tqdm torchsummary

3️⃣ Open in Google Colab and Set GPU Runtime

Each lab is a self-contained Colab notebook. Before running any lab:

1. Upload the notebook to Google Drive or open directly from GitHub.
2. In the menu: Runtime → Change runtime type → Hardware accelerator → GPU (T4).
3. Confirm GPU is available — the notebook will print the GPU device name on startup.

⚠️ All labs are designed for T4 GPU execution. CPU execution is possible but training times will increase by 10–30× for Labs 3 and 4.

4️⃣ Download the Dataset and Run Each Lab

The dataset download is embedded in every notebook. Alternatively, mount Google Drive with a pre-uploaded copy:

# Way 1 — Direct download (all labs, runs automatically)
!wget -c http://mlr.vedyakov.com/segmentation_new.zip -O segmentation.zip
!unzip -q ./segmentation.zip

# Way 2 — From Google Drive (if pre-uploaded)
from google.colab import drive
drive.mount('/content/gdrive/')
!unzip -q /content/gdrive/MyDrive/University/MLR/segmentation.zip

Then execute all cells sequentially (Runtime → Run all):

# 🔵 Lab 1 — Road Sign Classification (~15–25 min on T4)
Open:  Lab_1/src/Road_Sign_Classification.ipynb

# 🟠 Lab 2 — Road Sign Segmentation (~25–35 min on T4)
Open:  Lab_2/src/Road_Sign_Segmentation.ipynb

# 🟣 Lab 3 — Road Scene Autoencoder (~60–75 min on T4)
Open:  Lab_3/src/Road_Scene_Autoencoder.ipynb

# 🟢 Lab 4 — Autoencoder Comparison (~75–105 min on T4)
Open:  Lab_4/src/Autoencoder_Comparison_and_Latent_Space.ipynb

📁 Lab	Notebook	Estimated T4 Runtime
Lab 1	Road_Sign_Classification.ipynb	~15–25 min
Lab 2	Road_Sign_Segmentation.ipynb	~25–35 min
Lab 3	Road_Scene_Autoencoder.ipynb	~60–75 min
Lab 4	Autoencoder_Comparison_and_Latent_Space.ipynb	~75–105 min

---

📊 Results at a Glance

🔵 Lab 1 — Classification Performance

Model	Train Acc.	Val Acc.	Test Acc.	Train-Val Gap	Pass ✓
Custom CNN (no augmentation)	99.60%	97.20%	92.30%	2.40%	≥ 95.0% test ✅
Augmented CNN	97.16%	97.45%	—	0.29%	≥ 96.0% val ✅
ResNet50 Transfer Learning	99.41%	95.01%	—	4.40%	≥ 95.0% val ✅

🟠 Lab 2 — Segmentation Performance

Model	Train IoU	Val IoU	Test IoU	IoU Improvement	Pass ✓
Baseline SegNet	0.939	0.613	0.619	—	≥ 0.50 test ✅
Augmented SegNet	0.929	0.656	0.669	+0.050 (+8.1%)	≥ 0.65 test ✅

🟣 Lab 3 — Autoencoder Reconstruction Loss

Phase	Epochs	Starting Val MSE	Final Val MSE	Pass Target	Status
Phase 1	100	~1.000	~0.044	≤ 0.05	✅
Phase 2 (continued)	100	~0.044	≤ 0.010	≤ 0.01	✅

🟢 Lab 4 — Two-Model Comparison Across Four Experiments

Experiment	model64 (16,384 latent)	model32 (8,192 latent)	Winner
Training convergence	Val MSE ≤ 0.01 ✅	Val MSE ≤ 0.05 ✅	Both pass
Clustering quality	Tighter t-SNE separation	More cluster overlap	model64
Interpolation	Smooth, coherent transitions	Blurring at midpoints	model64
Vector arithmetic	Global semantics transfer cleanly	Structural noise	model64
Denoising MSE	0.224126	0.231699	model64

---

🧰 Tech Stack

🛠️ Tool	🔖 Version	🎯 Role in This Course	🧪 Used In
	3.8+	Core language — all notebooks, data pipelines, training loops	All
	≥ 1.12	Neural network definition, autograd, GPU tensors, BCEWithLogitsLoss, MSELoss, DataLoader	All
	≥ 0.13	Pre-trained ResNet50, TVF.resize(), TVF.normalize(), transforms.RandomRotation	All
	≥ 1.0	balanced_accuracy_score (Lab 1), TSNE projection, KMeans clustering (Labs 3–4)	Labs 1, 3, 4
	≥ 1.21	Array loading, CHW transpose, normalization, index splitting, label thresholding	All
	≥ 3.4	Loss curves, IoU curves, reconstruction grids, t-SNE scatter plots, interpolation figures	All
	≥ 0.11	darkgrid plot styling applied globally across all notebooks	All
	≥ 1.5	Model architecture inspection — parameter counts and layer shapes	Labs 2, 3, 4

No pre-built segmentation pipelines. No AutoML frameworks. No external model zoos beyond a single ResNet50 backbone in Lab 1. Every architecture in this course — from the SimpleCnn classifier with its four convolutional blocks, to the SegNet encoder-decoder with full MaxPool index propagation, to the SimpleAutoencoder with its 16,384-dimensional bottleneck — is implemented explicitly in PyTorch from first principles, ensuring depth of understanding at every layer of the machine learning stack.

---

👤 Author

Umer Ahmed Baig Mughal

🎓 MSc Robotics and Artificial Intelligence — ITMO University
🏛️ Faculty of Control Systems and Robotics
🔬 Specialization: Machine Learning · Computer Vision · Human-Robot Interaction · Autonomous Systems · Robotic Motion Control

---

📄 License

This repository is intended for academic and research use. All work was developed as part of the Machine Learning in Robotics course within the MSc Robotics and Artificial Intelligence program at ITMO University. Redistribution, modification, and use in derivative academic work are permitted with appropriate attribution to the original author.

---

Machine Learning in Robotics — MSc Robotics and Artificial Intelligence | ITMO University

⭐ If this repository helped you understand machine learning for robotic perception, consider giving it a star! ⭐