diff --git a/NLPRenaissanceChallenge/Sample_Notebook.ipynb b/NLPRenaissanceChallenge/Sample_Notebook.ipynb index 16233de..e611cbd 100644 --- a/NLPRenaissanceChallenge/Sample_Notebook.ipynb +++ b/NLPRenaissanceChallenge/Sample_Notebook.ipynb @@ -1,5 +1,4 @@ { -<<<<<<< Updated upstream "cells": [ { "cell_type": "markdown", @@ -11954,5213 +11953,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -======= - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# **RenAIssance: OCR for Seventeenth Century Spanish Texts**\n", - "\n", - "## **Overview**\n", - "RenAIssance project aims to utilize optical character recognition (OCR) to digitize text sources not yet addressed by current tools. The competition leverages machine learning techniques to enable OCR on various undigitized materials. Specifically, it focuses on AI applications to recognize text in seventeenth-century Spanish printed sources, a niche largely ignored by existing OCR solutions.\n", - "\n", - "Current OCR tools, like Adobe Acrobat’s OCR, struggle with transliteration from historical texts. These tools can recognize text from modern, clearly printed sources but fail with early print forms and manuscripts.\n", - "\n", - "

\n", - "\n", - "

\n", - "\n", - "## **Objective**\n", - "Build and evaluate a model to recognize text in scanned pages from historical Spanish texts. The aim is to improve transcription accuracy on low-resource, early print data that traditional OCR tools cannot handle.\n", - "\n", - "This notebook demonstrates the use of Convolutional-Recurrent Neural Network (CRNN) architectures to solve the problem of Optical Character Recognition (OCR) for seventeenth-century Spanish texts. The model is trained on the given dataset, with transcriptions available for the first 25 pages. The remaining 6 pages are used for testing the model's performance. The goal is to improve the accuracy of the OCR model to achieve good results.\n", - "\n", - "

\n", - "\n", - "

\n", - "\n", - "## **Dataset**\n", - "We use with two digitized books:\n", - "- Nobleza Virtuosa\n", - "- Noble Perfecto\n", - "\n", - "Each dataset contains:\n", - "- Scanned page images (~31 pages each).\n", - "- Ground truth transcriptions (.docx) for the first 25 pages.\n", - "- Preprocessed bounding boxes aligned to the text for training.\n", - "\n", - "The last 6 pages are reserved for testing (no transcription provided).\n", - "\n", - "Datasets have been preprocessed, split, and merged, allowing the model to be trained either per-book or on a combined dataset.\n", - "- You need to submit the predicted text for the remaining 6 pages as the output of your model for testing purposes.\n", - "\n", - "## **Task**\n", - "Train an OCR model to learn from the transcribed pages and generalize to unseen pages. The final output should be predicted transcriptions for the test pages.\n", - "\n", - "## **Evaluation Metrics**\n", - "See the attached Evaluation Document and Evaluation Pipeline for information on submission layout and guidelines.\n", - "\n", - "## **Deliverables**\n", - "- A trained OCR model\n", - "- Transcription outputs for the 6 test pages (see Evaluation Document)\n", - "- A brief report covering:\n", - " - Model architecture and training approach\n", - " - Preprocessing steps\n", - " - Evaluation results\n", - "- A usage notebook demonstrating the inference pipeline\n", - "- Completion of pre- and post-hackathon surveys" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1. Setup and Imports\n", - "#### Use Python 3.11" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: gdown in c:\\users\\katej\\onedrive\\documents\\github\\deeplearnhackathon\\.venv\\lib\\site-packages (6.1.0)\n", - "Collecting numpy\n", - " Using cached numpy-2.5.1-cp314-cp314-win_amd64.whl.metadata (6.6 kB)\n", - "Collecting opencv-python\n", - " Using cached opencv_python-5.0.0.93-cp37-abi3-win_amd64.whl.metadata (20 kB)\n", - "Note: you may need to restart the kernel to use updated packages.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11; 1.26.0 Requires-Python >=3.9,<3.13; 1.26.1 Requires-Python >=3.9,<3.13\n", - "ERROR: Could not find a version that satisfies the requirement tensorflow (from versions: none)\n", - "\n", - "[notice] A new release of pip is available: 25.3 -> 26.1.2\n", - "[notice] To update, run: c:\\Users\\katej\\OneDrive\\Documents\\GitHub\\DeepLearnHackathon\\.venv\\Scripts\\python.exe -m pip install --upgrade pip\n", - "ERROR: No matching distribution found for tensorflow\n" - ] - } - ], - "source": [ - "%pip install gdown numpy opencv-python tensorflow matplotlib tqdm pillow" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "ename": "ModuleNotFoundError", - "evalue": "No module named 'tensorflow'", - "output_type": "error", - "traceback": [ - "\u001b[31m---------------------------------------------------------------------------\u001b[39m", - "\u001b[31mModuleNotFoundError\u001b[39m Traceback (most recent call last)", - "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[17]\u001b[39m\u001b[32m, line 7\u001b[39m\n\u001b[32m 3\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m numpy \u001b[38;5;28;01mas\u001b[39;00m np\n\u001b[32m 4\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m cv2\n\u001b[32m 5\u001b[39m \u001b[38;5;28;01mfrom\u001b[39;00m pathlib \u001b[38;5;28;01mimport\u001b[39;00m Path\n\u001b[32m 6\u001b[39m \u001b[38;5;28;01mfrom\u001b[39;00m PIL \u001b[38;5;28;01mimport\u001b[39;00m Image\n\u001b[32m----> \u001b[39m\u001b[32m7\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m tensorflow \u001b[38;5;28;01mas\u001b[39;00m tf\n\u001b[32m 8\u001b[39m \u001b[38;5;28;01mfrom\u001b[39;00m tensorflow \u001b[38;5;28;01mimport\u001b[39;00m keras\n\u001b[32m 9\u001b[39m \u001b[38;5;28;01mfrom\u001b[39;00m tensorflow.keras \u001b[38;5;28;01mimport\u001b[39;00m layers, callbacks\n\u001b[32m 10\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m matplotlib.pyplot \u001b[38;5;28;01mas\u001b[39;00m plt\n", - "\u001b[31mModuleNotFoundError\u001b[39m: No module named 'tensorflow'" - ] - } - ], - "source": [ - "\n", - "import os\n", - "import json\n", - "import numpy as np\n", - "import cv2\n", - "from pathlib import Path\n", - "from PIL import Image\n", - "import tensorflow as tf\n", - "from tensorflow import keras\n", - "from tensorflow.keras import layers, callbacks\n", - "import matplotlib.pyplot as plt\n", - "from tqdm import tqdm\n", - "import gdown\n", - "\n", - "print(f\"TensorFlow version: {tf.__version__}\")\n", - "print(f\"GPU available: {tf.config.list_physical_devices('GPU')}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2. Download Preprocessed Data from Google Drive\n", - "#### This data has already been segmented using CRAFT (https://github.com/clovaai/CRAFT-pytorch) and preprocessed. It contains word images from all 25 training pages of each book. Feel free to use this data in your own solution, or segment the pages yourself using the pdfs of books and word docs of transcriptions provided." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Downloading data from Google Drive to data\\ocr_data...\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Retrieving folder contents\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Retrieving folder 1pR0X8FoBNdVtzjyP1_8GbsOO8VeBfqdJ CorrectedData\n", - "Processing file 1voQmAai6dPbtl3ol_fUErrxp1EBMVDEE outputCorrected.zip\n", - "Processing file 1BTjyyqGcAq0i6U84Db6xwXkKc2B4EuCm Padilla - 1 Nobleza virtuosa_testTranscription.docx\n", - "Processing file 1B3CjNTp3udRW-qMqKCD9PzkckBQsGvBR Padilla - 2 Noble perfecto_Extract.pdf\n", - "Processing file 1TPQaUA9BCPs0M_vTkgUdmYVHhtxRhdNj Padilla - 2 Noble perfecto_Transcription.docx\n", - "Processing file 19A-4DEUtZqn0I_nrM5PdG4RC7jG_IYjT Padilla - Nobleza virtuosa_testExtract.pdf\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Retrieving folder contents completed\n", - "Building directory structure\n", - "Building directory structure completed\n", - "Downloading...\n", - "From: https://drive.google.com/uc?id=1voQmAai6dPbtl3ol_fUErrxp1EBMVDEE\n", - "To: c:\\Users\\katej\\OneDrive\\Documents\\GitHub\\DeepLearnHackathon\\NLPRenaissanceChallenge\\data\\ocr_data\\CorrectedData\\outputCorrected.zip\n", - "100%|██████████| 15.9M/15.9M [00:02<00:00, 5.85MB/s]\n", - "Downloading...\n", - "From: https://drive.google.com/uc?id=1BTjyyqGcAq0i6U84Db6xwXkKc2B4EuCm\n", - "To: c:\\Users\\katej\\OneDrive\\Documents\\GitHub\\DeepLearnHackathon\\NLPRenaissanceChallenge\\data\\ocr_data\\Padilla - 1 Nobleza virtuosa_testTranscription.docx\n", - "100%|██████████| 19.3k/19.3k [00:00<00:00, 1.94MB/s]\n", - "Downloading...\n", - "From: https://drive.google.com/uc?id=1B3CjNTp3udRW-qMqKCD9PzkckBQsGvBR\n", - "To: c:\\Users\\katej\\OneDrive\\Documents\\GitHub\\DeepLearnHackathon\\NLPRenaissanceChallenge\\data\\ocr_data\\Padilla - 2 Noble perfecto_Extract.pdf\n", - "100%|██████████| 3.95M/3.95M [00:00<00:00, 4.11MB/s]\n", - "Downloading...\n", - "From: https://drive.google.com/uc?id=1TPQaUA9BCPs0M_vTkgUdmYVHhtxRhdNj\n", - "To: c:\\Users\\katej\\OneDrive\\Documents\\GitHub\\DeepLearnHackathon\\NLPRenaissanceChallenge\\data\\ocr_data\\Padilla - 2 Noble perfecto_Transcription.docx\n", - "100%|██████████| 19.3k/19.3k [00:00Model: \"crnn_ocr\"\n", - "\n" - ], - "text/plain": [ - "\u001b[1mModel: \"crnn_ocr\"\u001b[0m\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
-       "┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃\n",
-       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
-       "│ image (InputLayer)              │ (None, 32, 256, 1)     │             0 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ conv2d_15 (Conv2D)              │ (None, 32, 256, 32)    │           320 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ max_pooling2d_15 (MaxPooling2D) │ (None, 16, 128, 32)    │             0 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ conv2d_16 (Conv2D)              │ (None, 16, 128, 64)    │        18,496 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ max_pooling2d_16 (MaxPooling2D) │ (None, 8, 64, 64)      │             0 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ conv2d_17 (Conv2D)              │ (None, 8, 64, 128)     │        73,856 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ max_pooling2d_17 (MaxPooling2D) │ (None, 8, 32, 128)     │             0 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ reshape_5 (Reshape)             │ (None, 32, 1024)       │             0 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ dense_8 (Dense)                 │ (None, 32, 64)         │        65,600 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ bidirectional_4 (Bidirectional) │ (None, 32, 256)        │       197,632 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ dropout_4 (Dropout)             │ (None, 32, 256)        │             0 │\n",
-       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
-       "│ dense_9 (Dense)                 │ (None, 32, 45)         │        11,565 │\n",
-       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
-       "
\n" - ], - "text/plain": [ - "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n", - "┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n", - "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n", - "│ image (\u001b[38;5;33mInputLayer\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m, \u001b[38;5;34m256\u001b[0m, \u001b[38;5;34m1\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ conv2d_15 (\u001b[38;5;33mConv2D\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m, \u001b[38;5;34m256\u001b[0m, \u001b[38;5;34m32\u001b[0m) │ \u001b[38;5;34m320\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ max_pooling2d_15 (\u001b[38;5;33mMaxPooling2D\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m16\u001b[0m, \u001b[38;5;34m128\u001b[0m, \u001b[38;5;34m32\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ conv2d_16 (\u001b[38;5;33mConv2D\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m16\u001b[0m, \u001b[38;5;34m128\u001b[0m, \u001b[38;5;34m64\u001b[0m) │ \u001b[38;5;34m18,496\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ max_pooling2d_16 (\u001b[38;5;33mMaxPooling2D\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m8\u001b[0m, \u001b[38;5;34m64\u001b[0m, \u001b[38;5;34m64\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ conv2d_17 (\u001b[38;5;33mConv2D\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m8\u001b[0m, \u001b[38;5;34m64\u001b[0m, \u001b[38;5;34m128\u001b[0m) │ \u001b[38;5;34m73,856\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ max_pooling2d_17 (\u001b[38;5;33mMaxPooling2D\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m8\u001b[0m, \u001b[38;5;34m32\u001b[0m, \u001b[38;5;34m128\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ reshape_5 (\u001b[38;5;33mReshape\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m, \u001b[38;5;34m1024\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ dense_8 (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m, \u001b[38;5;34m64\u001b[0m) │ \u001b[38;5;34m65,600\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ bidirectional_4 (\u001b[38;5;33mBidirectional\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m, \u001b[38;5;34m256\u001b[0m) │ \u001b[38;5;34m197,632\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ dropout_4 (\u001b[38;5;33mDropout\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m, \u001b[38;5;34m256\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", - "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", - "│ dense_9 (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m, \u001b[38;5;34m45\u001b[0m) │ \u001b[38;5;34m11,565\u001b[0m │\n", - "└─────────────────────────────────┴────────────────────────┴───────────────┘\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "
 Total params: 367,469 (1.40 MB)\n",
-       "
\n" - ], - "text/plain": [ - "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m367,469\u001b[0m (1.40 MB)\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "
 Trainable params: 367,469 (1.40 MB)\n",
-       "
\n" - ], - "text/plain": [ - "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m367,469\u001b[0m (1.40 MB)\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "
 Non-trainable params: 0 (0.00 B)\n",
-       "
\n" - ], - "text/plain": [ - "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "def build_crnn_model_simple(image_width, image_height, num_chars):\n", - " \"\"\"Build CRNN model without CTC layer — loss computed in training loop.\n", - " \n", - " Args:\n", - " image_width: Width of input images\n", - " image_height: Height of input images\n", - " num_chars: Number of character classes (not including CTC blank)\n", - " \"\"\"\n", - " input_img = layers.Input(shape=(image_height, image_width, 1), name=\"image\")\n", - " \n", - " # Convolutional feature extraction\n", - " x = layers.Conv2D(32, (3, 3), activation=\"relu\", padding=\"same\")(input_img)\n", - " x = layers.MaxPooling2D((2, 2))(x)\n", - " x = layers.Conv2D(64, (3, 3), activation=\"relu\", padding=\"same\")(x)\n", - " x = layers.MaxPooling2D((2, 2))(x)\n", - " x = layers.Conv2D(128, (3, 3), activation=\"relu\", padding=\"same\")(x)\n", - " x = layers.MaxPooling2D((1, 2))(x)\n", - " \n", - " # Reshape for RNN\n", - " # Output shape from conv/pooling: (batch, height, width, channels)\n", - " # Reshape to: (batch, width_as_timesteps, height * channels)\n", - " conv_shape = x.shape\n", - " new_shape = (conv_shape[2], conv_shape[1] * conv_shape[3])\n", - " x = layers.Reshape(target_shape=new_shape)(x)\n", - " x = layers.Dense(64, activation=\"relu\")(x)\n", - " \n", - " # Bidirectional LSTM\n", - " x = layers.Bidirectional(layers.LSTM(128, return_sequences=True))(x)\n", - " x = layers.Dropout(0.25)(x)\n", - " \n", - " # Output layer: num_chars + 1 classes\n", - " # Indices 0 to num_chars-1 are character classes\n", - " # Index num_chars is the CTC blank token\n", - " x = layers.Dense(num_chars + 1, activation=\"softmax\")(x)\n", - " \n", - " model = keras.Model(inputs=input_img, outputs=x, name=\"crnn_ocr\")\n", - " return model\n", - "\n", - "# Build model\n", - "image_width = 256\n", - "image_height = 32\n", - "num_chars = len(char_to_num)\n", - "\n", - "model = build_crnn_model_simple(image_width, image_height, num_chars)\n", - "model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001))\n", - "model.summary()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 6. Train Model (Quick Demo - 5 Epochs)" - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "metadata": {}, - "outputs": [], - "source": [ - "def train_step(x_batch, y_batch, ctc_blank):\n", - " \"\"\"Single training step with CTC loss.\n", - " \n", - " Args:\n", - " x_batch: Input images tensor\n", - " y_batch: Padded label sequences tensor\n", - " ctc_blank: CTC blank token index\n", - " \"\"\"\n", - " with tf.GradientTape() as tape:\n", - " y_pred = model(x_batch, training=True)\n", - " \n", - " # Compute actual label lengths (count non-blank tokens)\n", - " # This is crucial: CTC needs the TRUE length, not the padded length\n", - " batch_len = tf.cast(tf.shape(y_batch)[0], tf.int64)\n", - " input_length = tf.cast(tf.shape(y_pred)[1], tf.int64)\n", - " \n", - " # Count non-blank labels per sample to get actual label length\n", - " label_length = tf.reduce_sum(\n", - " tf.cast(tf.not_equal(y_batch, ctc_blank), tf.int64),\n", - " axis=1,\n", - " keepdims=True\n", - " )\n", - " \n", - " # Expand input_length to match batch dimension\n", - " input_length = input_length * tf.ones(shape=(batch_len, 1), dtype=\"int64\")\n", - " \n", - " # Compute CTC loss\n", - " loss = keras.backend.ctc_batch_cost(\n", - " tf.cast(y_batch, tf.int32), # Ensure int32 for CTC\n", - " y_pred, \n", - " input_length, \n", - " label_length\n", - " )\n", - " loss = tf.reduce_mean(loss)\n", - " \n", - " gradients = tape.gradient(loss, model.trainable_weights)\n", - " model.optimizer.apply_gradients(zip(gradients, model.trainable_weights))\n", - " return loss" - ] - }, - { - "cell_type": "code", - "execution_count": 31, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "Epoch 1/5\n", - " Batch 10/129 - Loss: 56.2888\n", - " Batch 20/129 - Loss: 27.4298\n", - " Batch 30/129 - Loss: 25.8352\n", - " Batch 40/129 - Loss: 26.4110\n", - " Batch 50/129 - Loss: 26.5665\n", - " Batch 60/129 - Loss: 27.6026\n", - " Batch 70/129 - Loss: 23.7627\n", - " Batch 80/129 - Loss: 24.6654\n", - " Batch 90/129 - Loss: 26.2586\n", - " Batch 100/129 - Loss: 27.7444\n", - " Batch 110/129 - Loss: 24.2355\n", - " Batch 120/129 - Loss: 23.9770\n", - "Epoch 1/5 - Average Loss: 28.0436\n", - "\n", - "Epoch 2/5\n", - " Batch 10/129 - Loss: 23.1655\n", - " Batch 20/129 - Loss: 24.9011\n", - " Batch 30/129 - Loss: 23.4498\n", - " Batch 40/129 - Loss: 23.4062\n", - " Batch 50/129 - Loss: 24.8446\n", - " Batch 60/129 - Loss: 25.8159\n", - " Batch 70/129 - Loss: 21.1903\n", - " Batch 80/129 - Loss: 22.5904\n", - " Batch 90/129 - Loss: 24.0818\n", - " Batch 100/129 - Loss: 24.3697\n", - " Batch 110/129 - Loss: 22.5029\n", - " Batch 120/129 - Loss: 21.5721\n", - "Epoch 2/5 - Average Loss: 23.3197\n", - "\n", - "Epoch 3/5\n", - " Batch 10/129 - Loss: 21.3029\n", - " Batch 20/129 - Loss: 23.1018\n", - " Batch 30/129 - Loss: 21.0584\n", - " Batch 40/129 - Loss: 21.2048\n", - " Batch 50/129 - Loss: 23.3718\n", - " Batch 60/129 - Loss: 24.2753\n", - " Batch 70/129 - Loss: 18.8899\n", - " Batch 80/129 - Loss: 20.8114\n", - " Batch 90/129 - Loss: 22.4912\n", - " Batch 100/129 - Loss: 22.2156\n", - " Batch 110/129 - Loss: 21.8696\n", - " Batch 120/129 - Loss: 20.0223\n", - "Epoch 3/5 - Average Loss: 21.5842\n", - "\n", - "Epoch 4/5\n", - " Batch 10/129 - Loss: 20.5082\n", - " Batch 20/129 - Loss: 22.2917\n", - " Batch 30/129 - Loss: 19.7389\n", - " Batch 40/129 - Loss: 20.3331\n", - " Batch 50/129 - Loss: 22.5834\n", - " Batch 60/129 - Loss: 23.7035\n", - " Batch 70/129 - Loss: 17.4008\n", - " Batch 80/129 - Loss: 19.6259\n", - " Batch 90/129 - Loss: 21.9884\n", - " Batch 100/129 - Loss: 21.3705\n", - " Batch 110/129 - Loss: 21.4003\n", - " Batch 120/129 - Loss: 19.2727\n", - "Epoch 4/5 - Average Loss: 20.7397\n", - "\n", - "Epoch 5/5\n", - " Batch 10/129 - Loss: 20.4018\n", - " Batch 20/129 - Loss: 21.7015\n", - " Batch 30/129 - Loss: 18.9435\n", - " Batch 40/129 - Loss: 19.6614\n", - " Batch 50/129 - Loss: 22.0996\n", - " Batch 60/129 - Loss: 23.3163\n", - " Batch 70/129 - Loss: 16.7045\n", - " Batch 80/129 - Loss: 19.0667\n", - " Batch 90/129 - Loss: 21.5118\n", - " Batch 100/129 - Loss: 21.0212\n", - " Batch 110/129 - Loss: 20.7359\n", - " Batch 120/129 - Loss: 19.0305\n", - "Epoch 5/5 - Average Loss: 20.2456\n", - "\n", - "Training complete!\n" - ] - } - ], - "source": [ - "# Cell 3: Custom training loop\n", - "epochs = 5\n", - "batch_size = 32\n", - "ctc_blank = num_chars # CTC blank token index\n", - "history_losses = []\n", - "\n", - "for epoch in range(epochs):\n", - " print(f\"\\nEpoch {epoch + 1}/{epochs}\")\n", - " losses = []\n", - " \n", - " # Iterate through batches manually\n", - " for batch_idx in range(0, len(x_train_input), batch_size):\n", - " x_batch = x_train_input[batch_idx:batch_idx + batch_size]\n", - " y_batch = y_train_padded[batch_idx:batch_idx + batch_size]\n", - " \n", - " # Convert to tensors with correct dtypes\n", - " x_batch = tf.convert_to_tensor(x_batch, dtype=tf.float32)\n", - " y_batch = tf.convert_to_tensor(y_batch, dtype=tf.int32) # CTC requires int32\n", - " \n", - " loss = train_step(x_batch, y_batch, ctc_blank)\n", - " losses.append(float(loss))\n", - " \n", - " # Print progress every 10 batches\n", - " num_batches = (len(x_train_input) + batch_size - 1) // batch_size\n", - " current_batch = (batch_idx // batch_size) + 1\n", - " if current_batch % 10 == 0:\n", - " avg_loss_recent = np.mean(losses[-10:])\n", - " print(f\" Batch {current_batch}/{num_batches} - Loss: {avg_loss_recent:.4f}\")\n", - " \n", - " avg_loss = np.mean(losses)\n", - " history_losses.append(avg_loss)\n", - " print(f\"Epoch {epoch + 1}/{epochs} - Average Loss: {avg_loss:.4f}\")\n", - "\n", - "print(\"\\nTraining complete!\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 7. Create Inference Model and Predict" - ] - }, - { - "cell_type": "code", - "execution_count": 35, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Inference model built successfully.\n", - "Input shape: (None, 32, 256, 1)\n", - "Output shape: (None, None)\n" - ] - } - ], - "source": [ - "def build_inference_model(trained_model, image_width, image_height, num_chars):\n", - " \"\"\"\n", - " Build inference model that takes only images and outputs character predictions.\n", - " Uses greedy CTC decoding to convert network outputs to text.\n", - " \n", - " Args:\n", - " trained_model: The trained CRNN model\n", - " image_width: Width of input images\n", - " image_height: Height of input images\n", - " num_chars: Number of character classes (not including CTC blank)\n", - " \"\"\"\n", - " input_img = layers.Input(shape=(image_height, image_width, 1), name=\"image\")\n", - " \n", - " # Get the logits output from the trained model\n", - " logits = trained_model(input_img) # Shape: (batch, timesteps, num_chars + 1)\n", - " \n", - " # Transpose to time-major format using Lambda layer\n", - " # (batch, timesteps, classes) -> (timesteps, batch, classes)\n", - " logits_time_major = layers.Lambda(\n", - " lambda x: tf.transpose(x, perm=[1, 0, 2]),\n", - " name=\"transpose_to_time_major\"\n", - " )(logits)\n", - " \n", - " # CTC greedy decoder using Lambda layer\n", - " def ctc_decode(args):\n", - " logits_tm, batch_size = args\n", - " sequence_length = tf.fill([batch_size], tf.shape(logits_tm)[0])\n", - " decoded, _ = tf.nn.ctc_greedy_decoder(\n", - " inputs=logits_tm,\n", - " sequence_length=sequence_length\n", - " )\n", - " # Convert SparseTensor to dense\n", - " decoded_dense = tf.sparse.to_dense(decoded[0], default_value=-1)\n", - " return decoded_dense\n", - " \n", - " # Get batch size from logits\n", - " batch_size = layers.Lambda(lambda x: tf.shape(x)[0])(logits)\n", - " \n", - " # Apply CTC decoding\n", - " decoded_dense = layers.Lambda(\n", - " ctc_decode,\n", - " name=\"ctc_decode\"\n", - " )([logits_time_major, batch_size])\n", - " \n", - " inference_model = keras.Model(inputs=input_img, outputs=decoded_dense, name=\"crnn_inference\")\n", - " return inference_model\n", - "\n", - "# Build inference model\n", - "inference_model = build_inference_model(model, image_width, image_height, num_chars)\n", - "print(\"Inference model built successfully.\")\n", - "print(f\"Input shape: {inference_model.input_shape}\")\n", - "print(f\"Output shape: {inference_model.output_shape}\")" - ] - }, - { - "cell_type": "code", - "execution_count": 45, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 79ms/step\n", - "\n", - "Predicted text: su)\n", - "Ground truth: (para\n" - ] - } - ], - "source": [ - "def decode_predictions(pred_indices, num_to_char):\n", - " \"\"\"\n", - " Convert CTC decoder output (indices) to text strings.\n", - " \n", - " Args:\n", - " pred_indices: Dense tensor of shape (batch, max_length) from inference model\n", - " num_to_char: Dictionary mapping indices to characters\n", - " \n", - " Returns:\n", - " List of decoded text strings\n", - " \"\"\"\n", - " decoded_texts = []\n", - " for indices in pred_indices:\n", - " # Remove -1 padding values and convert indices to characters\n", - " valid_indices = indices[indices >= 0]\n", - " text = ''.join([num_to_char.get(int(idx), '') for idx in valid_indices])\n", - " decoded_texts.append(text)\n", - " return decoded_texts\n", - "\n", - "# Test inference on a sample\n", - "sample_image = np.expand_dims(x_train[:1], axis=-1)\n", - "pred_indices = inference_model.predict(sample_image)\n", - "decoded_text = decode_predictions(pred_indices, num_to_char)\n", - "print(f\"\\nPredicted text: {decoded_text[0]}\")\n", - "print(f\"Ground truth: {y_train[0]}\")" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.9" - } - }, - "nbformat": 4, - "nbformat_minor": 4 ->>>>>>> Stashed changes }