Production-grade genpark-bpe-byte-pair-encoding-tokenizer-skill skill for AI agents
-
Updated
Sep 10, 2026 - Python
#
llm-preprocessing
Here are 8 public repositories matching this topic...
Production-grade genpark-bpe-byte-pair-encoding-tokenizer-skill skill for AI agents
-
Updated
Sep 10, 2026 - Python
Automated data ingestion pipeline for extracting plain text from proprietary formats (PDF, DOCX, ODT, H5). Optimized for preparing context for LLMs and Vector Databases.
-
Updated
Jan 4, 2026 - Python
An end-to-end Data Science pipeline for analyzing mathematical misconceptions, featuring automated ETL, MySQL integration, and feature engineering for LLM training. Dockerized & CI/CD enabled.
mysql python docker educational-data-mining etl-pipeline data-science-projects github-actions llm-preprocessing
-
Updated
Feb 12, 2026 - Python
pdf text extraction utility
javascript pdf text extractor data-extraction text-processing chunking pdf-to-text llm-preprocessing
-
Updated
Dec 7, 2025
MDify is a document-to-Markdown conversion library for extracting structured content from complex PDFs and document images, including tables, charts, and scanned documents.
python markdown pdf ocr computer-vision python3 md document-analysis table-extraction document-ai llm docling llm-preprocessing
-
Updated
Mar 22, 2026 - Python
Preflight checks for document extraction pipelines — validate, render, and screen PDFs before they reach your LLM. Pure-Python wheel, in-memory only.
python pdf python-library pdf-to-image pdf-processing invoice-processing document-ai ocr-preprocessing pypdfium2 llm-preprocessing blank-detection
-
Updated
Apr 13, 2026 - Python
100% client-side privacy-first converter that transforms docs, spreadsheets, code & data into token-optimized .toon files for LLMs. Powered by MarkItDownJS with RAG-ready chunking, semantic optimization & gzip packing — cutting tokens 15-40%. Zero server. Zero telemetry.
document-conversion file-optimization ai-optimization client-side-privacy gpt-tokenizer zero-server token-optimization llm-preprocessing rag-chunking markitdownjs
-
Updated
Jul 3, 2026 - HTML
Add this topic to your repo
To associate your repository with the llm-preprocessing topic, visit your repo's landing page and select "manage topics."