Tokenization at scale - #287

Draft

le1nux wants to merge 27 commits into

mainfrom

tokenization_at_scale

Commits on Jan 2, 2025

feat: added enhanced logging
le1nux
committed
bbee65c
refactor: scaled the index creation and tokenization by supporting multprocessing
le1nux
committed
90f5d5a

Commits on Jan 5, 2025

refactor: finalized processing strategies
le1nux
committed
e876c97
feat: introduced process controller
le1nux
committed
3fe7690
feat: added custom process implementation
le1nux
committed
3d79cc4
feat: added interface for the processing strategies
le1nux
committed
8f41a74
refactor: bug fixing in TokenizationInstantiationModel
le1nux
committed
a05bd9f
refactor: API uses now process controller
le1nux
committed
ac8f74e

Commits on Jan 8, 2025

feat: added multiple queue destinations for strategy
le1nux
committed
95f30d4
feat: temp env var decorator
le1nux
committed
d5d59a7

Commits on Jan 9, 2025

refactor: moved WorkerTypes
le1nux
committed
d242fd3
refactor: refactored tokenization strategies
le1nux
committed
3222469
feat: added ProcessorException and ProcessingStrategyDoneException
le1nux
committed
be14b0d

Commits on Jan 10, 2025

Commits on Jan 12, 2025

feat: enhanced tokenization instantiation with pydantic model
le1nux
committed
4da0959
feat: added PreprocessingNumberConversion
le1nux
committed
004daeb

Commits on Jan 13, 2025

feat: integrated global index creation
le1nux
committed
8be1ad5
chore: minor renaming fix
le1nux
committed
438858f
chore: fixed imports
le1nux
committed
b5273b2
chore: adapted configs after renamings
le1nux
committed
ea694a1
chore: Merge branch 'main' into tokenization_at_scale
le1nux
committed
b956f90
refactor: improved code quality in strategies.py
le1nux
committed
1a515ba
refactor: integrated changes from PR #283 into tokenization strategy
le1nux
committed
716a98c
chore: Merge branch 'tokenization_at_scale' of github.com:Modalities/modalities into tokenization_at_scale
le1nux
committed
3100919