diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 82864d7..ea17bdb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: strategy: fail-fast: false matrix: - check: [bluecheck, doc8, docs, flake8, isortcheck, mypy, pylint, rstcheck] + check: [bluecheck, doc8, docs, flake8, isortcheck, mypy, pylint] steps: - uses: actions/checkout@v7 - uses: actions/setup-python@v7 diff --git a/CHANGELOG.md b/CHANGELOG.md index c93dfc9..bceac6b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ diskcache-community forked from DiskCache at upstream commit ebfa37c (3 March 20 ### Docs +- The README is rewritten in Markdown around install, usage and the fork story; the comparison tables move to a new `comparisons` page in the documentation, which gets its own landing page, and the 2016 testimonials are dropped (PR #19). - The documentation is published on Read the Docs at https://diskcache-community.readthedocs.io/, built from `master` on every push; the README links point there instead of at the `.rst` sources (PR #18). ## 5.7.0 (2026-10-11) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9003148..618dbaf 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -13,7 +13,7 @@ uv pip install -e . "django==4.2.*" pytest pytest-cov pytest-django pytest-xdist ## Running tests -`pytest` reads its options from the `[pytest]` section of `tox.ini`: tests run in parallel with `-n auto`, branch coverage of the `diskcache` package must stay at 98 % or above, and the doctests in `README.rst` and `docs/*.rst` run as part of the suite. The Django tests need two environment variables, which `tox` sets for you: +`pytest` reads its options from the `[pytest]` section of `tox.ini`: tests run in parallel with `-n auto`, branch coverage of the `diskcache` package must stay at 98 % or above, and the doctests in `docs/*.rst` run as part of the suite. The Django tests need two environment variables, which `tox` sets for you: ```bash DJANGO_SETTINGS_MODULE=tests.settings PYTHONPATH=. pytest diff --git a/README.md b/README.md new file mode 100644 index 0000000..3e9452d --- /dev/null +++ b/README.md @@ -0,0 +1,182 @@ +# DiskCache + +**Disk and file backed persistent cache for Python, built on SQLite. Pure Python, thread-safe, process-safe, with a Django backend.** + +[![PyPI](https://img.shields.io/pypi/v/diskcache-community.svg)](https://pypi.org/project/diskcache-community/) +[![Python versions](https://img.shields.io/pypi/pyversions/diskcache-community.svg)](https://pypi.org/project/diskcache-community/) +[![CI](https://github.com/secondwindoss/diskcache-community/actions/workflows/ci.yml/badge.svg)](https://github.com/secondwindoss/diskcache-community/actions/workflows/ci.yml) +[![Documentation](https://app.readthedocs.org/projects/diskcache-community/badge/?version=latest)](https://diskcache-community.readthedocs.io/en/latest/) +[![License](https://img.shields.io/pypi/l/diskcache-community.svg)](https://github.com/secondwindoss/diskcache-community/blob/master/LICENSE) + +> [!IMPORTANT] +> This is a community fork of [python-diskcache](https://github.com/grantjenks/python-diskcache), +> whose last release was 5.6.3 in August 2023. It is published on PyPI as +> `diskcache-community` and still imported as `diskcache`. It carries the +> fix path for [CVE-2025-69872](https://github.com/advisories/GHSA-w8v5-vhqr-4h9v) +> and stays compatible with existing 5.6.3 cache directories. +> See [Why this fork](#why-this-fork) and [Migrating from diskcache](#migrating-from-diskcache). + +## Install + +```sh +pip install diskcache-community +``` + +If the upstream `diskcache` package is already installed, remove it first. +Both distributions write to the same `diskcache/` package directory and +cannot coexist; see [Migrating from diskcache](#migrating-from-diskcache). + +## Usage + +```python +from diskcache import Cache + +cache = Cache('cache') # the directory is created with mode 0700 +cache.set('answer', 42, expire=3600, tag='math') +cache['config'] = {'debug': False} +cache.get('answer') # 42 +cache.get('missing', default=None) +cache.evict('math') # delete every key tagged 'math' +``` + +Memoize function results across processes and restarts: + +```python +@cache.memoize(expire=600, ignore={'session'}) +def fetch(url, session): + return session.get(url).text +``` + +Shard across several SQLite files for higher write throughput, and plug the +same cache into Django: + +```python +from diskcache import FanoutCache + +cache = FanoutCache('cache', shards=8, timeout=1) +``` + +```python +# settings.py +CACHES = { + 'default': { + 'BACKEND': 'diskcache.DjangoCache', + 'LOCATION': '/var/tmp/django_cache', + 'SHARDS': 8, + 'DISK': 'diskcache.SafeDisk', # restricted unpickling, see below + } +} +``` + +Caches shared with untrusted writers should use `SafeDisk`, which unpickles +only a fixed set of built-in types and keeps file reads inside the cache +directory: + +```python +from diskcache import Cache, SafeDisk + +cache = Cache('cache', disk=SafeDisk) +``` + +## Features + +- `Cache`, `FanoutCache` and `DjangoCache` for key-value caching with + expiry, tags, size limits and LRU, LFU and other eviction policies +- `Deque` and `Index`: persistent, cross-process `collections.deque` and `dict` +- Recipes for memoization with stampede prevention, cross-process locks, + semaphores and throttling +- Thread-safe and process-safe, with fork-safe connection handling +- Read-only mode, automatic when the directory is not writable +- Opt-in `SafeDisk` for caches whose directory other users can write to +- Pure Python on top of the standard library `sqlite3`; no server, no compiler +- Tested on CPython 3.10 to 3.14 on Linux, macOS and Windows, with + [benchmarks](https://diskcache-community.readthedocs.io/en/latest/cache-benchmarks.html) + against the alternatives + +## Why this fork + +The upstream project stopped releasing after 5.6.3 (August 2023) and its last +commit landed in March 2024. In 2025, CVE-2025-69872 was published against +every released version: the cache reads pickled values with `pickle.load`, so +anyone who can write into the cache directory can run code in the process +that reads from it. The package still works, but scanners flag it and there +is no fixed upstream version. + +This fork publishes the same library as `diskcache-community` and fixes it in +two steps. Version 5.7.0 keeps the `cache.db` schema, the file layout, the +serialization modes and the memoize key format unchanged, so a 5.6.3 process +opens a directory written by this fork and the reverse. It adds the opt-in +`SafeDisk`, creates cache directories with mode `0700`, opens read-only +directories read-only, closes the per-thread SQLite connections it used to +leak, and survives `fork`. Version 6.0.0 will make safe reading the default. +The [changelog](CHANGELOG.md) lists every change and +[SECURITY.md](SECURITY.md) describes what is and is not fixed. + +DiskCache was written and maintained by Grant Jenks from 2016 to 2023. This +fork keeps his license, copyright and author credit; see [NOTICE](NOTICE). + +## Migrating from diskcache + +The import name and the public API are unchanged, so no code changes are +needed. Only the distribution changes: + +```sh +pip uninstall diskcache +pip install diskcache-community +``` + +Never keep both installed. Uninstalling either one deletes the shared +package directory, and the fork warns at import time when both are present. +If `diskcache` arrives through a dependency, exclude it. With uv: + +```toml +[tool.uv] +exclude-dependencies = ["diskcache"] +``` + +With pip, install the dependent with `--no-deps` or exclude `diskcache` in a +constraints file. + +## Documentation + +The documentation is at +[diskcache-community.readthedocs.io](https://diskcache-community.readthedocs.io/en/latest/): + +- [Tutorial](https://diskcache-community.readthedocs.io/en/latest/tutorial.html) +- [API reference](https://diskcache-community.readthedocs.io/en/latest/api.html) +- [Cache benchmarks](https://diskcache-community.readthedocs.io/en/latest/cache-benchmarks.html) + and [DjangoCache benchmarks](https://diskcache-community.readthedocs.io/en/latest/djangocache-benchmarks.html) +- [Comparisons](https://diskcache-community.readthedocs.io/en/latest/comparisons.html) + with other key-value stores and caching libraries +- Case studies: + [web crawler](https://diskcache-community.readthedocs.io/en/latest/case-study-web-crawler.html), + [landing page caching](https://diskcache-community.readthedocs.io/en/latest/case-study-landing-page-caching.html) +- [Development](https://diskcache-community.readthedocs.io/en/latest/development.html) + +## Comparison + +| | diskcache | dbm | shelve | sqlitedict | pickleDB | +|---|---|---|---|---|---| +| Atomic | Always | Maybe | Maybe | Maybe | No | +| Thread-safe | Yes | No | No | Yes | No | +| Process-safe | Yes | No | No | Maybe | No | +| Eviction | LRU, LFU, more | None | None | None | None | +| Serialization | Customizable | None | Pickle | Customizable | JSON | +| Data types | Mapping, deque | Mapping | Mapping | Mapping | Mapping | + +The full [comparisons page](https://diskcache-community.readthedocs.io/en/latest/comparisons.html) +covers features, quality and timings in detail. + +## Contributing and security + +Bug reports and pull requests are welcome on +[GitHub](https://github.com/secondwindoss/diskcache-community/issues). +[CONTRIBUTING.md](CONTRIBUTING.md) explains the compatibility contract and +how to run the tests. Report vulnerabilities privately as described in +[SECURITY.md](SECURITY.md). + +## License + +Copyright 2016-2023 Grant Jenks. Modifications in this fork are copyright +2026 diskcache-community contributors. Licensed under the +[Apache License, Version 2.0](LICENSE). diff --git a/README.rst b/README.rst deleted file mode 100644 index 8235370..0000000 --- a/README.rst +++ /dev/null @@ -1,462 +0,0 @@ -DiskCache: Disk Backed Cache -============================ - -`DiskCache`_ is an Apache2 licensed disk and file backed cache library, written -in pure-Python, and compatible with Django. - -**This is a community fork.** It is published on PyPI as -``diskcache-community`` and imported as ``diskcache``. See `About this fork`_ -for why it exists and how to migrate. - -The cloud-based computing of 2023 puts a premium on memory. Gigabytes of empty -space is left on disks as processes vie for memory. Among these processes is -Memcached (and sometimes Redis) which is used as a cache. Wouldn't it be nice -to leverage empty disk space for caching? - -Django is Python's most popular web framework and ships with several caching -backends. Unfortunately the file-based cache in Django is essentially -broken. The culling method is random and large caches repeatedly scan a cache -directory which slows linearly with growth. Can you really allow it to take -sixty milliseconds to store a key in a cache with a thousand items? - -In Python, we can do better. And we can do it in pure-Python! - -:: - - In [1]: import pylibmc - In [2]: client = pylibmc.Client(['127.0.0.1'], binary=True) - In [3]: client[b'key'] = b'value' - In [4]: %timeit client[b'key'] - - 10000 loops, best of 3: 25.4 µs per loop - - In [5]: import diskcache as dc - In [6]: cache = dc.Cache('tmp') - In [7]: cache[b'key'] = b'value' - In [8]: %timeit cache[b'key'] - - 100000 loops, best of 3: 11.8 µs per loop - -**Note:** Micro-benchmarks have their place but are not a substitute for real -measurements. DiskCache offers cache benchmarks to defend its performance -claims. Micro-optimizations are avoided but your mileage may vary. - -DiskCache efficiently makes gigabytes of storage space available for -caching. By leveraging rock-solid database libraries and memory-mapped files, -cache performance can match and exceed industry-standard solutions. There's no -need for a C compiler or running another process. Performance is a feature and -testing has 100% coverage with unit tests and hours of stress. - -About this fork ---------------- - -`diskcache-community`_ is a community fork of `python-diskcache`_ by `Grant -Jenks`_. The last upstream release, 5.6.3, was published on 2023-08-31 and the -last upstream commit landed on 2024-03-03. Since then CVE-2025-69872 -(`GHSA-w8v5-vhqr-4h9v`_) was published against every released version: the -cache reads pickled values from its directory with ``pickle.load``, so anyone -who can write into the cache directory can run code in the process that reads -from it. The package still installs and works, but dependency scanners flag it -and no fixed upstream version exists. This fork publishes the same library as -``diskcache-community`` and carries the fix in two steps: 5.7.0 stays -compatible with 5.6.3 caches and adds an opt-in ``SafeDisk``, selected with -``Cache(directory, disk=SafeDisk)`` and likewise on ``FanoutCache``, -``Deque``, ``Index`` and the Django ``DISK`` option; 6.0.0 makes safe -reading the default. 5.7.0 is the first release of the fork; ``SECURITY.md`` -in the repository describes what it does and does not change. - -What stays the same: - -- the import name, ``import diskcache``, and the public API -- the ``cache.db`` schema, the file layout and the serialization modes, so a - 5.6.3 process opens a directory written by this fork and the reverse -- the ``JSONDisk`` bytes, the memoize key format and the persisted settings - -What changes for everyone: directories the cache creates get mode ``0700``, -so only the owner can read them; existing directories keep their permissions. - -Migrating from ``diskcache``: both distributions install into the same -``diskcache/`` package directory, so they cannot be installed side by side. -Uninstall the upstream package first, and never keep both, because -uninstalling either one deletes the shared package. The fork warns at import -time when both are installed:: - - $ pip uninstall diskcache - $ pip install diskcache-community - -If ``diskcache`` arrives through a dependency, tell the resolver to drop it. -With uv, add this to ``pyproject.toml``:: - - [tool.uv] - exclude-dependencies = ["diskcache"] - -With pip, install the dependent with ``--no-deps`` and list the rest of its -requirements yourself, or exclude ``diskcache`` through a constraints file. -End users of transitive dependents cannot switch with pip alone and should ask -the dependent project to switch. - -DiskCache was written and maintained by Grant Jenks from 2016 to 2023. This -fork keeps his license, copyright and author credit; see ``NOTICE`` in the -repository. - -.. _`diskcache-community`: https://github.com/secondwindoss/diskcache-community -.. _`python-diskcache`: https://github.com/grantjenks/python-diskcache -.. _`Grant Jenks`: https://github.com/grantjenks -.. _`GHSA-w8v5-vhqr-4h9v`: https://github.com/advisories/GHSA-w8v5-vhqr-4h9v - -Testimonials ------------- - -`Daren Hasenkamp`_, Founder -- - - "It's a useful, simple API, just like I love about Redis. It has reduced - the amount of queries hitting my Elasticsearch cluster by over 25% for a - website that gets over a million users/day (100+ hits/second)." - -`Mathias Petermann`_, Senior Linux System Engineer -- - - "I implemented it into a wrapper for our Ansible lookup modules and we were - able to speed up some Ansible runs by almost 3 times. DiskCache is saving - us a ton of time." - -Does your company or website use `DiskCache`_? Let us know in `Discussions`_. - -.. _`Daren Hasenkamp`: https://www.linkedin.com/in/daren-hasenkamp-93006438/ -.. _`Mathias Petermann`: https://www.linkedin.com/in/mathias-petermann-a8aa273b/ -.. _`Discussions`: https://github.com/secondwindoss/diskcache-community/discussions - -Features --------- - -- Pure-Python -- Fully Documented -- Benchmark comparisons (alternatives, Django cache backends) -- 100% test coverage -- Hours of stress testing -- Performance matters -- Django compatible API -- Thread-safe and process-safe -- Supports multiple eviction policies (LRU and LFU included) -- Keys support "tag" metadata and eviction -- Tested on CPython 3.10, 3.11, 3.12, 3.13 and 3.14 -- Tested on Linux, macOS and Windows with GitHub Actions - -.. image:: https://github.com/secondwindoss/diskcache-community/actions/workflows/ci.yml/badge.svg - :target: https://github.com/secondwindoss/diskcache-community/actions/workflows/ci.yml - -Quickstart ----------- - -Installing `DiskCache`_ is simple with `pip `_:: - - $ pip install diskcache-community - -You can access documentation in the interpreter with Python's built-in help -function:: - - >>> import diskcache - >>> help(diskcache) # doctest: +SKIP - -The core of `DiskCache`_ is three data types intended for caching. `Cache`_ -objects manage a SQLite database and filesystem directory to store key and -value pairs. `FanoutCache`_ provides a sharding layer to utilize multiple -caches and `DjangoCache`_ integrates that with `Django`_:: - - >>> from diskcache import Cache, FanoutCache, DjangoCache - >>> help(Cache) # doctest: +SKIP - >>> help(FanoutCache) # doctest: +SKIP - >>> help(DjangoCache) # doctest: +SKIP - -Built atop the caching data types, are `Deque`_ and `Index`_ which work as a -cross-process, persistent replacements for Python's ``collections.deque`` and -``dict``. These implement the sequence and mapping container base classes:: - - >>> from diskcache import Deque, Index - >>> help(Deque) # doctest: +SKIP - >>> help(Index) # doctest: +SKIP - -Finally, a number of `recipes`_ for cross-process synchronization are provided -using an underlying cache. Features like memoization with cache stampede -prevention, cross-process locking, and cross-process throttling are available:: - - >>> from diskcache import memoize_stampede, Lock, throttle - >>> help(memoize_stampede) # doctest: +SKIP - >>> help(Lock) # doctest: +SKIP - >>> help(throttle) # doctest: +SKIP - -Python's docstrings are a quick way to get started but not intended as a -replacement for the `DiskCache Tutorial`_ and `DiskCache API Reference`_. - -.. _`Cache`: https://diskcache-community.readthedocs.io/en/latest/tutorial.html#cache -.. _`FanoutCache`: https://diskcache-community.readthedocs.io/en/latest/tutorial.html#fanoutcache -.. _`DjangoCache`: https://diskcache-community.readthedocs.io/en/latest/tutorial.html#djangocache -.. _`Django`: https://www.djangoproject.com/ -.. _`Deque`: https://diskcache-community.readthedocs.io/en/latest/tutorial.html#deque -.. _`Index`: https://diskcache-community.readthedocs.io/en/latest/tutorial.html#index -.. _`recipes`: https://diskcache-community.readthedocs.io/en/latest/tutorial.html#recipes - -User Guide ----------- - -For those wanting more details, this part of the documentation describes -tutorial, benchmarks, API, and development. - -* `DiskCache Tutorial`_ -* `DiskCache Cache Benchmarks`_ -* `DiskCache DjangoCache Benchmarks`_ -* `Case Study: Web Crawler`_ -* `Case Study: Landing Page Caching`_ -* `Talk: All Things Cached - SF Python 2017 Meetup`_ -* `DiskCache API Reference`_ -* `DiskCache Development`_ - -.. _`DiskCache Tutorial`: https://diskcache-community.readthedocs.io/en/latest/tutorial.html -.. _`DiskCache Cache Benchmarks`: https://diskcache-community.readthedocs.io/en/latest/cache-benchmarks.html -.. _`DiskCache DjangoCache Benchmarks`: https://diskcache-community.readthedocs.io/en/latest/djangocache-benchmarks.html -.. _`Talk: All Things Cached - SF Python 2017 Meetup`: https://diskcache-community.readthedocs.io/en/latest/sf-python-2017-meetup-talk.html -.. _`Case Study: Web Crawler`: https://diskcache-community.readthedocs.io/en/latest/case-study-web-crawler.html -.. _`Case Study: Landing Page Caching`: https://diskcache-community.readthedocs.io/en/latest/case-study-landing-page-caching.html -.. _`DiskCache API Reference`: https://diskcache-community.readthedocs.io/en/latest/api.html -.. _`DiskCache Development`: https://diskcache-community.readthedocs.io/en/latest/development.html - -Comparisons ------------ - -Comparisons to popular projects related to `DiskCache`_. - -Key-Value Stores -................ - -`DiskCache`_ is mostly a simple key-value store. Feature comparisons with four -other projects are shown in the tables below. - -* `dbm`_ is part of Python's standard library and implements a generic - interface to variants of the DBM database — dbm.gnu or dbm.ndbm. If none of - these modules is installed, the slow-but-simple dbm.dumb is used. -* `shelve`_ is part of Python's standard library and implements a “shelf” as a - persistent, dictionary-like object. The difference with “dbm” databases is - that the values can be anything that the pickle module can handle. -* `sqlitedict`_ is a lightweight wrapper around Python's sqlite3 database with - a simple, Pythonic dict-like interface and support for multi-thread - access. Keys are arbitrary strings, values arbitrary pickle-able objects. -* `pickleDB`_ is a lightweight and simple key-value store. It is built upon - Python's simplejson module and was inspired by Redis. It is licensed with the - BSD three-clause license. - -.. _`dbm`: https://docs.python.org/3/library/dbm.html -.. _`shelve`: https://docs.python.org/3/library/shelve.html -.. _`sqlitedict`: https://github.com/RaRe-Technologies/sqlitedict -.. _`pickleDB`: https://pythonhosted.org/pickleDB/ - -**Features** - -================ ============= ========= ========= ============ ============ -Feature diskcache dbm shelve sqlitedict pickleDB -================ ============= ========= ========= ============ ============ -Atomic? Always Maybe Maybe Maybe No -Persistent? Yes Yes Yes Yes Yes -Thread-safe? Yes No No Yes No -Process-safe? Yes No No Maybe No -Backend? SQLite DBM DBM SQLite File -Serialization? Customizable None Pickle Customizable JSON -Data Types? Mapping/Deque Mapping Mapping Mapping Mapping -Ordering? Insert/Sorted None None None None -Eviction? LRU/LFU/more None None None None -Vacuum? Automatic Maybe Maybe Manual Automatic -Transactions? Yes No No Maybe No -Multiprocessing? Yes No No No No -Forkable? Yes No No No No -Metadata? Yes No No No No -================ ============= ========= ========= ============ ============ - -**Quality** - -================ ============= ========= ========= ============ ============ -Project diskcache dbm shelve sqlitedict pickleDB -================ ============= ========= ========= ============ ============ -Tests? Yes Yes Yes Yes Yes -Coverage? Yes Yes Yes Yes No -Stress? Yes No No No No -CI Tests? Linux/Windows Yes Yes Linux No -Python? 2/3/PyPy All All 2/3 2/3 -License? Apache2 Python Python Apache2 3-Clause BSD -Docs? Extensive Summary Summary Readme Summary -Benchmarks? Yes No No No No -Sources? GitHub GitHub GitHub GitHub GitHub -Pure-Python? Yes Yes Yes Yes Yes -Server? No No No No No -Integrations? Django None None None None -================ ============= ========= ========= ============ ============ - -**Timings** - -These are rough measurements. See `DiskCache Cache Benchmarks`_ for more -rigorous data. - -================ ============= ========= ========= ============ ============ -Project diskcache dbm shelve sqlitedict pickleDB -================ ============= ========= ========= ============ ============ -get 25 µs 36 µs 41 µs 513 µs 92 µs -set 198 µs 900 µs 928 µs 697 µs 1,020 µs -delete 248 µs 740 µs 702 µs 1,717 µs 1,020 µs -================ ============= ========= ========= ============ ============ - -Caching Libraries -................. - -* `joblib.Memory`_ provides caching functions and works by explicitly saving - the inputs and outputs to files. It is designed to work with non-hashable and - potentially large input and output data types such as numpy arrays. -* `klepto`_ extends Python’s `lru_cache` to utilize different keymaps and - alternate caching algorithms, such as `lfu_cache` and `mru_cache`. Klepto - uses a simple dictionary-sytle interface for all caches and archives. - -.. _`klepto`: https://pypi.org/project/klepto/ -.. _`joblib.Memory`: https://joblib.readthedocs.io/en/latest/memory.html - -Data Structures -............... - -* `dict`_ is a mapping object that maps hashable keys to arbitrary - values. Mappings are mutable objects. There is currently only one standard - Python mapping type, the dictionary. -* `pandas`_ is a Python package providing fast, flexible, and expressive data - structures designed to make working with “relational” or “labeled” data both - easy and intuitive. -* `Sorted Containers`_ is an Apache2 licensed sorted collections library, - written in pure-Python, and fast as C-extensions. Sorted Containers - implements sorted list, sorted dictionary, and sorted set data types. - -.. _`dict`: https://docs.python.org/3/library/stdtypes.html#typesmapping -.. _`pandas`: https://pandas.pydata.org/ -.. _`Sorted Containers`: https://pypi.org/project/sortedcontainers/ - -Pure-Python Databases -..................... - -* `ZODB`_ supports an isomorphic interface for database operations which means - there's little impact on your code to make objects persistent and there's no - database mapper that partially hides the database. -* `CodernityDB`_ is an open source, pure-Python, multi-platform, schema-less, - NoSQL database and includes an HTTP server version, and a Python client - library that aims to be 100% compatible with the embedded version. -* `TinyDB`_ is a tiny, document oriented database optimized for your - happiness. If you need a simple database with a clean API that just works - without lots of configuration, TinyDB might be the right choice for you. - -.. _`ZODB`: http://www.zodb.org/ -.. _`CodernityDB`: https://pypi.org/project/CodernityDB/ -.. _`TinyDB`: https://tinydb.readthedocs.io/ - -Object Relational Mappings (ORM) -................................ - -* `Django ORM`_ provides models that are the single, definitive source of - information about data and contains the essential fields and behaviors of the - stored data. Generally, each model maps to a single SQL database table. -* `SQLAlchemy`_ is the Python SQL toolkit and Object Relational Mapper that - gives application developers the full power and flexibility of SQL. It - provides a full suite of well known enterprise-level persistence patterns. -* `Peewee`_ is a simple and small ORM. It has few (but expressive) concepts, - making it easy to learn and intuitive to use. Peewee supports Sqlite, MySQL, - and PostgreSQL with tons of extensions. -* `SQLObject`_ is a popular Object Relational Manager for providing an object - interface to your database, with tables as classes, rows as instances, and - columns as attributes. -* `Pony ORM`_ is a Python ORM with beautiful query syntax. Use Python syntax - for interacting with the database. Pony translates such queries into SQL and - executes them in the database in the most efficient way. - -.. _`Django ORM`: https://docs.djangoproject.com/en/dev/topics/db/ -.. _`SQLAlchemy`: https://www.sqlalchemy.org/ -.. _`Peewee`: http://docs.peewee-orm.com/ -.. _`SQLObject`: http://sqlobject.org/ -.. _`Pony ORM`: https://ponyorm.com/ - -SQL Databases -............. - -* `SQLite`_ is part of Python's standard library and provides a lightweight - disk-based database that doesn’t require a separate server process and allows - accessing the database using a nonstandard variant of the SQL query language. -* `MySQL`_ is one of the world’s most popular open source databases and has - become a leading database choice for web-based applications. MySQL includes a - standardized database driver for Python platforms and development. -* `PostgreSQL`_ is a powerful, open source object-relational database system - with over 30 years of active development. Psycopg is the most popular - PostgreSQL adapter for the Python programming language. -* `Oracle DB`_ is a relational database management system (RDBMS) from the - Oracle Corporation. Originally developed in 1977, Oracle DB is one of the - most trusted and widely used enterprise relational database engines. -* `Microsoft SQL Server`_ is a relational database management system developed - by Microsoft. As a database server, it stores and retrieves data as requested - by other software applications. - -.. _`SQLite`: https://docs.python.org/3/library/sqlite3.html -.. _`MySQL`: https://dev.mysql.com/downloads/connector/python/ -.. _`PostgreSQL`: http://initd.org/psycopg/ -.. _`Oracle DB`: https://pypi.org/project/cx_Oracle/ -.. _`Microsoft SQL Server`: https://pypi.org/project/pyodbc/ - -Other Databases -............... - -* `Memcached`_ is free and open source, high-performance, distributed memory - object caching system, generic in nature, but intended for use in speeding up - dynamic web applications by alleviating database load. -* `Redis`_ is an open source, in-memory data structure store, used as a - database, cache and message broker. It supports data structures such as - strings, hashes, lists, sets, sorted sets with range queries, and more. -* `MongoDB`_ is a cross-platform document-oriented database program. Classified - as a NoSQL database program, MongoDB uses JSON-like documents with - schema. PyMongo is the recommended way to work with MongoDB from Python. -* `LMDB`_ is a lightning-fast, memory-mapped database. With memory-mapped - files, it has the read performance of a pure in-memory database while - retaining the persistence of standard disk-based databases. -* `BerkeleyDB`_ is a software library intended to provide a high-performance - embedded database for key/value data. Berkeley DB is a programmatic toolkit - that provides built-in database support for desktop and server applications. -* `LevelDB`_ is a fast key-value storage library written at Google that - provides an ordered mapping from string keys to string values. Data is stored - sorted by key and users can provide a custom comparison function. - -.. _`Memcached`: https://pypi.org/project/python-memcached/ -.. _`MongoDB`: https://api.mongodb.com/python/current/ -.. _`Redis`: https://redis.io/clients#python -.. _`LMDB`: https://lmdb.readthedocs.io/ -.. _`BerkeleyDB`: https://pypi.org/project/bsddb3/ -.. _`LevelDB`: https://plyvel.readthedocs.io/ - -Reference ---------- - -* `DiskCache Documentation`_ -* `DiskCache at PyPI`_ -* `DiskCache at GitHub`_ -* `DiskCache Issue Tracker`_ - -.. _`DiskCache Documentation`: https://github.com/secondwindoss/diskcache-community/tree/master/docs -.. _`DiskCache at PyPI`: https://pypi.org/project/diskcache-community/ -.. _`DiskCache at GitHub`: https://github.com/secondwindoss/diskcache-community/ -.. _`DiskCache Issue Tracker`: https://github.com/secondwindoss/diskcache-community/issues/ - -License -------- - -Copyright 2016-2023 Grant Jenks - -Modifications in this fork are copyright 2026 diskcache-community contributors -and are distributed under the same license. - -Licensed under the Apache License, Version 2.0 (the "License"); you may not use -this file except in compliance with the License. You may obtain a copy of the -License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software distributed -under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR -CONDITIONS OF ANY KIND, either express or implied. See the License for the -specific language governing permissions and limitations under the License. - -.. _`DiskCache`: https://github.com/secondwindoss/diskcache-community diff --git a/docs/comparisons.rst b/docs/comparisons.rst new file mode 100644 index 0000000..b858f0f --- /dev/null +++ b/docs/comparisons.rst @@ -0,0 +1,207 @@ +Comparisons +=========== + +Comparisons to popular projects related to DiskCache. + +Key-Value Stores +................ + +DiskCache is mostly a simple key-value store. Feature comparisons with four +other projects are shown in the tables below. + +* `dbm`_ is part of Python's standard library and implements a generic + interface to variants of the DBM database — dbm.gnu or dbm.ndbm. If none of + these modules is installed, the slow-but-simple dbm.dumb is used. +* `shelve`_ is part of Python's standard library and implements a “shelf” as a + persistent, dictionary-like object. The difference with “dbm” databases is + that the values can be anything that the pickle module can handle. +* `sqlitedict`_ is a lightweight wrapper around Python's sqlite3 database with + a simple, Pythonic dict-like interface and support for multi-thread + access. Keys are arbitrary strings, values arbitrary pickle-able objects. +* `pickleDB`_ is a lightweight and simple key-value store. It is built upon + Python's simplejson module and was inspired by Redis. It is licensed with the + BSD three-clause license. + +.. _`dbm`: https://docs.python.org/3/library/dbm.html +.. _`shelve`: https://docs.python.org/3/library/shelve.html +.. _`sqlitedict`: https://github.com/RaRe-Technologies/sqlitedict +.. _`pickleDB`: https://pythonhosted.org/pickleDB/ + +**Features** + +================ ============= ========= ========= ============ ============ +Feature diskcache dbm shelve sqlitedict pickleDB +================ ============= ========= ========= ============ ============ +Atomic? Always Maybe Maybe Maybe No +Persistent? Yes Yes Yes Yes Yes +Thread-safe? Yes No No Yes No +Process-safe? Yes No No Maybe No +Backend? SQLite DBM DBM SQLite File +Serialization? Customizable None Pickle Customizable JSON +Data Types? Mapping/Deque Mapping Mapping Mapping Mapping +Ordering? Insert/Sorted None None None None +Eviction? LRU/LFU/more None None None None +Vacuum? Automatic Maybe Maybe Manual Automatic +Transactions? Yes No No Maybe No +Multiprocessing? Yes No No No No +Forkable? Yes No No No No +Metadata? Yes No No No No +================ ============= ========= ========= ============ ============ + +**Quality** + +================ ============= ========= ========= ============ ============ +Project diskcache dbm shelve sqlitedict pickleDB +================ ============= ========= ========= ============ ============ +Tests? Yes Yes Yes Yes Yes +Coverage? Yes Yes Yes Yes No +Stress? Yes No No No No +CI Tests? Linux/Mac/Win Yes Yes Linux No +Python? 3.10+ All All 2/3 2/3 +License? Apache2 Python Python Apache2 3-Clause BSD +Docs? Extensive Summary Summary Readme Summary +Benchmarks? Yes No No No No +Sources? GitHub GitHub GitHub GitHub GitHub +Pure-Python? Yes Yes Yes Yes Yes +Server? No No No No No +Integrations? Django None None None None +================ ============= ========= ========= ============ ============ + +**Timings** + +These are rough measurements. See :doc:`cache-benchmarks` for more +rigorous data. + +================ ============= ========= ========= ============ ============ +Project diskcache dbm shelve sqlitedict pickleDB +================ ============= ========= ========= ============ ============ +get 25 µs 36 µs 41 µs 513 µs 92 µs +set 198 µs 900 µs 928 µs 697 µs 1,020 µs +delete 248 µs 740 µs 702 µs 1,717 µs 1,020 µs +================ ============= ========= ========= ============ ============ + +Caching Libraries +................. + +* `joblib.Memory`_ provides caching functions and works by explicitly saving + the inputs and outputs to files. It is designed to work with non-hashable and + potentially large input and output data types such as numpy arrays. +* `klepto`_ extends Python’s `lru_cache` to utilize different keymaps and + alternate caching algorithms, such as `lfu_cache` and `mru_cache`. Klepto + uses a simple dictionary-sytle interface for all caches and archives. + +.. _`klepto`: https://pypi.org/project/klepto/ +.. _`joblib.Memory`: https://joblib.readthedocs.io/en/latest/memory.html + +Data Structures +............... + +* `dict`_ is a mapping object that maps hashable keys to arbitrary + values. Mappings are mutable objects. There is currently only one standard + Python mapping type, the dictionary. +* `pandas`_ is a Python package providing fast, flexible, and expressive data + structures designed to make working with “relational” or “labeled” data both + easy and intuitive. +* `Sorted Containers`_ is an Apache2 licensed sorted collections library, + written in pure-Python, and fast as C-extensions. Sorted Containers + implements sorted list, sorted dictionary, and sorted set data types. + +.. _`dict`: https://docs.python.org/3/library/stdtypes.html#typesmapping +.. _`pandas`: https://pandas.pydata.org/ +.. _`Sorted Containers`: https://pypi.org/project/sortedcontainers/ + +Pure-Python Databases +..................... + +* `ZODB`_ supports an isomorphic interface for database operations which means + there's little impact on your code to make objects persistent and there's no + database mapper that partially hides the database. +* `CodernityDB`_ is an open source, pure-Python, multi-platform, schema-less, + NoSQL database and includes an HTTP server version, and a Python client + library that aims to be 100% compatible with the embedded version. +* `TinyDB`_ is a tiny, document oriented database optimized for your + happiness. If you need a simple database with a clean API that just works + without lots of configuration, TinyDB might be the right choice for you. + +.. _`ZODB`: http://www.zodb.org/ +.. _`CodernityDB`: https://pypi.org/project/CodernityDB/ +.. _`TinyDB`: https://tinydb.readthedocs.io/ + +Object Relational Mappings (ORM) +................................ + +* `Django ORM`_ provides models that are the single, definitive source of + information about data and contains the essential fields and behaviors of the + stored data. Generally, each model maps to a single SQL database table. +* `SQLAlchemy`_ is the Python SQL toolkit and Object Relational Mapper that + gives application developers the full power and flexibility of SQL. It + provides a full suite of well known enterprise-level persistence patterns. +* `Peewee`_ is a simple and small ORM. It has few (but expressive) concepts, + making it easy to learn and intuitive to use. Peewee supports Sqlite, MySQL, + and PostgreSQL with tons of extensions. +* `SQLObject`_ is a popular Object Relational Manager for providing an object + interface to your database, with tables as classes, rows as instances, and + columns as attributes. +* `Pony ORM`_ is a Python ORM with beautiful query syntax. Use Python syntax + for interacting with the database. Pony translates such queries into SQL and + executes them in the database in the most efficient way. + +.. _`Django ORM`: https://docs.djangoproject.com/en/dev/topics/db/ +.. _`SQLAlchemy`: https://www.sqlalchemy.org/ +.. _`Peewee`: http://docs.peewee-orm.com/ +.. _`SQLObject`: http://sqlobject.org/ +.. _`Pony ORM`: https://ponyorm.com/ + +SQL Databases +............. + +* `SQLite`_ is part of Python's standard library and provides a lightweight + disk-based database that doesn’t require a separate server process and allows + accessing the database using a nonstandard variant of the SQL query language. +* `MySQL`_ is one of the world’s most popular open source databases and has + become a leading database choice for web-based applications. MySQL includes a + standardized database driver for Python platforms and development. +* `PostgreSQL`_ is a powerful, open source object-relational database system + with over 30 years of active development. Psycopg is the most popular + PostgreSQL adapter for the Python programming language. +* `Oracle DB`_ is a relational database management system (RDBMS) from the + Oracle Corporation. Originally developed in 1977, Oracle DB is one of the + most trusted and widely used enterprise relational database engines. +* `Microsoft SQL Server`_ is a relational database management system developed + by Microsoft. As a database server, it stores and retrieves data as requested + by other software applications. + +.. _`SQLite`: https://docs.python.org/3/library/sqlite3.html +.. _`MySQL`: https://dev.mysql.com/downloads/connector/python/ +.. _`PostgreSQL`: http://initd.org/psycopg/ +.. _`Oracle DB`: https://pypi.org/project/cx_Oracle/ +.. _`Microsoft SQL Server`: https://pypi.org/project/pyodbc/ + +Other Databases +............... + +* `Memcached`_ is free and open source, high-performance, distributed memory + object caching system, generic in nature, but intended for use in speeding up + dynamic web applications by alleviating database load. +* `Redis`_ is an open source, in-memory data structure store, used as a + database, cache and message broker. It supports data structures such as + strings, hashes, lists, sets, sorted sets with range queries, and more. +* `MongoDB`_ is a cross-platform document-oriented database program. Classified + as a NoSQL database program, MongoDB uses JSON-like documents with + schema. PyMongo is the recommended way to work with MongoDB from Python. +* `LMDB`_ is a lightning-fast, memory-mapped database. With memory-mapped + files, it has the read performance of a pure in-memory database while + retaining the persistence of standard disk-based databases. +* `BerkeleyDB`_ is a software library intended to provide a high-performance + embedded database for key/value data. Berkeley DB is a programmatic toolkit + that provides built-in database support for desktop and server applications. +* `LevelDB`_ is a fast key-value storage library written at Google that + provides an ordered mapping from string keys to string values. Data is stored + sorted by key and users can provide a custom comparison function. + +.. _`Memcached`: https://pypi.org/project/python-memcached/ +.. _`MongoDB`: https://api.mongodb.com/python/current/ +.. _`Redis`: https://redis.io/clients#python +.. _`LMDB`: https://lmdb.readthedocs.io/ +.. _`BerkeleyDB`: https://pypi.org/project/bsddb3/ +.. _`LevelDB`: https://plyvel.readthedocs.io/ diff --git a/docs/development.rst b/docs/development.rst index f4f1638..0329e8c 100644 --- a/docs/development.rst +++ b/docs/development.rst @@ -87,7 +87,7 @@ the current interpreter. The options in the ``[pytest]`` section of files and fail the run if branch coverage of the ``diskcache`` package drops below 98%. The lint checks are tox environments as well:: - $ tox -e bluecheck,isortcheck,flake8,mypy,pylint,doc8,rstcheck,docs + $ tox -e bluecheck,isortcheck,flake8,mypy,pylint,doc8,docs `Read the Docs `_ builds the ``docs`` directory from ``master`` on every push, configured by diff --git a/docs/index.rst b/docs/index.rst index cea1122..b3a60ff 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,7 +1,30 @@ -.. include:: ../README.rst +DiskCache: Disk Backed Cache +============================ + +DiskCache is an Apache2 licensed disk and file backed cache library, written +in pure Python, and compatible with Django. It stores keys and values in a +SQLite database and in files next to it, is safe to share between threads and +processes, evicts by LRU, LFU and other policies, and ships ``Deque`` and +``Index`` containers plus recipes for memoization, locks and throttling. + +This documentation describes `diskcache-community`_, a community fork of +`python-diskcache`_. The fork is published on PyPI as ``diskcache-community`` +and imported as ``diskcache``. It keeps the public API, the ``cache.db`` +schema and the file layout of the last upstream release, 5.6.3, so existing +cache directories keep working, and it carries the fix path for +CVE-2025-69872 through the opt-in :class:`SafeDisk `. +The README in the repository explains why the fork exists and how to migrate. + +Install it with pip, after removing the upstream ``diskcache`` package if it +is present:: + + $ pip uninstall diskcache + $ pip install diskcache-community + +Start with the :doc:`tutorial` and keep the :doc:`api` at hand. .. toctree:: - :hidden: + :maxdepth: 1 tutorial cache-benchmarks @@ -9,5 +32,9 @@ case-study-web-crawler case-study-landing-page-caching sf-python-2017-meetup-talk + comparisons api development + +.. _`diskcache-community`: https://github.com/secondwindoss/diskcache-community +.. _`python-diskcache`: https://github.com/grantjenks/python-diskcache diff --git a/pyproject.toml b/pyproject.toml index 207371f..ddf05c0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta" name = "diskcache-community" dynamic = ["version"] description = "Disk Cache -- Disk and file backed persistent cache." -readme = "README.rst" +readme = "README.md" requires-python = ">=3.10" license = "Apache-2.0" license-files = ["LICENSE", "NOTICE"] diff --git a/tox.ini b/tox.ini index d7ba744..d884cba 100644 --- a/tox.ini +++ b/tox.ini @@ -1,5 +1,5 @@ [tox] -envlist=bluecheck,doc8,docs,isortcheck,flake8,mypy,pylint,rstcheck,py38,py39,py310,py311 +envlist=bluecheck,doc8,docs,isortcheck,flake8,mypy,pylint,py38,py39,py310,py311 skip_missing_interpreters=True [testenv] @@ -56,10 +56,6 @@ deps= django==4.2.* pylint<3.3 -[testenv:rstcheck] -commands=rstcheck {toxinidir}/README.rst -deps=rstcheck - [isort] multi_line_output = 3 include_trailing_comma = True