1 DiskCache: Disk Backed Cache
2 ============================
4 `DiskCache`_ is an Apache2 licensed disk and file backed cache library, written
5 in pure-Python, and compatible with Django.
7 The cloud-based computing of 2019 puts a premium on memory. Gigabytes of empty
8 space is left on disks as processes vie for memory. Among these processes is
9 Memcached (and sometimes Redis) which is used as a cache. Wouldn't it be nice
10 to leverage empty disk space for caching?
12 Django is Python's most popular web framework and ships with several caching
13 backends. Unfortunately the file-based cache in Django is essentially
14 broken. The culling method is random and large caches repeatedly scan a cache
15 directory which slows linearly with growth. Can you really allow it to take
16 sixty milliseconds to store a key in a cache with a thousand items?
18 In Python, we can do better. And we can do it in pure-Python!
22 In [1]: import pylibmc
23 In [2]: client = pylibmc.Client(['127.0.0.1'], binary=True)
24 In [3]: client[b'key'] = b'value'
25 In [4]: %timeit client[b'key']
27 10000 loops, best of 3: 25.4 µs per loop
29 In [5]: import diskcache as dc
30 In [6]: cache = dc.Cache('tmp')
31 In [7]: cache[b'key'] = b'value'
32 In [8]: %timeit cache[b'key']
34 100000 loops, best of 3: 11.8 µs per loop
36 **Note:** Micro-benchmarks have their place but are not a substitute for real
37 measurements. DiskCache offers cache benchmarks to defend its performance
38 claims. Micro-optimizations are avoided but your mileage may vary.
40 DiskCache efficiently makes gigabytes of storage space available for
41 caching. By leveraging rock-solid database libraries and memory-mapped files,
42 cache performance can match and exceed industry-standard solutions. There's no
43 need for a C compiler or running another process. Performance is a feature and
44 testing has 100% coverage with unit tests and hours of stress.
49 `Daren Hasenkamp`_, Founder --
51 "It's a useful, simple API, just like I love about Redis. It has reduced
52 the amount of queries hitting my Elasticsearch cluster by over 25% for a
53 website that gets over a million users/day (100+ hits/second)."
55 `Mathias Petermann`_, Senior Linux System Engineer --
57 "I implemented it into a wrapper for our Ansible lookup modules and we were
58 able to speed up some Ansible runs by almost 3 times. DiskCache is saving
61 Does your company or website use `DiskCache`_? Send us a `message
62 <contact@grantjenks.com>`_ and let us know.
64 .. _`Daren Hasenkamp`: https://www.linkedin.com/in/daren-hasenkamp-93006438/
65 .. _`Mathias Petermann`: https://www.linkedin.com/in/mathias-petermann-a8aa273b/
72 - Benchmark comparisons (alternatives, Django cache backends)
74 - Hours of stress testing
76 - Django compatible API
77 - Thread-safe and process-safe
78 - Supports multiple eviction policies (LRU and LFU included)
79 - Keys support "tag" metadata and eviction
80 - Developed on Python 3.7
81 - Tested on CPython 2.7, 3.4, 3.5, 3.6, 3.7 and PyPy
82 - Tested on Linux, Mac OS X, and Windows
83 - Tested using Travis CI and AppVeyor CI
85 .. image:: https://api.travis-ci.org/grantjenks/python-diskcache.svg?branch=master
86 :target: http://www.grantjenks.com/docs/diskcache/
88 .. image:: https://ci.appveyor.com/api/projects/status/github/grantjenks/python-diskcache?branch=master&svg=true
89 :target: http://www.grantjenks.com/docs/diskcache/
94 Installing `DiskCache`_ is simple with `pip <http://www.pip-installer.org/>`_::
96 $ pip install diskcache
98 You can access documentation in the interpreter with Python's built-in help
104 The core of `DiskCache`_ is three data types intended for caching. `Cache`_
105 objects manage a SQLite database and filesystem directory to store key and
106 value pairs. `FanoutCache`_ provides a sharding layer to utilize multiple
107 caches and `DjangoCache`_ integrates that with `Django`_::
109 >>> from diskcache import Cache, FanoutCache, DjangoCache
111 >>> help(FanoutCache)
112 >>> help(DjangoCache)
114 Built atop the caching data types, are `Deque`_ and `Index`_ which work as a
115 cross-process, persistent replacements for Python's ``collections.deque`` and
116 ``dict``. These implement the sequence and mapping container base classes::
118 >>> from diskcache import Deque, Index
122 Finally, a number of `recipes`_ for cross-process synchronization are provided
123 using an underlying cache. Features like memoization with cache stampede
124 prevention, cross-process locking, and cross-process throttling are available::
126 >>> from diskcache import memoize_stampede, Lock, throttle
127 >>> help(memoize_stampede)
131 Python's docstrings are a quick way to get started but not intended as a
132 replacement for the `DiskCache Tutorial`_ and `DiskCache API Reference`_.
134 .. _`Cache`: http://www.grantjenks.com/docs/diskcache/tutorial.html#cache
135 .. _`FanoutCache`: http://www.grantjenks.com/docs/diskcache/tutorial.html#fanoutcache
136 .. _`DjangoCache`: http://www.grantjenks.com/docs/diskcache/tutorial.html#djangocache
137 .. _`Django`: https://www.djangoproject.com/
138 .. _`Deque`: http://www.grantjenks.com/docs/diskcache/tutorial.html#deque
139 .. _`Index`: http://www.grantjenks.com/docs/diskcache/tutorial.html#index
140 .. _`recipes`: http://www.grantjenks.com/docs/diskcache/tutorial.html#recipes
145 For those wanting more details, this part of the documentation describes
146 tutorial, benchmarks, API, and development.
148 * `DiskCache Tutorial`_
149 * `DiskCache Cache Benchmarks`_
150 * `DiskCache DjangoCache Benchmarks`_
151 * `Case Study: Web Crawler`_
152 * `Case Study: Landing Page Caching`_
153 * `Talk: All Things Cached - SF Python 2017 Meetup`_
154 * `DiskCache API Reference`_
155 * `DiskCache Development`_
157 .. _`DiskCache Tutorial`: http://www.grantjenks.com/docs/diskcache/tutorial.html
158 .. _`DiskCache Cache Benchmarks`: http://www.grantjenks.com/docs/diskcache/cache-benchmarks.html
159 .. _`DiskCache DjangoCache Benchmarks`: http://www.grantjenks.com/docs/diskcache/djangocache-benchmarks.html
160 .. _`Talk: All Things Cached - SF Python 2017 Meetup`: http://www.grantjenks.com/docs/diskcache/sf-python-2017-meetup-talk.html
161 .. _`Case Study: Web Crawler`: http://www.grantjenks.com/docs/diskcache/case-study-web-crawler.html
162 .. _`Case Study: Landing Page Caching`: http://www.grantjenks.com/docs/diskcache/case-study-landing-page-caching.html
163 .. _`DiskCache API Reference`: http://www.grantjenks.com/docs/diskcache/api.html
164 .. _`DiskCache Development`: http://www.grantjenks.com/docs/diskcache/development.html
169 Comparisons to popular projects related to `DiskCache`_.
174 `DiskCache`_ is mostly a simple key-value store. Feature comparisons with four
175 other projects are shown in the tables below.
177 * `dbm`_ is part of Python's standard library and implements a generic
178 interface to variants of the DBM database — dbm.gnu or dbm.ndbm. If none of
179 these modules is installed, the slow-but-simple dbm.dumb is used.
180 * `shelve`_ is part of Python's standard library and implements a “shelf” as a
181 persistent, dictionary-like object. The difference with “dbm” databases is
182 that the values can be anything that the pickle module can handle.
183 * `sqlitedict`_ is a lightweight wrapper around Python's sqlite3 database with
184 a simple, Pythonic dict-like interface and support for multi-thread
185 access. Keys are arbitrary strings, values arbitrary pickle-able objects.
186 * `pickleDB`_ is a lightweight and simple key-value store. It is built upon
187 Python's simplejson module and was inspired by Redis. It is licensed with the
188 BSD three-caluse license.
190 .. _`dbm`: https://docs.python.org/3/library/dbm.html
191 .. _`shelve`: https://docs.python.org/3/library/shelve.html
192 .. _`sqlitedict`: https://github.com/RaRe-Technologies/sqlitedict
193 .. _`pickleDB`: https://pythonhosted.org/pickleDB/
197 ================ ============= ========= ========= ============ ============
198 Feature diskcache dbm shelve sqlitedict pickleDB
199 ================ ============= ========= ========= ============ ============
200 Atomic? Always Maybe Maybe Maybe No
201 Persistent? Yes Yes Yes Yes Yes
202 Thread-safe? Yes No No Yes No
203 Process-safe? Yes No No Maybe No
204 Backend? SQLite DBM DBM SQLite File
205 Serialization? Customizable None Pickle Customizable JSON
206 Data Types? Mapping/Deque Mapping Mapping Mapping Mapping
207 Ordering? Insert/Sorted None None None None
208 Eviction? LRU/LFU/more None None None None
209 Vacuum? Automatic Maybe Maybe Manual Automatic
210 Transactions? Yes No No Maybe No
211 Multiprocessing? Yes No No No No
212 Forkable? Yes No No No No
213 Metadata? Yes No No No No
214 ================ ============= ========= ========= ============ ============
218 ================ ============= ========= ========= ============ ============
219 Project diskcache dbm shelve sqlitedict pickleDB
220 ================ ============= ========= ========= ============ ============
221 Tests? Yes Yes Yes Yes Yes
222 Coverage? Yes Yes Yes Yes No
223 Stress? Yes No No No No
224 CI Tests? Linux/Windows Yes Yes Linux No
225 Python? 2/3/PyPy All All 2/3 2/3
226 License? Apache2 Python Python Apache2 3-Clause BSD
227 Docs? Extensive Summary Summary Readme Summary
228 Benchmarks? Yes No No No No
229 Sources? GitHub GitHub GitHub GitHub GitHub
230 Pure-Python? Yes Yes Yes Yes Yes
231 Server? No No No No No
232 Integrations? Django None None None None
233 ================ ============= ========= ========= ============ ============
237 These are rough measurements. See `DiskCache Cache Benchmarks`_ for more
240 ================ ============= ========= ========= ============ ============
241 Project diskcache dbm shelve sqlitedict pickleDB
242 ================ ============= ========= ========= ============ ============
243 get 25 µs 36 µs 41 µs 513 µs 92 µs
244 set 198 µs 900 µs 928 µs 697 µs 1,020 µs
245 delete 248 µs 740 µs 702 µs 1,717 µs 1,020 µs
246 ================ ============= ========= ========= ============ ============
251 * `joblib.Memory`_ provides caching functions and works by explicitly saving
252 the inputs and outputs to files. It is designed to work with non-hashable and
253 potentially large input and output data types such as numpy arrays.
254 * `klepto`_ extends Python’s `lru_cache` to utilize different keymaps and
255 alternate caching algorithms, such as `lfu_cache` and `mru_cache`. Klepto
256 uses a simple dictionary-sytle interface for all caches and archives.
258 .. _`klepto`: https://pypi.org/project/klepto/
259 .. _`joblib.Memory`: https://joblib.readthedocs.io/en/latest/memory.html
264 * `dict`_ is a mapping object that maps hashable keys to arbitrary
265 values. Mappings are mutable objects. There is currently only one standard
266 Python mapping type, the dictionary.
267 * `pandas`_ is a Python package providing fast, flexible, and expressive data
268 structures designed to make working with “relational” or “labeled” data both
270 * `Sorted Containers`_ is an Apache2 licensed sorted collections library,
271 written in pure-Python, and fast as C-extensions. Sorted Containers
272 implements sorted list, sorted dictionary, and sorted set data types.
274 .. _`dict`: https://docs.python.org/3/library/stdtypes.html#typesmapping
275 .. _`pandas`: https://pandas.pydata.org/
276 .. _`Sorted Containers`: http://www.grantjenks.com/docs/sortedcontainers/
278 Pure-Python Databases
279 .....................
281 * `ZODB`_ supports an isomorphic interface for database operations which means
282 there's little impact on your code to make objects persistent and there's no
283 database mapper that partially hides the datbase.
284 * `CodernityDB`_ is an open source, pure-Python, multi-platform, schema-less,
285 NoSQL database and includes an HTTP server version, and a Python client
286 library that aims to be 100% compatible with the embedded version.
287 * `TinyDB`_ is a tiny, document oriented database optimized for your
288 happiness. If you need a simple database with a clean API that just works
289 without lots of configuration, TinyDB might be the right choice for you.
291 .. _`ZODB`: http://www.zodb.org/
292 .. _`CodernityDB`: https://pypi.org/project/CodernityDB/
293 .. _`TinyDB`: https://tinydb.readthedocs.io/
295 Object Relational Mappings (ORM)
296 ................................
298 * `Django ORM`_ provides models that are the single, definitive source of
299 information about data and contains the essential fields and behaviors of the
300 stored data. Generally, each model maps to a single SQL database table.
301 * `SQLAlchemy`_ is the Python SQL toolkit and Object Relational Mapper that
302 gives application developers the full power and flexibility of SQL. It
303 provides a full suite of well known enterprise-level persistence patterns.
304 * `Peewee`_ is a simple and small ORM. It has few (but expressive) concepts,
305 making it easy to learn and intuitive to use. Peewee supports Sqlite, MySQL,
306 and PostgreSQL with tons of extensions.
307 * `SQLObject`_ is a popular Object Relational Manager for providing an object
308 interface to your database, with tables as classes, rows as instances, and
309 columns as attributes.
310 * `Pony ORM`_ is a Python ORM with beautiful query syntax. Use Python syntax
311 for interacting with the database. Pony translates such queries into SQL and
312 executes them in the database in the most efficient way.
314 .. _`Django ORM`: https://docs.djangoproject.com/en/dev/topics/db/
315 .. _`SQLAlchemy`: https://www.sqlalchemy.org/
316 .. _`Peewee`: http://docs.peewee-orm.com/
317 .. _`dataset`: https://dataset.readthedocs.io/
318 .. _`SQLObject`: http://sqlobject.org/
319 .. _`Pony ORM`: https://ponyorm.com/
324 * `SQLite`_ is part of Python's standard library and provides a lightweight
325 disk-based database that doesn’t require a separate server process and allows
326 accessing the database using a nonstandard variant of the SQL query language.
327 * `MySQL`_ is one of the world’s most popular open source databases and has
328 become a leading database choice for web-based applications. MySQL includes a
329 standardized database driver for Python platforms and development.
330 * `PostgreSQL`_ is a powerful, open source object-relational database system
331 with over 30 years of active development. Psycopg is the most popular
332 PostgreSQL adapter for the Python programming language.
333 * `Oracle DB`_ is a relational database management system (RDBMS) from the
334 Oracle Corporation. Originally developed in 1977, Oracle DB is one of the
335 most trusted and widely used enterprise relational database engines.
336 * `Microsoft SQL Server`_ is a relational database management system developed
337 by Microsoft. As a database server, it stores and retrieves data as requested
338 by other software applications.
340 .. _`SQLite`: https://docs.python.org/3/library/sqlite3.html
341 .. _`MySQL`: https://dev.mysql.com/downloads/connector/python/
342 .. _`PostgreSQL`: http://initd.org/psycopg/
343 .. _`Oracle DB`: https://pypi.org/project/cx_Oracle/
344 .. _`Microsoft SQL Server`: https://pypi.org/project/pyodbc/
349 * `Memcached`_ is free and open source, high-performance, distributed memory
350 object caching system, generic in nature, but intended for use in speeding up
351 dynamic web applications by alleviating database load.
352 * `Redis`_ is an open source, in-memory data structure store, used as a
353 database, cache and message broker. It supports data structures such as
354 strings, hashes, lists, sets, sorted sets with range queries, and more.
355 * `MongoDB`_ is a cross-platform document-oriented database program. Classified
356 as a NoSQL database program, MongoDB uses JSON-like documents with
357 schema. PyMongo is the recommended way to work with MongoDB from Python.
358 * `LMDB`_ is a lightning-fast, memory-mapped database. With memory-mapped
359 files, it has the read performance of a pure in-memory database while
360 retaining the persistence of standard disk-based databases.
361 * `BerkeleyDB`_ is a software library intended to provide a high-performance
362 embedded database for key/value data. Berkeley DB is a programmatic toolkit
363 that provides built-in database support for desktop and server applications.
364 * `LevelDB`_ is a fast key-value storage library written at Google that
365 provides an ordered mapping from string keys to string values. Data is stored
366 sorted by key and users can provide a custom comparison function.
368 .. _`Memcached`: https://pypi.org/project/python-memcached/
369 .. _`MongoDB`: https://api.mongodb.com/python/current/
370 .. _`Redis`: https://redis.io/clients#python
371 .. _`LMDB`: https://lmdb.readthedocs.io/
372 .. _`BerkeleyDB`: https://pypi.org/project/bsddb3/
373 .. _`LevelDB`: https://plyvel.readthedocs.io/
378 * `DiskCache Documentation`_
379 * `DiskCache at PyPI`_
380 * `DiskCache at GitHub`_
381 * `DiskCache Issue Tracker`_
383 .. _`DiskCache Documentation`: http://www.grantjenks.com/docs/diskcache/
384 .. _`DiskCache at PyPI`: https://pypi.python.org/pypi/diskcache/
385 .. _`DiskCache at GitHub`: https://github.com/grantjenks/python-diskcache/
386 .. _`DiskCache Issue Tracker`: https://github.com/grantjenks/python-diskcache/issues/
391 Copyright 2016-2019 Grant Jenks
393 Licensed under the Apache License, Version 2.0 (the "License"); you may not use
394 this file except in compliance with the License. You may obtain a copy of the
397 http://www.apache.org/licenses/LICENSE-2.0
399 Unless required by applicable law or agreed to in writing, software distributed
400 under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
401 CONDITIONS OF ANY KIND, either express or implied. See the License for the
402 specific language governing permissions and limitations under the License.
404 .. _`DiskCache`: http://www.grantjenks.com/docs/diskcache/