shelve

Manage shelves of pickled objects.

A "shelf" is a persistent, dictionary-like object.  The difference
with dbm databases is that the values (not the keys!) in a shelf can
be essentially arbitrary Python objects -- anything that the "pickle"
module can handle.  This includes most class instances, recursive data
types, and objects containing lots of shared sub-objects.  The keys
are ordinary strings.

To summarize the interface (key is a string, data is an arbitrary
object):

        import shelve
        d = shelve.open(filename) # open, with (g)dbm filename -- no suffix

        d[key] = data   # store data at key (overwrites old data if
                        # using an existing key)
        data = d[key]   # retrieve a COPY of the data at key (raise
                        # KeyError if no such key) -- NOTE that this
                        # access returns a *copy* of the entry!
        del d[key]      # delete data stored at key (raises KeyError
                        # if no such key)
        flag = key in d # true if the key exists
        list = d.keys() # a list of all existing keys (slow!)

        d.close()       # close it

Dependent on the implementation, closing a persistent dictionary may
or may not be necessary to flush changes to disk.

Normally, d[key] returns a COPY of the entry.  This needs care when
mutable entries are mutated: for example, if d[key] is a list,
        d[key].append(anitem)
does NOT modify the entry d[key] itself, as stored in the persistent
mapping -- it only modifies the copy, which is then immediately
discarded, so that the append has NO effect whatsoever.  To append an
item to d[key] in a way that will affect the persistent mapping, use:
        data = d[key]
        data.append(anitem)
        d[key] = data

To avoid the problem with mutable entries, you may pass the keyword
argument writeback=True in the call to shelve.open.  When you use:
        d = shelve.open(filename, writeback=True)
then d keeps a cache of all entries you access, and writes them all back
to the persistent mapping when you call d.close().  This ensures that
such usage as d[key].append(anitem) works as intended.

However, using keyword argument writeback=True may consume vast amount
of memory for the cache, and it may make d.close() very slow, if you
access many of d's entries after opening it in this way: d has no way to
check which of the entries you access are mutable and/or which ones you
actually mutate, so it must cache, and write back at close, all of the
entries that you access.  You can call d.sync() to write back all the
entries in the cache, and empty the cache (d.sync() also synchronizes
the persistent dictionary on disk, if feasible).

Classes

BsdDbShelf

Shelf implementation using the "BSD" db interface.

    This adds methods first(), next(), previous(), last() and
    set_location() that have no counterpart in [g]dbm databases.

    The actual database must be opened using one of the "bsddb"
    modules "open" routines (i.e. bsddb.hashopen, bsddb.btopen or
    bsddb.rnopen) and passed to the constructor.

    See the module's __doc__ string for an overview of the interface.

clear(self)

  D.clear() -> None.  Remove all items from D.

close(self)

first(self)

get(self, key, default=None)

items(self)

  D.items() -> a set-like object providing a view on D's items

keys(self)

  D.keys() -> a set-like object providing a view on D's keys

last(self)

next(self)

pop(self, key, default=<object object at 0x7f92bfcfe170>)

  D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
            If key is not found, d is returned if given, otherwise KeyError is raised.

popitem(self)

  D.popitem() -> (k, v), remove and return some (key, value) pair
             as a 2-tuple; but raise KeyError if D is empty.

previous(self)

set_location(self, key)

setdefault(self, key, default=None)

  D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D

sync(self)

update(self, other=(), /, **kwds)

   D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
              If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
              If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
              In either case, this is followed by: for k, v in F.items(): D[k] = v

values(self)

  D.values() -> an object providing a view on D's values

BytesIO

Buffered I/O implementation using an in-memory bytes buffer.

close(self, /)

  Disable all I/O operations.

detach(self, /)

  Disconnect this buffer from its underlying raw stream and return it.

  After the raw stream has been detached, the buffer is in an unusable
  state.

fileno(self, /)

  Returns underlying file descriptor if one exists.

  OSError is raised if the IO object does not use a file descriptor.

flush(self, /)

  Does nothing.

getbuffer(self, /)

  Get a read-write view over the contents of the BytesIO object.

getvalue(self, /)

  Retrieve the entire contents of the BytesIO object.

isatty(self, /)

  Always returns False.

  BytesIO objects are not connected to a TTY-like device.

read(self, size=-1, /)

  Read at most size bytes, returned as a bytes object.

  If the size argument is negative, read until EOF is reached.
  Return an empty bytes object at EOF.

read1(self, size=-1, /)

  Read at most size bytes, returned as a bytes object.

  If the size argument is negative or omitted, read until EOF is reached.
  Return an empty bytes object at EOF.

readable(self, /)

  Returns True if the IO object can be read.

readinto(self, buffer, /)

  Read bytes into buffer.

  Returns number of bytes read (0 for EOF), or None if the object
  is set not to block and has no data to read.

readinto1(self, buffer, /)

readline(self, size=-1, /)

  Next line from the file, as a bytes object.

  Retain newline.  A non-negative size argument limits the maximum
  number of bytes to return (an incomplete line may be returned then).
  Return an empty bytes object at EOF.

readlines(self, size=None, /)

  List of bytes objects, each a line from the file.

  Call readline() repeatedly and return a list of the lines so read.
  The optional size argument, if given, is an approximate bound on the
  total number of bytes in the lines returned.

seek(self, pos, whence=0, /)

  Change stream position.

  Seek to byte offset pos relative to position indicated by whence:
       0  Start of stream (the default).  pos should be >= 0;
       1  Current position - pos may be negative;
       2  End of stream - pos usually negative.
  Returns the new absolute position.

seekable(self, /)

  Returns True if the IO object can be seeked.

tell(self, /)

  Current file position, an integer.

truncate(self, size=None, /)

  Truncate the file to at most size bytes.

  Size defaults to the current file position, as returned by tell().
  The current file position is unchanged.  Returns the new size.

writable(self, /)

  Returns True if the IO object can be written.

write(self, b, /)

  Write bytes to file.

  Return the number of bytes written.

writelines(self, lines, /)

  Write lines to the file.

  Note that newlines are not added.  lines can be any iterable object
  producing bytes-like objects. This is equivalent to calling write() for
  each element.

closed = <attribute 'closed' of '_io.BytesIO' objects>
  True if the file is closed.

DbfilenameShelf

Shelf implementation using the "dbm" generic dbm interface.

    This is initialized with the filename for the dbm database.
    See the module's __doc__ string for an overview of the interface.

clear(self)

  D.clear() -> None.  Remove all items from D.

close(self)

get(self, key, default=None)

items(self)

  D.items() -> a set-like object providing a view on D's items

keys(self)

  D.keys() -> a set-like object providing a view on D's keys

pop(self, key, default=<object object at 0x7f92bfcfe170>)

  D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
            If key is not found, d is returned if given, otherwise KeyError is raised.

popitem(self)

  D.popitem() -> (k, v), remove and return some (key, value) pair
             as a 2-tuple; but raise KeyError if D is empty.

setdefault(self, key, default=None)

  D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D

sync(self)

update(self, other=(), /, **kwds)

   D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
              If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
              If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
              In either case, this is followed by: for k, v in F.items(): D[k] = v

values(self)

  D.values() -> an object providing a view on D's values

Pickler

This takes a binary file for writing a pickle data stream.

The optional *protocol* argument tells the pickler to use the given
protocol; supported protocols are 0, 1, 2, 3, 4 and 5.  The default
protocol is 4. It was introduced in Python 3.4, and is incompatible
with previous versions.

Specifying a negative protocol version selects the highest protocol
version supported.  The higher the protocol used, the more recent the
version of Python needed to read the pickle produced.

The *file* argument must have a write() method that accepts a single
bytes argument. It can thus be a file object opened for binary
writing, an io.BytesIO instance, or any other custom object that meets
this interface.

If *fix_imports* is True and protocol is less than 3, pickle will try
to map the new Python 3 names to the old module names used in Python
2, so that the pickle data stream is readable with Python 2.

If *buffer_callback* is None (the default), buffer views are
serialized into *file* as part of the pickle stream.

If *buffer_callback* is not None, then it can be called any number
of times with a buffer view.  If the callback returns a false value
(such as None), the given buffer is out-of-band; otherwise the
buffer is serialized in-band, i.e. inside the pickle stream.

It is an error if *buffer_callback* is not None and *protocol*
is None or smaller than 5.

clear_memo(self, /)

  Clears the pickler's "memo".

  The memo is the data structure that remembers which objects the
  pickler has already seen, so that shared or recursive objects are
  pickled by reference and not by value.  This method is useful when
  re-using picklers.

dump(self, obj, /)

  Write a pickled representation of the given object to the open file.

bin = <member 'bin' of '_pickle.Pickler' objects>

dispatch_table = <member 'dispatch_table' of '_pickle.Pickler' objects>

fast = <member 'fast' of '_pickle.Pickler' objects>

memo = <attribute 'memo' of '_pickle.Pickler' objects>

persistent_id = <attribute 'persistent_id' of '_pickle.Pickler' objects>

Shelf

Base class for shelf implementations.

    This is initialized with a dictionary-like object.
    See the module's __doc__ string for an overview of the interface.

clear(self)

  D.clear() -> None.  Remove all items from D.

close(self)

get(self, key, default=None)

items(self)

  D.items() -> a set-like object providing a view on D's items

keys(self)

  D.keys() -> a set-like object providing a view on D's keys

pop(self, key, default=<object object at 0x7f92bfcfe170>)

  D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
            If key is not found, d is returned if given, otherwise KeyError is raised.

popitem(self)

  D.popitem() -> (k, v), remove and return some (key, value) pair
             as a 2-tuple; but raise KeyError if D is empty.

setdefault(self, key, default=None)

  D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D

sync(self)

update(self, other=(), /, **kwds)

   D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
              If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
              If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
              In either case, this is followed by: for k, v in F.items(): D[k] = v

values(self)

  D.values() -> an object providing a view on D's values

Unpickler

This takes a binary file for reading a pickle data stream.

The protocol version of the pickle is detected automatically, so no
protocol argument is needed.  Bytes past the pickled object's
representation are ignored.

The argument *file* must have two methods, a read() method that takes
an integer argument, and a readline() method that requires no
arguments.  Both methods should return bytes.  Thus *file* can be a
binary file object opened for reading, an io.BytesIO object, or any
other custom object that meets this interface.

Optional keyword arguments are *fix_imports*, *encoding* and *errors*,
which are used to control compatibility support for pickle stream
generated by Python 2.  If *fix_imports* is True, pickle will try to
map the old Python 2 names to the new names used in Python 3.  The

	encoding* and *errors* tell pickle how to decode 8-bit string

instances pickled by Python 2; these default to 'ASCII' and 'strict',
respectively.  The *encoding* can be 'bytes' to read these 8-bit
string instances as bytes objects.

find_class(self, module_name, global_name, /)

  Return an object from a specified module.

  If necessary, the module will be imported. Subclasses may override
  this method (e.g. to restrict unpickling of arbitrary classes and
  functions).

  This method is called whenever a class or a function object is
  needed.  Both arguments passed are str objects.

load(self, /)

  Load a pickle.

  Read a pickled object representation from the open file object given
  in the constructor, and return the reconstituted object hierarchy
  specified therein.

memo = <attribute 'memo' of '_pickle.Unpickler' objects>

persistent_load = <attribute 'persistent_load' of '_pickle.Unpickler' objects>

Functions

open

open(filename, flag='c', protocol=None, writeback=False)

  Open a persistent dictionary for reading and writing.

      The filename parameter is the base filename for the underlying
      database.  As a side-effect, an extension may be added to the
      filename and more than one file may be created.  The optional flag
      parameter has the same interpretation as the flag parameter of
      dbm.open(). The optional protocol parameter specifies the
      version of the pickle protocol.

      See the module's __doc__ string for an overview of the interface.

Modules

collections