Sizing individual objects

Introduction

This module exposes 9 functions and 2 classes to obtain lengths and sizes of Python objects (for Python 2.6 or later).

Earlier versions of this module supported Python versions down to Python 2.2. If you are using Python 2.5 or older, please consider downgrading Pympler to version 0.3.x.

Public Functions 1

Function asizeof calculates the combined (approximate) size in bytes of one or several Python objects.

Function asizesof returns a tuple containing the (approximate) size in bytes for each given Python object separately.

Function asized returns for each object an instance of class Asized containing all the size information of the object and a tuple with the referents 2.

Functions basicsize and itemsize return the basic- respectively itemsize of the given object, both in bytes. For objects as array.array, numpy.array, numpy.matrix, etc. where the item size varies depending on the instance-specific data type, function itemsize returns that item size.

Function flatsize returns the flat size of a Python object in bytes defined as the basic size plus the item size times the length of the given object.

Function leng returns the length of an object, like standard function len but extended for several types. E.g. the leng of a multi-precision int (or long) is the number of digits 4. The length of most mutable sequence objects includes an estimate of the over-allocation and therefore, the leng value may differ from the standard len result. For objects like array.array, numpy.array, numpy.matrix, etc. function leng returns the proper number of items.

Function refs returns (a generator for) the referents 2 of the given object.

Certain classes are known to be sub-classes of or to behave as dict objects. Function adict can be used to register other class objects to be treated like dict.

Public Classes 1

Class Asizer may be used to accumulate the results of several asizeof or asizesof calls. After creating an Asizer instance, use methods asizeof and asizesof as needed to size any number of additional objects.

Call methods exclude_refs and/or exclude_types to exclude references to respectively instances or types of certain objects.

Use one of the print_… methods to report the statistics.

An instance of class Asized is returned for each object sized by the asized function or method.

Duplicate Objects

Any duplicate, given objects are sized only once and the size is included in the accumulated total only once. But functions asizesof and asized will return a size value respectively an Asized instance for each given object, including duplicates.

Definitions 3

The length of an objects like dict, list, set, str, tuple, etc. is defined as the number of items held in or allocated by the object. Held items are references to other objects, called the referents.

The size of an object is defined as the sum of the flat size of the object plus the sizes of any referents 2. Referents are visited recursively up to the specified detail level. However, the size of objects referenced multiple times is included only once in the total size.

The flat size of an object is defined as the basic size of the object plus the item size times the number of allocated items, references to referents. The flat size does include the size for the references to the referents, but not the size of the referents themselves.

The flat size returned by function flatsize equals the result of function asizeof with options code=True, ignored=False, limit=0 and option align set to the same value.

The accurate flat size for an object is obtained from function sys.getsizeof() where available. Otherwise, the length and size of sequence objects as dicts, lists, sets, etc. is based on an estimate for the number of allocated items. As a result, the reported length and size may differ substantially from the actual length and size.

The basic and item size are obtained from the __basicsize__ respectively __itemsize__ attributes of the (type of the) object. Where necessary (e.g. sequence objects), a zero __itemsize__ is replaced by the size of a corresponding C type.

The overhead for Python’s garbage collector (GC) is included in the basic size of (GC managed) objects as well as the space needed for refcounts (used only in certain Python builds).

Optionally, size values can be aligned to any power-of-2 multiple.

Size of (byte)code

The (byte)code size of objects like classes, functions, methods, modules, etc. can be included by setting option code=True.

Iterators are handled like sequences: iterated object(s) are sized like referents 2, but only up to the specified level or recursion limit (and only if function gc.get_referents() returns the referent object of iterators).

Generators are sized as (byte)code only, but the generated objects are never sized.

Old- and New-style Classes

All old- and new-style class, instance and type objects are handled uniformly such that (a) instance objects are distinguished from class objects and (b) instances of different old-style classes can be dealt with separately.

Class and type objects are represented as <class ....* def> respectively <type ... def> where the * indicates an old-style class and the ... def suffix marks the definition object. Instances of classes are shown as <class module.name*> without the ... def suffix. The * after the name indicates an instance of an old-style class.

Ignored Objects

To avoid excessive sizes, several object types are ignored 3 by default, e.g. built-in functions, built-in types and classes 5, function globals and module referents. However, any instances thereof and module objects will be sized when passed as given objects. Ignored object types are included unless option ignored is set accordingly.

In addition, many __...__ attributes of callable objects are ignored 3, except crucial ones, e.g. class attributes __dict__, __doc__, __name__ and __slots__. For more details, see the type-specific _..._refs() and _len_...() functions below.

Footnotes

1(1,2)

The functions and classes in this module are not thread-safe.

2(1,2,3,4)

The referents of an object are the objects referenced by that object. For example, the referents of a list are the objects held in the list, the referents of a dict are the key and value objects in the dict, etc.

3(1,2,3)

These definitions and other assumptions are rather arbitrary and may need corrections or adjustments.

4

See Python source file .../Include/longinterp.h for the C typedef of digit used in multi-precision int (or long) objects. The C sizeof(digit) in bytes can be obtained in Python from the int (or long) __itemsize__ attribute. Function leng determines the number of digits of an int (or long) object.

5

Type``s and ``class``es are considered built-in if the ``__module__ of the type or class is listed in the private _builtin_modules.

Asizer

class pympler.asizeof.Asized(size, flat, refs=(), name=None)

Stores the results of an asized object in the following 4 attributes:

size – total size of the object (including referents)

flat – flat size of the object (in bytes)

name – name or repr of the object

refs – tuple containing an Asized instance for each referent

class pympler.asizeof.Asizer(**opts)

Sizer state and options to accumulate sizes.

asized(*objs, **opts)

Size each object and return an Asized instance with size information and referents up to the given detail level (and with modified options, see method set).

If only one object is given, the return value is the Asized instance for that object. The Asized size of duplicate and ignored objects will be zero.

asizeof(*objs, **opts)

Return the combined size of the given objects (with modified options, see method set).

asizesof(*objs, **opts)

Return the individual sizes of the given objects (with modified options, see method set).

The size of duplicate and ignored objects will be zero.

exclude_refs(*objs)

Exclude any references to the specified objects from sizing.

While any references to the given objects are excluded, the objects will be sized if specified as positional arguments in subsequent calls to methods asizeof and asizesof.

exclude_types(*objs)

Exclude the specified object instances and types from sizing.

All instances and types of the given objects are excluded, even objects specified as positional arguments in subsequent calls to methods asizeof and asizesof.

print_profiles(w=0, cutoff=0, **print3options)

Print the profiles above cutoff percentage.

The available options and defaults are:

w=0 – indentation for each line

cutoff=0 – minimum percentage printed

print3options – some keyword arguments, like Python 3+ print

print_stats(objs=(), opts={}, sized=(), sizes=(), stats=3, **print3options)

Prints the statistics.

The available options and defaults are:

w=0 – indentation for each line

objs=() – optional, list of objects

opts={} – optional, dict of options used

sized=() – optional, tuple of Asized instances returned

sizes=() – optional, tuple of sizes returned

stats=3 – print stats, see function asizeof

print3options – some keyword arguments, like Python 3+ print

print_summary(w=0, objs=(), **print3options)

Print the summary statistics.

The available options and defaults are:

w=0 – indentation for each line

objs=() – optional, list of objects

print3options – some keyword arguments, like Python 3+ print

print_typedefs(w=0, **print3options)

Print the types and dict tables.

The available options and defaults are:

w=0 – indentation for each line

print3options – some keyword arguments, like Python 3+ print

reset(above=1024, align=8, clip=80, code=False, cutoff=10, derive=False, detail=0, frames=False, ignored=True, infer=False, limit=100, stats=0, stream=None, **extra)

Reset sizing options, state, etc. to defaults.

The available options and default values are:

above=0 – threshold for largest objects stats

align=8 – size alignment

code=False – incl. (byte)code size

cutoff=10 – limit large objects or profiles stats

derive=False – derive from super type

detail=0 – Asized refs level

frames=False – ignore frame objects

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0 – print statistics, see function asizeof

stream=None – output stream for printing

See function asizeof for a description of the options.

set(above=None, align=None, code=None, cutoff=None, frames=None, detail=None, limit=None, stats=None)

Set some sizing options. See also reset.

The available options are:

above – threshold for largest objects stats

align – size alignment

code – incl. (byte)code size

cutoff – limit large objects or profiles stats

detail – Asized refs level

frames – size or ignore frame objects

limit – recursion limit

stats – print statistics, see function asizeof

Any options not set remain unchanged from the previous setting.

Public Functions

pympler.asizeof.adict(*classes)

Install one or more classes to be handled as dict.

pympler.asizeof.asized(*objs, **opts)

Return a tuple containing an Asized instance for each object passed as positional argument.

The available options and defaults are:

above=0 – threshold for largest objects stats

align=8 – size alignment

code=False – incl. (byte)code size

cutoff=10 – limit large objects or profiles stats

derive=False – derive from super type

detail=0 – Asized refs level

frames=False – ignore stack frame objects

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0 – print statistics

If only one object is given, the return value is the Asized instance for that object. Otherwise, the length of the returned tuple matches the number of given objects.

The Asized size of duplicate and ignored objects will be zero.

Set detail to the desired referents level and limit to the maximum recursion depth.

See function asizeof for descriptions of the other options.

pympler.asizeof.asizeof(*objs, **opts)

Return the combined size (in bytes) of all objects passed as positional arguments.

The available options and defaults are:

above=0 – threshold for largest objects stats

align=8 – size alignment

clip=80 – clip repr() strings

code=False – incl. (byte)code size

cutoff=10 – limit large objects or profiles stats

derive=False – derive from super type

frames=False – ignore stack frame objects

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0 – print statistics

Set align to a power of 2 to align sizes. Any value less than 2 avoids size alignment.

If all is True and if no positional arguments are supplied. size all current gc objects, including module, global and stack frame objects.

A positive clip value truncates all repr() strings to at most clip characters.

The (byte)code size of callable objects like functions, methods, classes, etc. is included only if code is True.

If derive is True, new types are handled like an existing (super) type provided there is one and only of those.

By default certain base types like object, super, etc. are ignored. Set ignored to False to include those.

If infer is True, new types are inferred from attributes (only implemented for dict types on callable attributes as get, has_key, items, keys and values).

Set limit to a positive value to accumulate the sizes of the referents of each object, recursively up to the limit. Using limit=0 returns the sum of the flat sizes of the given objects. High limit values may cause runtime errors and miss objects for sizing.

A positive value for stats prints up to 9 statistics, (1) a summary of the number of objects sized and seen and a list of the largests objects with size over above bytes, (2) a simple profile of the sized objects by type and (3+) up to 6 tables showing the static, dynamic, derived, ignored, inferred and dict types used, found respectively installed. The fractional part of the stats value (x 100) is the number of largest objects shown for (stats*1.+) or the cutoff percentage for simple profiles for (*stats*=2.+). For example, *stats=1.10 shows the summary and the 10 largest objects, also the default.

See this module documentation for the definition of flat size.

pympler.asizeof.asizesof(*objs, **opts)

Return a tuple containing the size (in bytes) of all objects passed as positional arguments.

The available options and defaults are:

above=1024 – threshold for largest objects stats

align=8 – size alignment

clip=80 – clip repr() strings

code=False – incl. (byte)code size

cutoff=10 – limit large objects or profiles stats

derive=False – derive from super type

frames=False – ignore stack frame objects

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0 – print statistics

See function asizeof for a description of the options.

The length of the returned tuple equals the number of given objects.

The size of duplicate and ignored objects will be zero.

pympler.asizeof.basicsize(obj, **opts)

Return the basic size of an object (in bytes).

The available options and defaults are:

derive=False – derive type from super type

infer=False – try to infer types

save=False – save the object’s type definition if new

See this module documentation for the definition of basic size.

pympler.asizeof.flatsize(obj, align=0, **opts)

Return the flat size of an object (in bytes), optionally aligned to the given power of 2.

See function basicsize for a description of other available options.

See this module documentation for the definition of flat size.

pympler.asizeof.itemsize(obj, **opts)

Return the item size of an object (in bytes).

See function basicsize for a description of the available options.

See this module documentation for the definition of item size.

pympler.asizeof.leng(obj, **opts)

Return the length of an object (in items).

See function basicsize for a description of the available options.

pympler.asizeof.refs(obj, **opts)

Return (a generator for) specific referents of an object.

See function basicsize for a description of the available options.