Sizing individual objects¶
Introduction¶
This module exposes 9 functions and 2 classes to obtain lengths and sizes of Python objects (for Python 2.6 or later).
Earlier versions of this module supported Python versions down to Python 2.2. If you are using Python 2.5 or older, please consider downgrading Pympler to version 0.3.x.
Public Functions [1]
Function asizeof calculates the combined (approximate) size in bytes of one or several Python objects.
Function asizesof returns a tuple containing the (approximate) size in bytes for each given Python object separately.
Function asized returns for each object an instance of class Asized containing all the size information of the object and a tuple with the referents [2].
Functions basicsize and itemsize return the basic- respectively itemsize of the given object, both in bytes. For objects as
array.array
,numpy.array
,numpy.matrix
, etc. where the item size varies depending on the instance-specific data type, function itemsize returns that item size.Function flatsize returns the flat size of a Python object in bytes defined as the basic size plus the item size times the length of the given object.
Function leng returns the length of an object, like standard function
len
but extended for several types. E.g. the leng of a multi-precision int (or long) is the number ofdigits
[4]. The length of most mutable sequence objects includes an estimate of the over-allocation and therefore, the leng value may differ from the standardlen
result. For objects likearray.array
,numpy.array
,numpy.matrix
, etc. function leng returns the proper number of items.Function refs returns (a generator for) the referents [2] of the given object.
Certain classes are known to be sub-classes of or to behave as
dict
objects. Function adict can be used to register other class objects to be treated likedict
.
Public Classes [1]
Class Asizer may be used to accumulate the results of several asizeof or asizesof calls. After creating an Asizer instance, use methods asizeof and asizesof as needed to size any number of additional objects.
Call methods exclude_refs and/or exclude_types to exclude references to respectively instances or types of certain objects.
Use one of the print_… methods to report the statistics.
An instance of class Asized is returned for each object sized by the asized function or method.
Duplicate Objects
Any duplicate, given objects are sized only once and the size is included in the accumulated total only once. But functions asizesof and asized will return a size value respectively an Asized instance for each given object, including duplicates.
Definitions [3]
The length of an objects like
dict
,list
,set
,str
,tuple
, etc. is defined as the number of items held in or allocated by the object. Held items are references to other objects, called the referents.The size of an object is defined as the sum of the flat size of the object plus the sizes of any referents [2]. Referents are visited recursively up to the specified detail level. However, the size of objects referenced multiple times is included only once in the total size.
The flat size of an object is defined as the basic size of the object plus the item size times the number of allocated items, references to referents. The flat size does include the size for the references to the referents, but not the size of the referents themselves.
The flat size returned by function flatsize equals the result of function asizeof with options code=True, ignored=False, limit=0 and option align set to the same value.
The accurate flat size for an object is obtained from function
sys.getsizeof()
where available. Otherwise, the length and size of sequence objects asdicts
,lists
,sets
, etc. is based on an estimate for the number of allocated items. As a result, the reported length and size may differ substantially from the actual length and size.The basic and item size are obtained from the
__basicsize__
respectively__itemsize__
attributes of the (type of the) object. Where necessary (e.g. sequence objects), a zero__itemsize__
is replaced by the size of a corresponding C type.The overhead for Python’s garbage collector (GC) is included in the basic size of (GC managed) objects as well as the space needed for
refcounts
(used only in certain Python builds).Optionally, size values can be aligned to any power-of-2 multiple.
Size of (byte)code
The (byte)code size of objects like classes, functions, methods, modules, etc. can be included by setting option code=True.
Iterators are handled like sequences: iterated object(s) are sized like referents [2], but only up to the specified level or recursion limit (and only if function
gc.get_referents()
returns the referent object of iterators).Generators are sized as (byte)code only, but the generated objects are never sized.
Old- and New-style Classes
All old- and new-style
class
, instance andtype
objects are handled uniformly such that (a) instance objects are distinguished from class objects and (b) instances of different old-style classes can be dealt with separately.Class and type objects are represented as
<class ....* def>
respectively<type ... def>
where the*
indicates an old-style class and the... def
suffix marks the definition object. Instances of classes are shown as<class module.name*>
without the... def
suffix. The*
after the name indicates an instance of an old-style class.
Ignored Objects
To avoid excessive sizes, several object types are ignored [3] by default, e.g. built-in functions, built-in types and classes [5], function globals and module referents. However, any instances thereof and module objects will be sized when passed as given objects. Ignored object types are included unless option ignored is set accordingly.
In addition, many
__...__
attributes of callable objects are ignored [3], except crucial ones, e.g. class attributes__dict__
,__doc__
,__name__
and__slots__
. For more details, see the type-specific_..._refs()
and_len_...()
functions below.
Footnotes
[1] | (1, 2) The functions and classes in this module are not thread-safe. |
[2] | (1, 2, 3, 4) The referents of an object are the objects referenced by
that object. For example, the referents of a list are the
objects held in the list , the referents of a dict are
the key and value objects in the dict , etc. |
[3] | (1, 2, 3) These definitions and other assumptions are rather arbitrary and may need corrections or adjustments. |
[4] | See Python source file .../Include/longinterp.h for the
C typedef of digit used in multi-precision int (or long)
objects. The C sizeof(digit) in bytes can be obtained in
Python from the int (or long) __itemsize__ attribute.
Function leng determines the number of digits of an int
(or long) object. |
[5] | Type``s and ``class``es are considered built-in if the
``__module__ of the type or class is listed in the private
_builtin_modules . |
Asizer¶
-
class
pympler.asizeof.
Asized
(size, flat, refs=(), name=None) Stores the results of an asized object in the following 4 attributes:
size – total size of the object (including referents)
flat – flat size of the object (in bytes)
name – name or
repr
of the objectrefs – tuple containing an Asized instance for each referent
-
class
pympler.asizeof.
Asizer
(**opts) Sizer state and options to accumulate sizes.
-
asized
(*objs, **opts) Size each object and return an Asized instance with size information and referents up to the given detail level (and with modified options, see method set).
If only one object is given, the return value is the Asized instance for that object. The Asized size of duplicate and ignored objects will be zero.
-
asizeof
(*objs, **opts) Return the combined size of the given objects (with modified options, see method set).
-
asizesof
(*objs, **opts) Return the individual sizes of the given objects (with modified options, see method set).
The size of duplicate and ignored objects will be zero.
-
exclude_refs
(*objs) Exclude any references to the specified objects from sizing.
While any references to the given objects are excluded, the objects will be sized if specified as positional arguments in subsequent calls to methods asizeof and asizesof.
-
exclude_types
(*objs) Exclude the specified object instances and types from sizing.
All instances and types of the given objects are excluded, even objects specified as positional arguments in subsequent calls to methods asizeof and asizesof.
-
print_profiles
(w=0, cutoff=0, **print3options) Print the profiles above cutoff percentage.
The available options and defaults are:
w=0 – indentation for each line
cutoff=0 – minimum percentage printed
print3options – some keyword arguments, like Python 3+ print
-
print_stats
(objs=(), opts={}, sized=(), sizes=(), stats=3, **print3options) Prints the statistics.
The available options and defaults are:
w=0 – indentation for each line
objs=() – optional, list of objects
opts={} – optional, dict of options used
sized=() – optional, tuple of Asized instances returned
sizes=() – optional, tuple of sizes returned
stats=3 – print stats, see function asizeof
print3options – some keyword arguments, like Python 3+ print
-
print_summary
(w=0, objs=(), **print3options) Print the summary statistics.
The available options and defaults are:
w=0 – indentation for each line
objs=() – optional, list of objects
print3options – some keyword arguments, like Python 3+ print
-
print_typedefs
(w=0, **print3options) Print the types and dict tables.
The available options and defaults are:
w=0 – indentation for each line
print3options – some keyword arguments, like Python 3+ print
-
reset
(above=1024, align=8, clip=80, code=False, cutoff=10, derive=False, detail=0, frames=False, ignored=True, infer=False, limit=100, stats=0, stream=None, **extra) Reset sizing options, state, etc. to defaults.
The available options and default values are:
above=0 – threshold for largest objects stats
align=8 – size alignment
code=False – incl. (byte)code size
cutoff=10 – limit large objects or profiles stats
derive=False – derive from super type
detail=0 – Asized refs level
frames=False – ignore frame objects
ignored=True – ignore certain types
infer=False – try to infer types
limit=100 – recursion limit
stats=0 – print statistics, see function asizeof
stream=None – output stream for printing
See function asizeof for a description of the options.
-
set
(above=None, align=None, code=None, cutoff=None, frames=None, detail=None, limit=None, stats=None) Set some sizing options. See also reset.
The available options are:
above – threshold for largest objects stats
align – size alignment
code – incl. (byte)code size
cutoff – limit large objects or profiles stats
detail – Asized refs level
frames – size or ignore frame objects
limit – recursion limit
stats – print statistics, see function asizeof
Any options not set remain unchanged from the previous setting.
-
Public Functions¶
-
pympler.asizeof.
adict
(*classes) Install one or more classes to be handled as dict.
-
pympler.asizeof.
asized
(*objs, **opts) Return a tuple containing an Asized instance for each object passed as positional argument.
The available options and defaults are:
above=0 – threshold for largest objects stats
align=8 – size alignment
code=False – incl. (byte)code size
cutoff=10 – limit large objects or profiles stats
derive=False – derive from super type
detail=0 – Asized refs level
frames=False – ignore stack frame objects
ignored=True – ignore certain types
infer=False – try to infer types
limit=100 – recursion limit
stats=0 – print statistics
If only one object is given, the return value is the Asized instance for that object. Otherwise, the length of the returned tuple matches the number of given objects.
The Asized size of duplicate and ignored objects will be zero.
Set detail to the desired referents level and limit to the maximum recursion depth.
See function asizeof for descriptions of the other options.
-
pympler.asizeof.
asizeof
(*objs, **opts) Return the combined size (in bytes) of all objects passed as positional arguments.
The available options and defaults are:
above=0 – threshold for largest objects stats
align=8 – size alignment
clip=80 – clip
repr()
stringscode=False – incl. (byte)code size
cutoff=10 – limit large objects or profiles stats
derive=False – derive from super type
frames=False – ignore stack frame objects
ignored=True – ignore certain types
infer=False – try to infer types
limit=100 – recursion limit
stats=0 – print statistics
Set align to a power of 2 to align sizes. Any value less than 2 avoids size alignment.
If all is True and if no positional arguments are supplied. size all current gc objects, including module, global and stack frame objects.
A positive clip value truncates all repr() strings to at most clip characters.
The (byte)code size of callable objects like functions, methods, classes, etc. is included only if code is True.
If derive is True, new types are handled like an existing (super) type provided there is one and only of those.
By default certain base types like object, super, etc. are ignored. Set ignored to False to include those.
If infer is True, new types are inferred from attributes (only implemented for dict types on callable attributes as get, has_key, items, keys and values).
Set limit to a positive value to accumulate the sizes of the referents of each object, recursively up to the limit. Using limit=0 returns the sum of the flat sizes of the given objects. High limit values may cause runtime errors and miss objects for sizing.
A positive value for stats prints up to 9 statistics, (1) a summary of the number of objects sized and seen and a list of the largests objects with size over above bytes, (2) a simple profile of the sized objects by type and (3+) up to 6 tables showing the static, dynamic, derived, ignored, inferred and dict types used, found respectively installed. The fractional part of the stats value (x 100) is the number of largest objects shown for (stats*1.+) or the cutoff percentage for simple profiles for (*stats*=2.+). For example, *stats=1.10 shows the summary and the 10 largest objects, also the default.
See this module documentation for the definition of flat size.
-
pympler.asizeof.
asizesof
(*objs, **opts) Return a tuple containing the size (in bytes) of all objects passed as positional arguments.
The available options and defaults are:
above=1024 – threshold for largest objects stats
align=8 – size alignment
clip=80 – clip
repr()
stringscode=False – incl. (byte)code size
cutoff=10 – limit large objects or profiles stats
derive=False – derive from super type
frames=False – ignore stack frame objects
ignored=True – ignore certain types
infer=False – try to infer types
limit=100 – recursion limit
stats=0 – print statistics
See function asizeof for a description of the options.
The length of the returned tuple equals the number of given objects.
The size of duplicate and ignored objects will be zero.
-
pympler.asizeof.
basicsize
(obj, **opts) Return the basic size of an object (in bytes).
The available options and defaults are:
derive=False – derive type from super type
infer=False – try to infer types
save=False – save the object’s type definition if new
See this module documentation for the definition of basic size.
-
pympler.asizeof.
flatsize
(obj, align=0, **opts) Return the flat size of an object (in bytes), optionally aligned to the given power of 2.
See function basicsize for a description of other available options.
See this module documentation for the definition of flat size.
-
pympler.asizeof.
itemsize
(obj, **opts) Return the item size of an object (in bytes).
See function basicsize for a description of the available options.
See this module documentation for the definition of item size.
-
pympler.asizeof.
leng
(obj, **opts) Return the length of an object (in items).
See function basicsize for a description of the available options.
-
pympler.asizeof.
refs
(obj, **opts) Return (a generator for) specific referents of an object.
See function basicsize for a description of the available options.