Python标准库tracemalloc跟踪内存分配

显示前10项¶

显示内存分配最多的10个文件：

importtracemalloc
tracemalloc.start()
# ... run your application ...
snapshot=tracemalloc.take_snapshot()
top_stats=snapshot.statistics('lineno')
print("[ Top 10 ]")
forstatintop_stats[:10]:
print(stat)

Python测试套件的输出示例：

[Top10]
<frozenimportlib._bootstrap>:716:size=4855KiB,count=39328,average=126B
<frozenimportlib._bootstrap>:284:size=521KiB,count=3199,average=167B
/usr/lib/python3.4/collections/__init__.py:368:size=244KiB,count=2315,average=108B
/usr/lib/python3.4/unittest/case.py:381:size=185KiB,count=779,average=243B
/usr/lib/python3.4/unittest/case.py:402:size=154KiB,count=378,average=416B
/usr/lib/python3.4/abc.py:133:size=88.7KiB,count=347,average=262B
<frozenimportlib._bootstrap>:1446:size=70.4KiB,count=911,average=79B
<frozenimportlib._bootstrap>:1454:size=52.0KiB,count=25,average=2131B
<string>:5:size=49.7KiB,count=148,average=344B
/usr/lib/python3.4/sysconfig.py:411:size=48.0KiB,count=1,average=48.0KiB

We can see that Python loaded 4855KiB data (bytecode and constants) from

modules and that the collections module allocated 244KiB to build

namedtuple types.

更多选项，请参见 Snapshot.statistics()

计算差异¶

获取两个快照并显示差异：

importtracemalloc
tracemalloc.start()
# ... start your application ...
snapshot1=tracemalloc.take_snapshot()
# ... call the function leaking memory ...
snapshot2=tracemalloc.take_snapshot()
top_stats=snapshot2.compare_to(snapshot1,'lineno')
print("[ Top 10 differences ]")
forstatintop_stats[:10]:
print(stat)

Example of output before/after running some tests of the Python test suite:

[Top10differences]
<frozenimportlib._bootstrap>:716:size=8173KiB(+4428KiB),count=71332(+39369),average=117B
/usr/lib/python3.4/linecache.py:127:size=940KiB(+940KiB),count=8106(+8106),average=119B
/usr/lib/python3.4/unittest/case.py:571:size=298KiB(+298KiB),count=589(+589),average=519B
<frozenimportlib._bootstrap>:284:size=1005KiB(+166KiB),count=7423(+1526),average=139B
/usr/lib/python3.4/mimetypes.py:217:size=112KiB(+112KiB),count=1334(+1334),average=86B
/usr/lib/python3.4/http/server.py:848:size=96.0KiB(+96.0KiB),count=1(+1),average=96.0KiB
/usr/lib/python3.4/inspect.py:1465:size=83.5KiB(+83.5KiB),count=109(+109),average=784B
/usr/lib/python3.4/unittest/mock.py:491:size=77.7KiB(+77.7KiB),count=143(+143),average=557B
/usr/lib/python3.4/urllib/parse.py:476:size=71.8KiB(+71.8KiB),count=969(+969),average=76B
/usr/lib/python3.4/contextlib.py:38:size=67.2KiB(+67.2KiB),count=126(+126),average=546B

We can see that Python has loaded 8173KiB of module data (bytecode and

constants), and that this is 4428KiB more than had been loaded before the

tests, when the previous snapshot was taken. Similarly, the linecache

module has cached 940KiB of Python source code to format tracebacks, all

of it since the previous snapshot.

If the system has little free memory, snapshots can be written on disk using

the Snapshot.dump() method to analyze the snapshot offline. Then use the

Snapshot.load() method reload the snapshot.

获取一个内存块的溯源¶

一段找出程序中最大内存块溯源的代码:

importtracemalloc
# Store 25 frames
tracemalloc.start(25)
# ... run your application ...
snapshot=tracemalloc.take_snapshot()
top_stats=snapshot.statistics('traceback')
# pick the biggest memory block
stat=top_stats[0]
print("%s memory blocks: %.1f KiB"%(stat.count,stat.size/1024))
forlineinstat.traceback.format():
print(line)

一段Python单元测试的输出案例(限制最大25层堆栈)

903memoryblocks:870.1KiB
File"<frozen importlib._bootstrap>",line716
File"<frozen importlib._bootstrap>",line1036
File"<frozen importlib._bootstrap>",line934
File"<frozen importlib._bootstrap>",line1068
File"<frozen importlib._bootstrap>",line619
File"<frozen importlib._bootstrap>",line1581
File"<frozen importlib._bootstrap>",line1614
File"/usr/lib/python3.4/doctest.py",line101
importpdb
File"<frozen importlib._bootstrap>",line284
File"<frozen importlib._bootstrap>",line938
File"<frozen importlib._bootstrap>",line1068
File"<frozen importlib._bootstrap>",line619
File"<frozen importlib._bootstrap>",line1581
File"<frozen importlib._bootstrap>",line1614
File"/usr/lib/python3.4/test/support/__init__.py",line1728
importdoctest
File"/usr/lib/python3.4/test/test_pickletools.py",line21
support.run_doctest(pickletools)
File"/usr/lib/python3.4/test/regrtest.py",line1276
test_runner()
File"/usr/lib/python3.4/test/regrtest.py",line976
display_failure=notverbose)
File"/usr/lib/python3.4/test/regrtest.py",line761
match_tests=ns.match_tests)
File"/usr/lib/python3.4/test/regrtest.py",line1563
main()
File"/usr/lib/python3.4/test/__main__.py",line3
regrtest.main_in_temp_cwd()
File"/usr/lib/python3.4/runpy.py",line73
exec(code,run_globals)
File"/usr/lib/python3.4/runpy.py",line160
"__main__",fname,loader,pkg_name)

We can see that the most memory was allocated in the importlib module to

load data (bytecode and constants) from modules: 870.1KiB. The traceback is

where the importlib loaded data most recently: on the importpdb

line of the doctest module. The traceback may change if a new module is

loaded.

Pretty top¶

Code to display the 10 lines allocating the most memory with a pretty output,

ignoring <frozenimportlib._bootstrap> and <unknown> files:

importlinecache
importos
importtracemalloc
defdisplay_top(snapshot,key_type='lineno',limit=10):
snapshot=snapshot.filter_traces((
tracemalloc.Filter(False,"<frozen importlib._bootstrap>"),
tracemalloc.Filter(False,"<unknown>"),
))
top_stats=snapshot.statistics(key_type)
print("Top %s lines"%limit)
forindex,statinenumerate(top_stats[:limit],1):
frame=stat.traceback[0]
print("#%s: %s:%s: %.1f KiB"
%(index,frame.filename,frame.lineno,stat.size/1024))
line=linecache.getline(frame.filename,frame.lineno).strip()
ifline:
print('    %s'%line)
other=top_stats[limit:]
ifother:
size=sum(stat.sizeforstatinother)
print("%s other: %.1f KiB"%(len(other),size/1024))
total=sum(stat.sizeforstatintop_stats)
print("Total allocated size: %.1f KiB"%(total/1024))
tracemalloc.start()
# ... run your application ...
snapshot=tracemalloc.take_snapshot()
display_top(snapshot)

Python测试套件的输出示例：

Top10lines
#1: Lib/base64.py:414: 419.8 KiB
_b85chars2=[(a+b)forain_b85charsforbin_b85chars]
#2: Lib/base64.py:306: 419.8 KiB
_a85chars2=[(a+b)forain_a85charsforbin_a85chars]
#3: collections/__init__.py:368: 293.6 KiB
exec(class_definition,namespace)
#4: Lib/abc.py:133: 115.2 KiB
cls=super().__new__(mcls,name,bases,namespace)
#5: unittest/case.py:574: 103.1 KiB
testMethod()
#6: Lib/linecache.py:127: 95.4 KiB
lines=fp.readlines()
#7: urllib/parse.py:476: 71.8 KiB
forain_hexdigforbin_hexdig}
#8: <string>:5: 62.0 KiB
#9: Lib/_weakrefset.py:37: 60.0 KiB
self.data=set()
#10: Lib/base64.py:142: 59.8 KiB
_b32tab2=[a+bforain_b32tabforbin_b32tab]
6220other:3602.8KiB
Totalallocatedsize:5303.1KiB

更多选项，请参见 Snapshot.statistics()

函数¶

tracemalloc.clear_traces()¶

Clear traces of memory blocks allocated by Python.

另见 stop().

tracemalloc.get_object_traceback(obj)¶

Get the traceback where the Python object obj was allocated.

Return a Traceback instance, or None if the tracemalloc

module is not tracing memory allocations or did not trace the allocation of

the object.

See also gc.get_referrers() and sys.getsizeof() functions.

tracemalloc.get_traceback_limit()¶

Get the maximum number of frames stored in the traceback of a trace.

The tracemalloc module must be tracing memory allocations to

get the limit, otherwise an exception is raised.

The limit is set by the start() function.

tracemalloc.get_traced_memory()¶

Get the current size and peak size of memory blocks traced by the

tracemalloc module as a tuple: (current:int,peak:int).

tracemalloc.get_tracemalloc_memory()¶

Get the memory usage in bytes of the tracemalloc module used to store

traces of memory blocks.

Return an int.

tracemalloc.is_tracing()¶

True if the tracemalloc module is tracing Python memory

allocations, False otherwise.

See also start() and stop() functions.

tracemalloc.start(nframe: int=1)¶

Start tracing Python memory allocations: install hooks on Python memory

allocators. Collected tracebacks of traces will be limited to nframe

frames. By default, a trace of a memory block only stores the most recent

frame: the limit is 1. nframe must be greater or equal to 1.

Storing more than 1 frame is only useful to compute statistics grouped

by 'traceback' or to compute cumulative statistics: see the

Snapshot.compare_to() and Snapshot.statistics() methods.

Storing more frames increases the memory and CPU overhead of the

tracemalloc module. Use the get_tracemalloc_memory() function

to measure how much memory is used by the tracemalloc module.

The PYTHONTRACEMALLOC environment variable

(PYTHONTRACEMALLOC=NFRAME) and the -Xtracemalloc=NFRAME

command line option can be used to start tracing at startup.

See also stop(), is_tracing() and get_traceback_limit()

functions.

tracemalloc.stop()¶

Stop tracing Python memory allocations: uninstall hooks on Python memory

allocators. Also clears all previously collected traces of memory blocks

allocated by Python.

Call take_snapshot() function to take a snapshot of traces before

clearing them.

See also start(), is_tracing() and clear_traces()

functions.

tracemalloc.take_snapshot()¶

Take a snapshot of traces of memory blocks allocated by Python. Return a new

Snapshot instance.

The snapshot does not include memory blocks allocated before the

tracemalloc module started to trace memory allocations.

Tracebacks of traces are limited to get_traceback_limit() frames. Use

the nframe parameter of the start() function to store more frames.

The tracemalloc module must be tracing memory allocations to take a

snapshot, see the start() function.

See also the get_object_traceback() function.

域过滤器¶

class tracemalloc.DomainFilter(inclusive: bool, domain: int)¶

Filter traces of memory blocks by their address space (domain).

3.6 新版功能.

inclusive¶

If inclusive is True (include), match memory blocks allocated

in the address space domain.

If inclusive is False (exclude), match memory blocks not allocated

in the address space domain.

domain¶

Address space of a memory block (int). Read-only property.

过滤器¶

class tracemalloc.Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False, domain: int=None)¶

对内存块的跟踪进行筛选。

See the fnmatch.fnmatch() function for the syntax of

filename_pattern. The '.pyc' file extension is

replaced with '.py'.

示例：

• Filter(True,subprocess.__file__) only includes traces of the

  subprocess module

• Filter(False,tracemalloc.__file__) excludes traces of the

  tracemalloc module

• Filter(False,"<unknown>") excludes empty tracebacks

在 3.5 版更改: The '.pyo' file extension is no longer replaced with '.py'.

在 3.6 版更改: Added the domain attribute.

domain¶

Address space of a memory block (int or None).

tracemalloc uses the domain 0 to trace memory allocations made by

Python. C extensions can use other domains to trace other resources.

inclusive¶

If inclusive is True (include), only match memory blocks allocated

in a file with a name matching filename_pattern at line number

lineno.

If inclusive is False (exclude), ignore memory blocks allocated in

a file with a name matching filename_pattern at line number

lineno.

lineno¶

Line number (int) of the filter. If lineno is None, the filter

matches any line number.

filename_pattern¶

Filename pattern of the filter (str). Read-only property.

all_frames¶

If all_frames is True, all frames of the traceback are checked. If

all_frames is False, only the most recent frame is checked.

This attribute has no effect if the traceback limit is 1. See the

get_traceback_limit() function and Snapshot.traceback_limit

attribute.

Frame¶

class tracemalloc.Frame¶

Frame of a traceback.

The Traceback class is a sequence of Frame instances.

filename¶

文件名(str).

lineno¶

行号 (int).

快照¶

class tracemalloc.Snapshot¶

Snapshot of traces of memory blocks allocated by Python.

The take_snapshot() function creates a snapshot instance.

compare_to(old_snapshot: Snapshot, key_type: str, cumulative: bool=False)¶

Compute the differences with an old snapshot. Get statistics as a sorted

list of StatisticDiff instances grouped by key_type.

See the Snapshot.statistics() method for key_type and cumulative

parameters.

The result is sorted from the biggest to the smallest by: absolute value

of StatisticDiff.size_diff, StatisticDiff.size, absolute

value of StatisticDiff.count_diff, Statistic.count and

then by StatisticDiff.traceback.

dump(filename)¶

将快照写入文件

使用 load() 重载快照。

filter_traces(filters)¶

Create a new Snapshot instance with a filtered traces

sequence, filters is a list of DomainFilter and

Filter instances. If filters is an empty list, return a new

Snapshot instance with a copy of the traces.

All inclusive filters are applied at once, a trace is ignored if no

inclusive filters match it. A trace is ignored if at least one exclusive

filter matches it.

在 3.6 版更改: DomainFilter instances are now also accepted in filters.

classmethod load(filename)¶

从文件载入快照。

另见 dump().

statistics(key_type: str, cumulative: bool=False)¶

获取 Statistic 信息列表，按 key_type 分组排序：

key_type	描述
'filename'	filename
'lineno'	文件名和行号
'traceback'	回溯

If cumulative is True, cumulate size and count of memory blocks of

all frames of the traceback of a trace, not only the most recent frame.

The cumulative mode can only be used with key_type equals to

'filename' and 'lineno'.

The result is sorted from the biggest to the smallest by:

Statistic.size, Statistic.count and then by

Statistic.traceback.

traceback_limit¶

Maximum number of frames stored in the traceback of traces:

result of the get_traceback_limit() when the snapshot was taken.

traces¶

Traces of all memory blocks allocated by Python: sequence of

Trace instances.

The sequence has an undefined order. Use the Snapshot.statistics()

method to get a sorted list of statistics.

统计¶

class tracemalloc.Statistic¶

统计内存分配

Snapshot.statistics() 返回 Statistic 实例的列表。.

参见 StatisticDiff 类。

count¶

内存块数（整形）。

size¶

Total size of memory blocks in bytes (int).

traceback¶

Traceback where the memory block was allocated, Traceback

instance.

StatisticDiff¶

class tracemalloc.StatisticDiff¶

Statistic difference on memory allocations between an old and a new

Snapshot instance.

Snapshot.compare_to() returns a list of StatisticDiff

instances. See also the Statistic class.

count¶

Number of memory blocks in the new snapshot (int): 0 if

the memory blocks have been released in the new snapshot.

count_diff¶

Difference of number of memory blocks between the old and the new

snapshots (int): 0 if the memory blocks have been allocated in

the new snapshot.

size¶

Total size of memory blocks in bytes in the new snapshot (int):

0 if the memory blocks have been released in the new snapshot.

size_diff¶

Difference of total size of memory blocks in bytes between the old and

the new snapshots (int): 0 if the memory blocks have been

allocated in the new snapshot.

traceback¶

Traceback where the memory blocks were allocated, Traceback

instance.

跟踪¶

class tracemalloc.Trace¶

Trace of a memory block.

The Snapshot.traces attribute is a sequence of Trace

instances.

在 3.6 版更改: Added the domain attribute.

domain¶

Address space of a memory block (int). Read-only property.

tracemalloc uses the domain 0 to trace memory allocations made by

Python. C extensions can use other domains to trace other resources.

size¶

Size of the memory block in bytes (int).

traceback¶

Traceback where the memory block was allocated, Traceback

instance.

回溯¶

class tracemalloc.Traceback¶

Sequence of Frame instances sorted from the oldest frame to the

most recent frame.

A traceback contains at least 1 frame. If the tracemalloc module

failed to get a frame, the filename "<unknown>" at line number 0 is

used.

When a snapshot is taken, tracebacks of traces are limited to

get_traceback_limit() frames. See the take_snapshot() function.

The Trace.traceback attribute is an instance of Traceback

instance.

在 3.7 版更改: Frames are now sorted from the oldest to the most recent, instead of most recent to oldest.

format(limit=None, most_recent_first=False)¶

Format the traceback as a list of lines with newlines. Use the

linecache module to retrieve lines from the source code.

If limit is set, format the limit most recent frames if limit

is positive. Otherwise, format the abs(limit) oldest frames.

If most_recent_first is True, the order of the formatted frames

is reversed, returning the most recent frame first instead of last.

Similar to the traceback.format_tb() function, except that

format() does not include newlines.

示例:

print("Traceback (most recent call first):")
forlineintraceback:
print(line)

输出:

Traceback(mostrecentcallfirst):
File"test.py",line9
obj=Object()
File"test.py",line12
tb=tracemalloc.get_object_traceback(f())

以上是 Python标准库tracemalloc跟踪内存分配 的全部内容， 来源链接： utcz.com/z/507844.html