在对象实例化父类时调试内存构建:我应该如何进行？

Question

我有一个在循环中实例化的对象，这将导致内存的积累。

for l in range(10):
    
    grido = RasterModelGrid((3000, 3000), 10)
    
    # Tracking memory
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('traceback')
    
    # pick the biggest memory block
    stat = top_stats[0]
    print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
    for line in stat.traceback.format():
        print(line)
    
    del grido
    
    # Tracking memory
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('traceback')
    
    # pick the biggest memory block
    stat = top_stats[0]
    print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024))
    for line in stat.traceback.format():
        print(line)

调试输出为：

    2 memory blocks: 281156.3 KiB
  File "/home/aargenti/eclipse-workspace/landlab/test_memory_leak.py", line 13
    grido = RasterModelGrid((3000, 3000), 10)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/grid/raster.py", line 238
    DualUniformRectilinearGraph.__init__(
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/dual_structured_quad.py", line 187
    UniformRectilinearGraph.__init__(self, shape, spacing=spacing, origin=origin)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 654
    StructuredQuadGraphExtras.__init__(self, node_y_and_x, sort=sort)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 518
    links=StructuredQuadLayoutCython.nodes_at_link(self.shape),
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 88
    nodes_at_link = np.empty((n_links, 2), dtype=int)
2 memory blocks: 281156.3 KiB
  File "/home/aargenti/eclipse-workspace/landlab/test_memory_leak.py", line 13
    grido = RasterModelGrid((3000, 3000), 10)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/grid/raster.py", line 238
    DualUniformRectilinearGraph.__init__(
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/dual_structured_quad.py", line 187
    UniformRectilinearGraph.__init__(self, shape, spacing=spacing, origin=origin)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 654
    StructuredQuadGraphExtras.__init__(self, node_y_and_x, sort=sort)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 518
    links=StructuredQuadLayoutCython.nodes_at_link(self.shape),
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 88
    nodes_at_link = np.empty((n_links, 2), dtype=int)
4 memory blocks: 562312.7 KiB
  File "/home/aargenti/eclipse-workspace/landlab/test_memory_leak.py", line 13
    grido = RasterModelGrid((3000, 3000), 10)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/grid/raster.py", line 238
    DualUniformRectilinearGraph.__init__(
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/dual_structured_quad.py", line 187
    UniformRectilinearGraph.__init__(self, shape, spacing=spacing, origin=origin)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 654
    StructuredQuadGraphExtras.__init__(self, node_y_and_x, sort=sort)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 518
    links=StructuredQuadLayoutCython.nodes_at_link(self.shape),
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 88
    nodes_at_link = np.empty((n_links, 2), dtype=int)
4 memory blocks: 562312.7 KiB
  File "/home/aargenti/eclipse-workspace/landlab/test_memory_leak.py", line 13
    grido = RasterModelGrid((3000, 3000), 10)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/grid/raster.py", line 238
    DualUniformRectilinearGraph.__init__(
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/dual_structured_quad.py", line 187
    UniformRectilinearGraph.__init__(self, shape, spacing=spacing, origin=origin)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 654
    StructuredQuadGraphExtras.__init__(self, node_y_and_x, sort=sort)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 518
    links=StructuredQuadLayoutCython.nodes_at_link(self.shape),
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 88
    nodes_at_link = np.empty((n_links, 2), dtype=int)

但是，当我直接用以下方法实例化父类时：

    grido = DualUniformRectilinearGraph((3000, 3000), 10)

而不是：

    grido = RasterModelGrid((3000, 3000), 10)

我没有这个记忆问题：

2 memory blocks: 281156.3 KiB
  File "/home/aargenti/eclipse-workspace/landlab/test_memory_leak.py", line 16
    grido = DualUniformRectilinearGraph((3000, 3000), 10)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/dual_structured_quad.py", line 187
    UniformRectilinearGraph.__init__(self, shape, spacing=spacing, origin=origin)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 654
    StructuredQuadGraphExtras.__init__(self, node_y_and_x, sort=sort)
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 518
    links=StructuredQuadLayoutCython.nodes_at_link(self.shape),
  File "/home/aargenti/eclipse-workspace/landlab/landlab/graph/structured_quad/structured_quad.py", line 88
    nodes_at_link = np.empty((n_links, 2), dtype=int)
571 memory blocks: 74.2 KiB
  File "/home/aargenti/eclipse-workspace/landlab/test_memory_leak.py", line 20
    top_stats = snapshot.statistics('traceback')
  File "/home/aargenti/anaconda3/envs/landlab/lib/python3.10/tracemalloc.py", line 533
    grouped = self._group_by(key_type, cumulative)
  File "/home/aargenti/anaconda3/envs/landlab/lib/python3.10/tracemalloc.py", line 498
    traceback = Traceback(frames)
  File "/home/aargenti/anaconda3/envs/landlab/lib/python3.10/tracemalloc.py", line 193
    self._frames = tuple(reversed(frames))

初始化由以下代码进行：

class RasterModelGrid(
    DiagonalsMixIn, DualUniformRectilinearGraph, ModelGrid, RasterModelGridPlotter
):

    def __init__(
        self,
        shape,
        xy_spacing=1.0,
        xy_of_lower_left=(0.0, 0.0),
        xy_of_reference=(0.0, 0.0),
        xy_axis_name=("x", "y"),
        xy_axis_units="-",
        bc=None,
    ):
        shape = tuple(shape)
        xy_spacing = np.asfarray(np.broadcast_to(xy_spacing, 2))
        self._xy_of_lower_left = tuple(np.asfarray(xy_of_lower_left))

        if shape[0] <= 0 or shape[1] <= 0:
            raise ValueError("number of rows and columns must be positive")

        DualUniformRectilinearGraph.__init__(
            self, shape, spacing=xy_spacing[::-1], origin=self.xy_of_lower_left[::-1]
        )
        ModelGrid.__init__(
            self,
            xy_axis_name=xy_axis_name,
            xy_axis_units=xy_axis_units,
            xy_of_reference=xy_of_reference,
        )

        self._node_status = np.full(
            self.number_of_nodes, NodeStatus.CORE, dtype=np.uint8
        )
        self._node_status[self.perimeter_nodes] = NodeStatus.FIXED_VALUE

        if bc is None:
            bc = {"right": "open", "top": "open", "left": "open", "bottom": "open"}

        if "closed" in bc.values():
            self.set_closed_boundaries_at_grid_edges(*grid_edge_is_closed_from_dict(bc))

        self.looped_node_properties = {}

        # List of looped neighbor cells (all 8 neighbors) for
        # given *cell ids* can be created if requested by the user.
        self._looped_cell_neighbor_list = None

        # List of second ring looped neighbor cells (all 16 neighbors) for
        # given *cell ids* can be created if requested by the user.
        self._looped_second_ring_cell_neighbor_list_created = False

为什么在实例化子对象时会出现这种内存积累？我知道这不是一个最小的工作代码，但这不是我的代码，我也不熟悉它。如果你不知道哪里出了问题，你会建议我做什么测试，看看问题出在哪里？

Answer 1

考虑到您正在Ubuntu上运行，我建议您使用第二章。

在您的具体情况下，考虑到您正在看到内存积累，但不一定是崩溃，您应该为您的流程收集一个活动核心，如下所示：

echo 0x37 > /proc/pid-of-your-process/coredump_filter
gcore pid-of-your-process

chap core-file-just-created

当您到达chap提示符时，尝试：

describe used %ContainerPythonObject /redirectSuffix containers

它将创建一个名为core- file -name.tainers的文件。您可以在该文件中搜索RasterModelGrid，您可能会找到其中的许多，而且很可能每个文件都涉及到某种循环。给定python分配的地址，您可以从chap提示符中这样做来理解它是如何锚定的：

describe allocation address-you-just-found \
   /extend %ContainerPythonObject<- \
   /extend %PyDictKeysObject<- \
   /extend %PyDictValuesArray<- \
   /extend %PythonListItems<- \
   /skipUnfavoredReferences true \
   /commentExtensions true \
   /redirectSuffix RasterModelGridExplanation

如果您仔细查看生成的文件，这将使您能够理解为什么特定的RasterModelGridExplanation仍然在内存中，在内存中它仍然是可访问的，或者它只是涉及到一个循环，并且还没有被垃圾收集。