Possible memory leaks in Python federated programs

[petervdonovan]

In the following program:

target Python{
    timeout: 10 sec,
    logging: DEBUG
};


preamble {=
    import time
    import tracemalloc
    tracemalloc.start()
=}

reactor ClientReactor {
    input in_parameter
    output out_parameter
    
    reaction(startup) {=
     =}
     
     reaction(in_parameter) -> out_parameter{=
        time.sleep(2)
        param_temp = in_parameter.value
        print(len(param_temp))
        print("RECEIVED-----------------------------------------------------------------------------------------------")
        out_parameter.set(param_temp)
    =}
}

reactor ServerReactor {
    input in_parameter
    output out_parameter
    state large_param
    
    reaction(startup)-> out_parameter {=
        temp = [1.00005]*10
        self.large_param = temp * 100000
        out_parameter.set(self.large_param)
     =}
     
     reaction(in_parameter) -> out_parameter{=
        snapshot = tracemalloc.take_snapshot()
        top_stats = snapshot.statistics("lineno")
        
        print("[ Top 10 ]")
        for stat in top_stats[:10]:
            print(stat)
            
        time.sleep(2)
        param_temp = in_parameter.value
        out_parameter.set(param_temp)
    =}
}

federated reactor{
    client = new ClientReactor()
    server = new ServerReactor()
    server.out_parameter -> client.in_parameter after 0
    client.out_parameter -> server.in_parameter
}

The reported memory usage in the file <unknown> (which we believe to correspond to the pickle library) rises from around 30 MiB to around 60 MiB, but we expect it to stay around 30.8 MiB. The number of objects associated with <unknown> is expected to stay around 1000000, but it rises to 2000000. For these reasons, we are concerned that one object created in the beginning is never freed.

We also observe a slow increase (several objects per iteration) in the number of objects associated with <unknown>, and we are not sure why.

This is related to the issue addressed in #1873, but based on my most recent discussion with Jacky it seems that it is separate. That PR seems to indicate that we have been leaking large PyObjects that are allocated in a reaction written in Python; this problem apparently is fixed by the PR, as indicated by the fact that the MRE shown above has unbounded memory usage (hundreds of megabytes, and still growing when I killed it) in master at the time of this writing.