BrokenPipeError: [Errno 32] Broken pipe

[Thinker2016]

我根据JDet/projects/retinanet/README.md文件当中的链接https://cloud.tsinghua.edu.cn/f/f12bb566d4be43bfbdc7/下载文件ckpt_30.pkl，并复制到目录JDet/projects/retinanet/work_dirs/retinanet_gaofen/checkpoints当中，运行命令
python run_net.py --config-file=configs/retinanet_gaofen.py --task=train。由于已经存在checkpoint文件，测试过程正常，但最后打包的过程报错。如下所示：
Sat Sep 4 23:39:41 2021 Loading model parameters from work_dirs/retinanet_gaofen/checkpoints/ckpt_30.pkl
Sat Sep 4 23:39:41 2021 Start running
Sat Sep 4 23:39:41 2021 Testing...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1126/1126 [34:38<00:00, 1.85s/it]
Merge results...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 36020/36020 [18:55<00:00, 31.72it/s]
已杀死
(base) wenzhao@wenzhao-MS-7C82:~/SoftwareFactory/detection/JDet/JDet/projects/retinanet$ Process ForkPoolWorker-16:
Traceback (most recent call last):
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/pool.py", line 131, in worker
put((job, i, result))
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process ForkPoolWorker-23:
Traceback (most recent call last):
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/pool.py", line 131, in worker
put((job, i, result))
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
...
谢谢。

[Thinker2016]

好像是磁盘存储空间不足。

[Thinker2016]

好像是磁盘存储空间不足。

删掉了一些文件以后又试了一下，还是出现同样的错误，不是因为磁盘存储空间不足。

[Thinker2016]

我去掉了一些测试样本之后再次试了一下，并查看了JDet测试完成后打的zip包中的xml文件，发现给出的预测包含了大量probability低于0.1的预测框，如下所示：

<annotation>
    <source>
    <filename>0.tif</filename>
    <origin>GF2/GF3</origin>
    </source>
    <research>
        <version>4.0</version>
        <provider>placeholder_affiliation</provider>
        <author>placeholder_authorname</author>
        <!--参赛课题 -->
        <pluginname>placeholder_direction</pluginname>
        <pluginclass>placeholder_suject</pluginclass>
        <time>2020-07-2020-11</time>
    </research>
    <size>
        <width>1000</width>
        <height>1000</height>
        <depth>3</depth>
    </size>
    <!--存放目标检测信息-->
    <objects>
        <object>
            <coordinate>pixel</coordinate>
            <type>rectangle</type>
            <description>None</description>
            <possibleresult>
                <name>A330</name>                
                <probability>0.0688</probability>
            </possibleresult>
            <!--检测框坐标，首尾闭合的矩形，起始点无要求-->
            <points>  
                <point>561.6295, 1471.9349</point>
                <point>608.5075, 1492.8219</point>
                <point>602.3088, 1506.734</point>
                <point>555.4308, 1485.84</point>
                <point>561.6295, 1471.9349</point>
            </points>
        </object>
        <object>
            <coordinate>pixel</coordinate>
            <type>rectangle</type>
            <description>None</description>
            <possibleresult>
                <name>A330</name>                
                <probability>0.0571</probability>
            </possibleresult>
            <!--检测框坐标，首尾闭合的矩形，起始点无要求-->
            <points>  
                <point>565.6939, 1026.0375</point>
                <point>597.6301, 1023.0295</point>
                <point>599.4427, 1042.2744</point>
                <point>567.5065, 1045.282</point>
                <point>565.6939, 1026.0375</point>
            </points>
        </object>
        <object>
            <coordinate>pixel</coordinate>
            <type>rectangle</type>
            <description>None</description>
            <possibleresult>
                <name>A330</name>                
                <probability>0.0558</probability>
            </possibleresult>
            <!--检测框坐标，首尾闭合的矩形，起始点无要求-->
            <points>  
                <point>1016.1245, 1475.4329</point>
                <point>1048.4687, 1475.2568999999999</point>
                <point>1048.5765999999999, 1495.0795</point>
                <point>1016.2324, 1495.255</point>
                <point>1016.1245, 1475.4329</point>
            </points>
        </object>

...
咱们这个工具是如何设置目标检测的probability的阈值的？

[Thinker2016]

查出来了。在test的merge阶段，程序打开了20多个python进程，导致内存溢出。
这是什么原因？

[li-xl]

merge result的时候开启了多进程,所以会有多个python进程,不想多进程可以把https://github.com/Jittor/JDet/blob/master/python/jdet/data/devkits/result_merge.py的#283行改成mergebase

[li-xl]

probabilaty的阈值可以通过设置RetinaHead的score_threshold 来指定

[Thinker2016]

好的，谢谢你。