Question about ground truth boxes which are used in training and PrototypicalCalibrationBlock

[leesb7426]

I found some errors in your code.
In meta_voc.py, when we got more objects of a class than a shot(K), the K samples of the class are randomly selected among objects.
However, the seed for sampling is not fixed.
As a result, the sampled objects for training and the sampled objects for PCB may not be the same.

When I set a seed to extract the same samples, the performances were dramatically degraded in 2-shot and 3-shot.
Avg performance:
Performances of VOC(split-1) with original code.
| AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
--> 1-shot
| 30.14 | 51.97 | 31.07 | 30.14 | 51.97 | 31.07 |
--> 2-shot
| 31.80 | 53.00 | 34.33 | 31.80 | 53.00 | 34.33 |
--> 3-shot
| 36.79 | 59.16 | 39.64 | 36.79 | 59.16 | 39.64 |
--> 5-shot
| 36.85 | 61.53 | 40.12 | 36.85 | 61.53 | 40.12 |
--> 10-shot
| 37.77 | 59.47 | 41.40 | 37.77 | 59.47 | 41.40 |

Performances of VOC(split-1) with fixing seed for sampling.
| AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
--> 1-shot
| 30.13 | 53.46 | 29.86 | 30.13 | 53.46 | 29.86 |
--> 2-shot
| 29.09 | 49.41 | 31.12 | 29.09 | 49.41 | 31.12 |
--> 3-shot
| 30.76 | 50.77 | 33.00 | 30.76 | 50.77 | 33.00 |
--> 5-shot
| 35.87 | 60.16 | 38.98 | 35.87 | 60.16 | 38.98 |
--> 10-shot
| 38.04 | 60.19 | 41.74 | 38.04 | 60.19 | 41.74 |

All performances:
Performances of VOC(split-1) with original code.
--> 1-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 30.21 | 53.89 | 30.09 | 30.21 | 53.89 | 30.09 |
\n| 1 | 29.97 | 50.93 | 31.75 | 29.97 | 50.93 | 31.75 |
\n| 2 | 29.98 | 52.97 | 30.34 | 29.98 | 52.97 | 30.34 |
\n| 3 | 29.69 | 50.40 | 30.14 | 29.69 | 50.40 | 30.14 |
\n| 4 | 30.84 | 53.84 | 31.65 | 30.84 | 53.84 | 31.65 |
\n| 5 | 30.22 | 51.04 | 32.58 | 30.22 | 51.04 | 32.58 |
\n| 6 | 28.80 | 49.20 | 29.87 | 28.80 | 49.20 | 29.87 |
\n| 7 | 30.62 | 52.85 | 31.61 | 30.62 | 52.85 | 31.61 |
\n| 8 | 31.46 | 54.46 | 31.72 | 31.46 | 54.46 | 31.72 |
\n| 9 | 29.58 | 50.07 | 30.96 | 29.58 | 50.07 | 30.96 |
\n| \xce\xbc | 30.14 | 51.97 | 31.07 | 30.14 | 51.97 | 31.07 |
\n| c | 0.43 | 1.09 | 0.54 | 0.43 | 1.09 | 0.54 |'

--> 2-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 31.00 | 50.91 | 34.27 | 31.00 | 50.91 | 34.27 |
\n| 1 | 33.64 | 54.77 | 36.80 | 33.64 | 54.77 | 36.80 |
\n| 2 | 29.67 | 51.01 | 30.59 | 29.67 | 51.01 | 30.59 |
\n| 3 | 33.10 | 55.59 | 34.85 | 33.10 | 55.59 | 34.85 |
\n| 4 | 29.30 | 50.05 | 31.24 | 29.30 | 50.05 | 31.24 |
\n| 5 | 33.53 | 54.35 | 37.70 | 33.53 | 54.35 | 37.70 |
\n| 6 | 33.85 | 56.59 | 36.54 | 33.85 | 56.59 | 36.54 |
\n| 7 | 31.73 | 52.02 | 34.55 | 31.73 | 52.02 | 34.55 |
\n| 8 | 30.56 | 51.27 | 32.35 | 30.56 | 51.27 | 32.35 |
\n| 9 | 31.61 | 53.47 | 34.46 | 31.61 | 53.47 | 34.46 |
\n| \xce\xbc | 31.80 | 53.00 | 34.33 | 31.80 | 53.00 | 34.33 |
\n| c | 0.99 | 1.33 | 1.39 | 0.99 | 1.33 | 1.39 |'

--> 3-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 36.01 | 57.80 | 39.41 | 36.01 | 57.80 | 39.41 |
\n| 1 | 36.86 | 59.94 | 39.32 | 36.86 | 59.94 | 39.32 |
\n| 2 | 37.61 | 60.41 | 40.12 | 37.61 | 60.41 | 40.12 |
\n| 3 | 36.32 | 58.89 | 38.80 | 36.32 | 58.89 | 38.80 |
\n| 4 | 35.89 | 57.38 | 39.11 | 35.89 | 57.38 | 39.11 |
\n| 5 | 37.37 | 59.26 | 40.41 | 37.37 | 59.26 | 40.41 |
\n| 6 | 36.53 | 58.73 | 38.78 | 36.53 | 58.73 | 38.78 |
\n| 7 | 35.94 | 58.29 | 39.30 | 35.94 | 58.29 | 39.30 |
\n| 8 | 37.95 | 60.48 | 40.80 | 37.95 | 60.48 | 40.80 |
\n| 9 | 37.39 | 60.38 | 40.37 | 37.39 | 60.38 | 40.37 |
\n| \xce\xbc | 36.79 | 59.16 | 39.64 | 36.79 | 59.16 | 39.64 |
\n| c | 0.44 | 0.66 | 0.43 | 0.44 | 0.66 | 0.43 |'

--> 5-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 36.25 | 60.70 | 39.63 | 36.25 | 60.70 | 39.63 |
\n| 1 | 37.34 | 61.82 | 40.60 | 37.34 | 61.82 | 40.60 |
\n| 2 | 37.53 | 62.76 | 40.59 | 37.53 | 62.76 | 40.59 |
\n| 3 | 36.82 | 61.46 | 39.94 | 36.82 | 61.46 | 39.94 |
\n| 4 | 36.15 | 60.92 | 39.71 | 36.15 | 60.92 | 39.71 |
\n| 5 | 35.80 | 59.40 | 39.25 | 35.80 | 59.40 | 39.25 |
\n| 6 | 36.34 | 60.42 | 39.54 | 36.34 | 60.42 | 39.54 |
\n| 7 | 37.33 | 62.64 | 40.73 | 37.33 | 62.64 | 40.73 |
\n| 8 | 37.63 | 62.97 | 41.25 | 37.63 | 62.97 | 41.25 |
\n| 9 | 37.33 | 62.18 | 39.96 | 37.33 | 62.18 | 39.96 |
\n| \xce\xbc | 36.85 | 61.53 | 40.12 | 36.85 | 61.53 | 40.12 |
\n| c | 0.39 | 0.68 | 0.37 | 0.39 | 0.68 | 0.37 |'

--> 10-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 38.03 | 59.83 | 42.59 | 38.03 | 59.83 | 42.59 |
\n| 1 | 35.99 | 55.93 | 40.42 | 35.99 | 55.93 | 40.42 |
\n| 2 | 37.99 | 59.70 | 41.05 | 37.99 | 59.70 | 41.05 |
\n| 3 | 37.93 | 59.53 | 41.94 | 37.93 | 59.53 | 41.94 |
\n| 4 | 37.91 | 60.19 | 41.63 | 37.91 | 60.19 | 41.63 |
\n| 5 | 37.68 | 58.66 | 41.22 | 37.68 | 58.66 | 41.22 |
\n| 6 | 37.28 | 58.61 | 40.34 | 37.28 | 58.61 | 40.34 |
\n| 7 | 38.28 | 60.35 | 40.89 | 38.28 | 60.35 | 40.89 |
\n| 8 | 38.25 | 61.55 | 41.90 | 38.25 | 61.55 | 41.90 |
\n| 9 | 38.34 | 60.39 | 41.99 | 38.34 | 60.39 | 41.99 |
\n| \xce\xbc | 37.77 | 59.47 | 41.40 | 37.77 | 59.47 | 41.40 |
\n| c | 0.41 | 0.89 | 0.43 | 0.41 | 0.89 | 0.43 |'

Performances of VOC(split-1) with fixing seed for sampling.
--> 1-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 30.15 | 53.19 | 30.08 | 30.15 | 53.19 | 30.08 |
\n| 1 | 29.85 | 52.90 | 29.37 | 29.85 | 52.90 | 29.37 |
\n| 2 | 30.52 | 54.41 | 30.51 | 30.52 | 54.41 | 30.51 |
\n| 3 | 30.09 | 52.72 | 29.73 | 30.09 | 52.72 | 29.73 |
\n| 4 | 30.55 | 53.91 | 30.96 | 30.55 | 53.91 | 30.96 |
\n| 5 | 30.16 | 53.38 | 29.83 | 30.16 | 53.38 | 29.83 |
\n| 6 | 30.21 | 53.44 | 29.77 | 30.21 | 53.44 | 29.77 |
\n| 7 | 29.89 | 53.39 | 29.30 | 29.89 | 53.39 | 29.30 |
\n| 8 | 29.80 | 52.90 | 29.53 | 29.80 | 52.90 | 29.53 |
\n| 9 | 30.11 | 54.39 | 29.55 | 30.11 | 54.39 | 29.55 |
\n| \xce\xbc | 30.13 | 53.46 | 29.86 | 30.13 | 53.46 | 29.86 |
\n| c | 0.15 | 0.35 | 0.31 | 0.15 | 0.35 | 0.31 |'

--> 2-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 28.62 | 49.23 | 31.43 | 28.62 | 49.23 | 31.43 |
\n| 1 | 28.67 | 48.65 | 30.34 | 28.67 | 48.65 | 30.34 |
\n| 2 | 29.21 | 49.89 | 30.79 | 29.21 | 49.89 | 30.79 |
\n| 3 | 28.80 | 48.55 | 31.55 | 28.80 | 48.55 | 31.55 |
\n| 4 | 29.02 | 49.07 | 31.11 | 29.02 | 49.07 | 31.11 |
\n| 5 | 29.01 | 50.02 | 30.51 | 29.01 | 50.02 | 30.51 |
\n| 6 | 29.05 | 48.80 | 30.78 | 29.05 | 48.80 | 30.78 |
\n| 7 | 29.50 | 50.18 | 31.46 | 29.50 | 50.18 | 31.46 |
\n| 8 | 29.23 | 49.88 | 31.28 | 29.23 | 49.88 | 31.28 |
\n| 9 | 29.79 | 49.81 | 31.95 | 29.79 | 49.81 | 31.95 |
\n| \xce\xbc | 29.09 | 49.41 | 31.12 | 29.09 | 49.41 | 31.12 |
\n| c | 0.21 | 0.36 | 0.30 | 0.21 | 0.36 | 0.30 |'

--> 3-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 30.79 | 50.88 | 33.44 | 30.79 | 50.88 | 33.44 |
\n| 1 | 31.04 | 50.74 | 33.55 | 31.04 | 50.74 | 33.55 |
\n| 2 | 31.12 | 51.27 | 33.69 | 31.12 | 51.27 | 33.69 |
\n| 3 | 30.59 | 51.02 | 33.14 | 30.59 | 51.02 | 33.14 |
\n| 4 | 30.30 | 50.43 | 32.69 | 30.30 | 50.43 | 32.69 |
\n| 5 | 30.71 | 50.90 | 32.63 | 30.71 | 50.90 | 32.63 |
\n| 6 | 31.22 | 50.67 | 33.35 | 31.22 | 50.67 | 33.35 |
\n| 7 | 30.88 | 50.76 | 32.11 | 30.88 | 50.76 | 32.11 |
\n| 8 | 30.80 | 50.80 | 32.76 | 30.80 | 50.80 | 32.76 |
\n| 9 | 30.16 | 50.28 | 32.61 | 30.16 | 50.28 | 32.61 |
\n| \xce\xbc | 30.76 | 50.77 | 33.00 | 30.76 | 50.77 | 33.00 |
\n| c | 0.20 | 0.16 | 0.30 | 0.20 | 0.16 | 0.30 |'

--> 5-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 35.84 | 60.28 | 39.79 | 35.84 | 60.28 | 39.79 |
\n| 1 | 35.89 | 59.92 | 38.88 | 35.89 | 59.92 | 38.88 |
\n| 2 | 35.78 | 60.51 | 39.01 | 35.78 | 60.51 | 39.01 |
\n| 3 | 35.90 | 60.07 | 39.02 | 35.90 | 60.07 | 39.02 |
\n| 4 | 35.88 | 59.74 | 37.86 | 35.88 | 59.74 | 37.86 |
\n| 5 | 35.92 | 60.21 | 39.00 | 35.92 | 60.21 | 39.00 |
\n| 6 | 35.86 | 60.27 | 39.38 | 35.86 | 60.27 | 39.38 |
\n| 7 | 35.89 | 60.11 | 38.39 | 35.89 | 60.11 | 38.39 |
\n| 8 | 35.97 | 60.27 | 39.47 | 35.97 | 60.27 | 39.47 |
\n| 9 | 35.75 | 60.21 | 39.02 | 35.75 | 60.21 | 39.02 |
\n| \xce\xbc | 35.87 | 60.16 | 38.98 | 35.87 | 60.16 | 38.98 |
\n| c | 0.04 | 0.13 | 0.32 | 0.04 | 0.13 | 0.32 |'

--> 10-shot
b'| | AP | AP50 | AP75 | nAP | nAP50 | nAP75 |
\n|:---|:------|:-------|:-------|:------|:--------|:--------|
\n| 0 | 38.03 | 60.52 | 41.19 | 38.03 | 60.52 | 41.19 |
\n| 1 | 38.24 | 59.83 | 42.42 | 38.24 | 59.83 | 42.42 |
\n| 2 | 38.10 | 59.79 | 41.78 | 38.10 | 59.79 | 41.78 |
\n| 3 | 38.29 | 60.53 | 42.64 | 38.29 | 60.53 | 42.64 |
\n| 4 | 39.06 | 60.68 | 42.44 | 39.06 | 60.68 | 42.44 |
\n| 5 | 37.65 | 60.41 | 40.93 | 37.65 | 60.41 | 40.93 |
\n| 6 | 37.30 | 59.76 | 40.94 | 37.30 | 59.76 | 40.94 |
\n| 7 | 37.56 | 59.60 | 41.43 | 37.56 | 59.60 | 41.43 |
\n| 8 | 37.50 | 60.42 | 41.32 | 37.50 | 60.42 | 41.32 |
\n| 9 | 38.70 | 60.39 | 42.31 | 38.70 | 60.39 | 42.31 |
\n| \xce\xbc | 38.04 | 60.19 | 41.74 | 38.04 | 60.19 | 41.74 |
\n| c | 0.33 | 0.23 | 0.39 | 0.33 | 0.23 | 0.39 |'

[ruee-z]

Have you successfully reproduced the results in the paper? There are some differences between the results I ran according to the original code and the results in the paper. I am not sure what happened

[leesb7426]

Did you set the fixed seed like me?

[ruee-z]

I didn't fix the seed, i just ran the code as the readme.md in the original code, as i want to reproduce the results in the paper, sorry, i haven't get your point why you need to fix the seed.

[leesb7426]

It is because the sampled objects for training and the sampled objects for PCB may not be the same.
Therefore, I guess the model can utilize more sample information than when it is given in training.