Does an improved default policy for NUM_CORES in detailed routing exist?

[oharboe]

Some rough numbers and takeaways:

First: virtual memory size is the total amount of memory allocated by the process. Resident memory size, measured by make elapsed, is the amount of physical memory in use. Measuring virtual memory size can not be done by the "time" command.

The current default policy of using all CPU cores, 48 in my case, vs. 6 below, doesn't have a material difference in memory usage (virtual memory size). I don't have accurate numbers from that, just what I recall from having observed detailed routing in the past on megaboom. In both cases, max resident set is near physical memory(90%+).

The speed doesn't improve, nor deteriorate above 24 cores(50%).

Since I don't care if I use virtual memory, I have to set it up anyway, then, based on the measurement below, it would seem that there is no improved policy that exists for megaboom, for the observable parameters I care about.

It is surprising to me that memory usage in detailed routing isn't a function of number of cores. I don't know why, but I always thought so, to the point where I fooled myself to believing I heard it somewhere. Am I missing something?

Numbers from "make elapsed", so seconds, then megabytes resident memory set.

NUM_CORES=48

5_3_route                                17246          60673
NUM_CORES=24
5_3_route                                17516          61012
NUM_CORES=12
5_3_route                                28831          60868
NUM_CORES=6

Still it uses more than 64gByte of virtual memory, runs for at least 40000 seconds(didn't finish yet, but I finished my investigations here).

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
2368797 oyvind    20   0   76,0g  56,6g   1920 R 270,0  90,4     63,22 openroad 

[maliberty]

Not speeding up over 24 cpu suggests hitting the limit due to serial time per Amdahl.

I expect most of the memory is in the netlist and segs/vias not the local per thread routing structures. In any case the RSS can't grow beyond the available memory so any excess would be in the virtual size which isn't measured here.

[oharboe]

Look at the bottom of my post, I observed virtual mem via "top", it didnt increase/change noticably with number of threads.

[maliberty]

Confirming my expectation.

[maliberty]

OR could print vm size as well as rss size if that is useful. The data is available in /proc's statm (or stat or pid).

[oharboe]

from "make elapsed", which relies on the "time" command, which reports only maximum resident memory size, not virtual memory size.

[maliberty]

I'm not sure how this response relates to my suggestion about reporting vm size along with rss. Are you requesting something different related to elapsed time measurement for the full tool run?

[oharboe]

I will try to clarify.

Today "time" in ORFS does the logging that goes into "make elapsed". If you are proposing to modify openroad so that it can replace "time" in ORFS, then that is great.

Yes; both maximum resident memory set size and maximum virtual memory size would be nice to have openroad report. Then openroad should replace "time" in ORFS and finally reported in "make elapsed"

🤞 that is a bit clearer....

[maliberty]

If you are proposing to modify openroad so that it can replace "time" in ORFS, then that is great.

I didn't propose that but we can discuss it. I don't see it replacing the need for 'make elapsed' to parse the logs from multiple runs but it could replace the need for time.

[oharboe]

Yes; the only reason I can see to modify openroad at this point would be to have a place to implement "time"'s missing feature: report maximum virtual memory size.

Though... even if this improves reporting in "make elapsed", from my observations there is no materially improved policy on number of threads to use that would immediately follow from this.

I think ultimately more detailed and accurate reporting would be a good step towards improving performance, but it is not a quick fix.

[oharboe]

Just one more datapoint. After detailed route completed with NUM_CORES=48, this is reported by "top":

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                
2424810 oyvind    20   0   80,3g  57,8g   9600 D  57,8  92,2     7d+2h openroad      

From above, with NUM_CORES=6, essentially no difference.

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
2368797 oyvind    20   0   76,0g  56,6g   1920 R 270,0  90,4     63,22 openroad 

[rovinski]

This is interesting, I don't think it used to be this way. I will stop recommending reducing threads to use less memory.

[oharboe]

Indeed surprising. I recommend trusting, but verifying my finding...