Sync staging with mainline #524
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Daniel Su <[email protected]>
* Update name and icon for browser tab to rocprofiler-compute. Signed-off-by: xuchen-amd <[email protected]> * Update name and icon for browser tab to rocprofiler-compute. Signed-off-by: xuchen-amd <[email protected]> --------- Signed-off-by: xuchen-amd <[email protected]>
* find/replace Omniperf to ROCm Compute Profiler Signed-off-by: Peter Park <[email protected]> * update name in Sphinx conf Signed-off-by: Peter Park <[email protected]> * mv what-is-omniperf.rst -> what-is-rocprof-compute.rst Signed-off-by: Peter Park <[email protected]> * update Tutorials section Signed-off-by: Peter Park <[email protected]> * add Omniperf as keyword to Conceptual section for internal search Signed-off-by: Peter Park <[email protected]> * update Reference section Signed-off-by: Peter Park <[email protected]> * black fmt conf.py Signed-off-by: Peter Park <[email protected]> * update profile mode and basic usage subsections Signed-off-by: Peter Park <[email protected]> * update how to use analyze mode subsection Signed-off-by: Peter Park <[email protected]> * update install section Signed-off-by: Peter Park <[email protected]> * fix sphinx warnings Signed-off-by: Peter Park <[email protected]> * fix cmd line examples in profile/mode.rst Signed-off-by: Peter Park <[email protected]> * update install decision tree image Signed-off-by: Peter Park <[email protected]> * fix TOC and index Signed-off-by: Peter Park <[email protected]> fix weird wording * fix cli text: deriving rocprofiler-compute metrics... Signed-off-by: Peter Park <[email protected]> * update standalone-gui.rst Signed-off-by: Peter Park <[email protected]> * restore removed doc updates from #428 Signed-off-by: Peter Park <[email protected]> * update ref to Omniperf in index.rst Signed-off-by: Peter Park <[email protected]> * fix grafana connection name to match image Signed-off-by: Peter Park <[email protected]> * update cmds in tutorials Signed-off-by: Peter Park <[email protected]> --------- Signed-off-by: Peter Park <[email protected]>
* MI300 roofline enablement in rocprofiler-compute requirements.txt - running some modules complained about numpy version too new, adding extra requirement that numpy be 1.x pmc_roof_perf.txt - adding TCC_BUBBLE_sum counter to profile soc_gfx940.py soc_gfx941.py soc_gfx942.py - remove console logs reading that roofline is temporarily disabled, uncommenting blocks that check for roofline csv and run roofline post-processing roofline_calc.py - add mi300 to supported soc - add new calculation for hbm_data for MI300 using tcc_bubble_sum, checks if counter > 0 to use - add to a few comments roofline-ubuntu-20_04-mi300-rocm6 - binary for the ubuntu systems to enable mi300 roofline calculations from rocm-amdgpu-bench Note- other distros will get roofline bins to enable mi300, but need to be further tested before putting into branch. Signed-off-by: Carrie Fallows <[email protected]> * Reformatting roofline_calc.py Signed-off-by: Carrie Fallows <[email protected]> --------- Signed-off-by: Carrie Fallows <[email protected]>
* Add pre commit hook for Python formatting Signed-off-by: coleramos425 <[email protected]> * Update formatting workflow to run on latest Python and add isort formatter Signed-off-by: coleramos425 <[email protected]> * Fix caught yaml formatting issues * Update pyproject file * Add pre-commit hook instruction to CONTRIBUTING guide * Remove target-version from black pyproject.toml * Fixed formatting errors found with black and isort Signed-off-by: David Galiffi <[email protected]> * Run hook: Whitespaces, fix end of file spaces --------- Signed-off-by: coleramos425 <[email protected]> Signed-off-by: David Galiffi <[email protected]> Co-authored-by: David Galiffi <[email protected]>
Bumps [cryptography](https://github.com/pyca/cryptography) from 43.0.0 to 43.0.1. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@43.0.0...43.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: David Galiffi <[email protected]>
Checks are failing if version too high and no need for lower version Signed-off-by: Carrie Fallows <[email protected]>
* Fix crash when loading web UI roofline for gfx942 * Fix formatting Signed-off-by: benrichard-amd <[email protected]> * Make same changs for gfx940, gfx942. Signed-off-by: benrichard-amd <[email protected]> * Fix formatting in soc_gfx940 and soc_gfx941. Signed-off-by: benrichard-amd <[email protected]> --------- Signed-off-by: benrichard-amd <[email protected]>
* Patch in missed name change for rebranding. Signed-off-by: xuchen-amd <[email protected]> * Patch in missed name change for rebranding. Signed-off-by: xuchen-amd <[email protected]> --------- Signed-off-by: xuchen-amd <[email protected]>
* Move dependabot.yml to .github/ * Bump rocm-docs-core to 1.8.5 * Bump rocm-docs-core to 1.9.0
Specify that "rocprofiler-compute" replaces / obsoletes the "omniperf" package.
Signed-off-by: Tim Gu <[email protected]>
* Roofline bins for MI300 on rhel and sles distributions Built from rocm-amdgpu-bench, tested on respective distro systems with MI300 hardware. Signed-off-by: Carrie Fallows <[email protected]> * Minor modifications removing hardcoded variables in roofline files. Signed-off-by: Carrie Fallows <[email protected]> --------- Signed-off-by: Carrie Fallows <[email protected]>
Signed-off-by: Carrie Fallows <[email protected]>
* rocprofv3 support initial commit -Can run rocprofv3 but ultimately fails. rocprofv3 says the counter capacity is exceeded and the output CSV file format is different from v1/v2. * Add rocprofv3 detection so v2 can still be used It's hacky but it'll do for now. * Add code path to convert rocprofv3 JSON output into CSV * Grab correct value for Queue ID * Use _sum suffix to sum TCC counters Previously we were specifying each channel for TCC counters. rocprofv3 does not support specifing each TCC channel, and instead will auto sum given the TCC counter name. The counter name with the _sum suffix is also supported and is also supported in v1 and v2. So we will use the TCC counter name with the _sum suffix. * Fix incorrect counter outputs when using rocprofv3 In the JSON output some counters appear multime times and must be summed to get the correct value. These summed values match the rocprofv3 output in CSV mode and also match the rocprofv2 output. * Remove duplicate Correlation_ID and Wave_Size in output * Handle json output that does not contain any dispatches Omniperf was assuming each JSON output from rocprofv3 would always contain dispatches. This is not the case. For example, in a multi-process workload where one of the processes does not dispatch any kernels. A JSON file will still be output for this process but it will not contain any dispatches. * Code cleanup * Update search path for rocprofv3 results Rocprofv3 was updated to include the hostname in the path where it outputs results. * Handle accumulate counters In v1/v2 rocprof uses the SQ_ACCUM_PREV_HIRES counter for the accumualte counters. v3 does not have this. So we need to define our own counters in counter_defs.yaml. For this we use the counter name + _ACCUM, for example SQ_INSTR_LEVEL_SMEM_ACCUM. To use rocprofv3 you will need to update counter_defs.yaml to include these new counter definitions. * Use correct GPU ID When converting JSON -> CSV we were assigning node_id to GPU_ID. Since the JSON contains non-GPU devices, the node_id for GPUs might not start at 0 as expected. This commit maps the agent ID to the appropriate GPU ID. * Parse scratch memory per work item from JSON * Support rocprofv3 CSV parsing JSON decoding is very slow for large files. Include support for parsing rocprofv3 CSV output and make that the default. CSV/JSON can be toggled via the ROCPROF_OUTPUT_FORMAT environment variable e.g. ROCPROF_OUTPUT_FORMAT=csv or ROCPROF_OUTPUT_FORMAT=json * black format after merge * format isort * change return of rocprof_cmd to try to resolve test's error * hack to pick last part of rocminfo's name * debug log of hacks * Modify test_profile_general.py ctest to include MI300 enablement. Currently failing because of explicitly excluded roofline files for the soc and autofailed asserts for roof-only tests- originally in place because roofline was not enabled on mi300 yet. Signed-off-by: Carrie Fallows <[email protected]> * black and isort formated * corrected line of copyright --------- Signed-off-by: Carrie Fallows <[email protected]> Co-authored-by: benrichard-amd <[email protected]> Co-authored-by: YANG WANG <[email protected]> Co-authored-by: Carrie Fallows <[email protected]>
* fix the error caused by ignoring the lack of counter csv file from rocprofv3 for timestamp * isort and black formated
* Propagate new chip_id logic to testing workflow Signed-off-by: coleramos425 <[email protected]> * Add a debug line to tests Signed-off-by: coleramos425 <[email protected]> * Trying to set rocprofv2 generally in CTest module Signed-off-by: coleramos425 <[email protected]> * Remove temp debugging lines from CI Signed-off-by: coleramos425 <[email protected]> * Add roofline entry for MI300 expected files in CI tests Signed-off-by: coleramos425 <[email protected]> * Make num_devices modifier global in scope Signed-off-by: coleramos425 <[email protected]> * Change kernel name in PyTest to confirm rocprofv2 bug Related to https://ontrack-internal.amd.com/browse/SWDEV-503453 Signed-off-by: coleramos425 <[email protected]> --------- Signed-off-by: coleramos425 <[email protected]>
* rocprofv3 support initial commit -Can run rocprofv3 but ultimately fails. rocprofv3 says the counter capacity is exceeded and the output CSV file format is different from v1/v2. * Add rocprofv3 detection so v2 can still be used It's hacky but it'll do for now. * Add code path to convert rocprofv3 JSON output into CSV * Grab correct value for Queue ID * Use _sum suffix to sum TCC counters Previously we were specifying each channel for TCC counters. rocprofv3 does not support specifing each TCC channel, and instead will auto sum given the TCC counter name. The counter name with the _sum suffix is also supported and is also supported in v1 and v2. So we will use the TCC counter name with the _sum suffix. * Fix incorrect counter outputs when using rocprofv3 In the JSON output some counters appear multime times and must be summed to get the correct value. These summed values match the rocprofv3 output in CSV mode and also match the rocprofv2 output. * Remove duplicate Correlation_ID and Wave_Size in output * Handle json output that does not contain any dispatches Omniperf was assuming each JSON output from rocprofv3 would always contain dispatches. This is not the case. For example, in a multi-process workload where one of the processes does not dispatch any kernels. A JSON file will still be output for this process but it will not contain any dispatches. * Code cleanup * Update search path for rocprofv3 results Rocprofv3 was updated to include the hostname in the path where it outputs results. * Handle accumulate counters In v1/v2 rocprof uses the SQ_ACCUM_PREV_HIRES counter for the accumualte counters. v3 does not have this. So we need to define our own counters in counter_defs.yaml. For this we use the counter name + _ACCUM, for example SQ_INSTR_LEVEL_SMEM_ACCUM. To use rocprofv3 you will need to update counter_defs.yaml to include these new counter definitions. * debug code * add logic code for multiplexing * minor fix * more fixes * rocprofv3 support initial commit -Can run rocprofv3 but ultimately fails. rocprofv3 says the counter capacity is exceeded and the output CSV file format is different from v1/v2. * Add rocprofv3 detection so v2 can still be used It's hacky but it'll do for now. * Add code path to convert rocprofv3 JSON output into CSV * Grab correct value for Queue ID * Use _sum suffix to sum TCC counters Previously we were specifying each channel for TCC counters. rocprofv3 does not support specifing each TCC channel, and instead will auto sum given the TCC counter name. The counter name with the _sum suffix is also supported and is also supported in v1 and v2. So we will use the TCC counter name with the _sum suffix. * Fix incorrect counter outputs when using rocprofv3 In the JSON output some counters appear multime times and must be summed to get the correct value. These summed values match the rocprofv3 output in CSV mode and also match the rocprofv2 output. * Remove duplicate Correlation_ID and Wave_Size in output * Handle json output that does not contain any dispatches Omniperf was assuming each JSON output from rocprofv3 would always contain dispatches. This is not the case. For example, in a multi-process workload where one of the processes does not dispatch any kernels. A JSON file will still be output for this process but it will not contain any dispatches. * Code cleanup * Update search path for rocprofv3 results Rocprofv3 was updated to include the hostname in the path where it outputs results. * Handle accumulate counters In v1/v2 rocprof uses the SQ_ACCUM_PREV_HIRES counter for the accumualte counters. v3 does not have this. So we need to define our own counters in counter_defs.yaml. For this we use the counter name + _ACCUM, for example SQ_INSTR_LEVEL_SMEM_ACCUM. To use rocprofv3 you will need to update counter_defs.yaml to include these new counter definitions. * count accu files as well * Use correct GPU ID When converting JSON -> CSV we were assigning node_id to GPU_ID. Since the JSON contains non-GPU devices, the node_id for GPUs might not start at 0 as expected. This commit maps the agent ID to the appropriate GPU ID. * fix error with csv file parse from json and merge during post-processing * implemented parsing of csv files from v3 output for optimization * Parse scratch memory per work item from JSON * Support rocprofv3 CSV parsing JSON decoding is very slow for large files. Include support for parsing rocprofv3 CSV output and make that the default. CSV/JSON can be toggled via the ROCPROF_OUTPUT_FORMAT environment variable e.g. ROCPROF_OUTPUT_FORMAT=csv or ROCPROF_OUTPUT_FORMAT=json * black format after merge * format isort * change return of rocprof_cmd to try to resolve test's error * hack to pick last part of rocminfo's name * debug log of hacks * Modify test_profile_general.py ctest to include MI300 enablement. Currently failing because of explicitly excluded roofline files for the soc and autofailed asserts for roof-only tests- originally in place because roofline was not enabled on mi300 yet. Signed-off-by: Carrie Fallows <[email protected]> * black and isort formated * formated by isort and black * change default rocprof's output to csv * repaired crash caused by missing csv counter file when running for timestamp * change name to spatial-multiplexing from multiplexing * make necessary modification for review * set the value of spatial_multiplexing in argument defautly to None * repair the part that blocks regular pmc files' generation --------- Signed-off-by: Carrie Fallows <[email protected]> Co-authored-by: benrichard-amd <[email protected]> Co-authored-by: fei.zheng <[email protected]> Co-authored-by: YANG WANG <[email protected]> Co-authored-by: Carrie Fallows <[email protected]>
Signed-off-by: xuchen-amd <[email protected]>
* Replacing FP32-only plot with an FP32&FP64 combo plot. Results will likely be negligible but the plot name indicates both should be graphed. Signed-off-by: Carrie Fallows <[email protected]> * Remove duplicate AI plot to clean up fp32 fp64 graph Signed-off-by: Carrie Fallows <[email protected]> --------- Signed-off-by: Carrie Fallows <[email protected]>
* Add gpu_series for roofline. * Use gpu_series in path names for roofline.
* quick fix for v2 * one more fix * revert a bit --------- Co-authored-by: ywang103-amd <[email protected]>
Bumps [rocm-docs-core](https://github.com/ROCm/rocm-docs-core) from 1.9.0 to 1.12.0. - [Release notes](https://github.com/ROCm/rocm-docs-core/releases) - [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md) - [Commits](ROCm/rocm-docs-core@v1.9.0...v1.12.0) --- updated-dependencies: - dependency-name: rocm-docs-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…kload directory path creation/search points. Affects manual testing, does not seem to affect ctests. (#513) Signed-off-by: Carrie Fallows <[email protected]>
* Improve formatting when displaying rocprof command. Signed-off-by: xuchen-amd <[email protected]> * Fix python formatting. Signed-off-by: xuchen-amd <[email protected]> * Strip unwanted characters (rocprofv1 specific) from rocprof commands. Signed-off-by: xuchen-amd <[email protected]> * Strip unwanted characters (rocprofv1 specific) from rocprof commands. Signed-off-by: xuchen-amd <[email protected]> * Save the unmodified arguments for rocprof for debug message display. Signed-off-by: xuchen-amd <[email protected]> --------- Signed-off-by: xuchen-amd <[email protected]>
rocprofv3 has a new -E option where extra counters can be passed (see accum_counters.yaml) instead of defining them in counter_defs.yaml.
* Replace occurences of os.path functions with equivalent functions from pathlib library * Remove unwanted imports of os.path and os * Add coding guidelines for using pathlib instead of os.path
Signed-off-by: coleramos425 <[email protected]>
coleramos425
requested review from
koomie,
a team,
feizheng10 and
dgaliffiAMD
as code owners
Signed-off-by: coleramos425 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Manually sync staging and mainline branches. Once this is merged, #517 will automate the process of keeping the two in sync (weekly cadence).