[GSProcessing] Enforce re-order for node label processing during classification #1136
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thvasilo
added
ready
thvasilo
force-pushed
the
reorder-node-ids
branch
2 times, most recently
from
4d0afcd to
865d943
Compare
thvasilo
force-pushed
the
reorder-node-ids
branch
from
865d943 to
40b16f9
Compare
thvasilo
force-pushed
the
reorder-node-ids
branch
from
40b16f9 to
811f4b1
Compare
jalencato
reviewed
Jan 17, 2025
...graphstorm_processing/data_transformations/dist_transformations/dist_label_transformation.py
Show resolved
Hide resolved
graphstorm-processing/graphstorm_processing/graph_loaders/dist_heterogeneous_loader.py
Show resolved
Hide resolved
graphstorm-processing/graphstorm_processing/graph_loaders/dist_heterogeneous_loader.py
Outdated
Show resolved
Hide resolved
graphstorm-processing/graphstorm_processing/graph_loaders/dist_heterogeneous_loader.py
Show resolved
Hide resolved
jalencato
approved these changes
Jan 24, 2025
| @@ -18,6 +18,8 @@ | |||
| import logging | |||
| from typing import Any, Dict, Optional | |||
|
|
|||
| from graphstorm_processing.constants import VALID_TASK_TYPES | |||
|
|
|||
There was a problem hiding this comment.
Remove additional line.
| @@ -194,8 +197,6 @@ def create_new_relative_path_from_existing( | |||
| "path/to/parquet-repartitioned-my-suffix/part-00003-filename.snappy.parquet" | |||
| """ | |||
| original_relative_path_obj = Path(original_relative_path) | |||
| # We expect files to have a path of the form /path/to/parquet/part-00001.snappy.parquet | |||
| assert original_relative_path_obj.parts[-2] == "parquet" # TODO: Remove this assumption? | |||
There was a problem hiding this comment.
Do we meet any issue for it?
| # Generate Spark-style part filename | ||
| part_filename = os.path.join( | ||
| base_path, | ||
| f"part-{file_idx:05d}-{unique_id}.parquet", |
There was a problem hiding this comment.
So unlike pyspark, pyarrow does not support the built-in naming?
There was a problem hiding this comment.
Yes, we could try using pyarrow.ds.dataset to do the writing to provide that but that actually complicates things. This allows us to have explicit distribution of rows.
jalencato
reviewed
Jan 24, 2025
| node_int_ids = np.arange(total_data_points) | ||
|
|
||
| rng = np.random.default_rng(42) | ||
| # Create random numerical labels with values 0-9 |
There was a problem hiding this comment.
Do we cover the case for None value?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Fixes #1135
Fixes #1138
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.