Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build_image_dataset.py crashes for delf training #10474

Open
avidullu opened this issue Jan 27, 2022 · 4 comments · May be fixed by #11249
Open

build_image_dataset.py crashes for delf training #10474

avidullu opened this issue Jan 27, 2022 · 4 comments · May be fixed by #11249
Assignees
Labels
models:research models that come under research directory type:bug Bug in the code

Comments

@avidullu
Copy link

Running instructions from https://github.com/tensorflow/models/blob/master/research/delf/delf/python/training/README.md#prepare-the-data-for-training without any GPUs on a Google Cloud VM encounters an error.

Below is the command with the error
python3 build_image_dataset.py --train_csv_path=$LANDMARK_DATA/train/train.csv --train_clean_csv_path=$LANDMARK_DATA/train/train_clean.csv --train_directory=$LANDMARK_DATA/train//// --output_directory=$LANDMARK_DATA/tfrecord/ --num_shards=128 --generate_train_validation_splits --validation_split_size=0.2 --test_csv_path=$LANDMARK_DATA/train/test.csv --test_directory=$LANDMARK_DATA/test////
2022-01-27 22:07:03.277440: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-01-27 22:07:03.277495: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-01-27 22:07:05.252430: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2022-01-27 22:07:05.252502: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-01-27 22:07:05.252525: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (gcsfuse-experiment): /proc/driver/nvidia/version does not exist
/home/avidullu/mldata/cvdfoundation/google-landmark/train/train_clean.csv
Traceback (most recent call last):
File "build_image_dataset.py", line 491, in
app.run(main)
File "/home/avidullu/.local/lib/python3.7/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/home/avidullu/.local/lib/python3.7/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "build_image_dataset.py", line 485, in main
FLAGS.seed)
File "build_image_dataset.py", line 439, in _build_train_tfrecord_dataset
image_dir)
File "build_image_dataset.py", line 144, in _get_clean_train_image_files_and_labels
df = pd.read_csv(csv_file)
File "/home/avidullu/.local/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/home/avidullu/.local/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/avidullu/.local/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 482, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home/avidullu/.local/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 811, in init
self._engine = self._make_engine(self.engine)
File "/home/avidullu/.local/lib/python3.7/site-packages/pandas/io/parsers/readers.py", line 1040, in _make_engine
return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
File "/home/avidullu/.local/lib/python3.7/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 51, in init
self._open_handles(src, kwds)
File "/home/avidullu/.local/lib/python3.7/site-packages/pandas/io/parsers/base_parser.py", line 229, in _open_handles
errors=kwds.get("encoding_errors", "strict"),
File "/home/avidullu/.local/lib/python3.7/site-packages/pandas/io/common.py", line 724, in get_handle
newline="",
AttributeError: 'GFile' object has no attribute 'readable'

with tf.io.gfile.GFile(csv_path, 'rb') as csv_file:
seems to be using a binary mode for read. On removing the 'b' from here and L116 the script makes progress.

@tensorflowbutler tensorflowbutler added the stat:awaiting response Waiting on input from the contributor label Jan 29, 2022
@tensorflowbutler
Copy link
Member

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

@avidullu
Copy link
Author

Responding inline

Q. What is the top-level directory of the model you are using
A. models/research/delf

Q. Have I written custom code?
A. No. Ran the commands as mentioned in research/delf/delf/python/training/README.md

Q. OS Platform and Distribution
A. GCP VM with Debian image. 32 CPU and 128GB RAM

Q. TensorFlow installed from?
A. From the script mentioned in /research/delf/INSTALL_INSTRUCTIONS.md

Q. TensorFlow version
A. 2.7

Q. Bazel version
A. 5.0

Q. CUDA/chDNN version
A. NA (no GPU on hardware)

Q. GPU model
A. NA (no GPU on hardware)

Q Exact command to reproduce
A. python3 build_image_dataset.py --train_csv_path=$LANDMARK_DATA/train/train.csv --train_clean_csv_path=$LANDMARK_DATA/train/train_clean.csv --train_directory=$LANDMARK_DATA/train/ --output_directory=$LANDMARK_DATA/tfrecord/ --num_shards=128 --generate_train_validation_splits --validation_split_size=0.2 --test_csv_path=$LANDMARK_DATA/train/test.csv --test_directory=$LANDMARK_DATA/test/

@kumariko kumariko removed the stat:awaiting response Waiting on input from the contributor label Jan 31, 2022
@kumariko kumariko self-assigned this Jan 31, 2022
@kumariko kumariko added models:research models that come under research directory type:bug Bug in the code labels Feb 1, 2022
@kumariko kumariko assigned andrefaraujo and unassigned kumariko Feb 2, 2022
@andrefaraujo
Copy link
Member

Thanks for reporting this!

@dan-anghel , I think you wrote this part of the code. Do you remember if the 'b' is really necessary when reading the CSV? What @avidullu reported makes sense to me, although I remember you had run the code several times, which seems to contradict it.

@khatchad
Copy link

khatchad commented Aug 7, 2024

@andrefaraujo

Thanks for reporting this!

@dan-anghel , I think you wrote this part of the code. Do you remember if the 'b' is really necessary when reading the CSV?

The CSV shouldn't be binary. Removing b solves the problem for me.

khatchad added a commit to ponder-lab/models that referenced this issue Aug 7, 2024
khatchad added a commit to ponder-lab/models that referenced this issue Aug 7, 2024
CSVs should be text files.
@khatchad khatchad linked a pull request Aug 7, 2024 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
models:research models that come under research directory type:bug Bug in the code
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants