'rez build --install --prefix' error #858

ClementHector · 2020-02-24T17:27:05Z

Hi there!

We encountered the following error while deploying a library of ours in a specific directory with rez on a python's docker image through Gitlab CI/CD.

We'd be grateful if you could tell us how to resolve this issue by ourselves or if you could fix it!
We might have missed some context or information in this comment; please tell us so if necessary.

Many thanks in advance.

rez build --install --prefix /mnt/rnd/PIPELINE/USER/HC
Building into: '/mnt/rnd/PIPELINE/USER/HC'

--------------------------------------------------------------------------------
Building smurfs-0.0.27...
--------------------------------------------------------------------------------
Resolving build environment: ellipsanime '~maya==2018.6' '~houdini==17.0.352' '~nuke==11.3.4' ellipsanime rez_utils-1
resolved by root@runner-c392f0d9-project-56-concurrent-0, on Mon Feb 24 17:08:30 2020, using Rez v2.40.4

requested packages:
ellipsanime         
~maya==2018.6       
~houdini==17.0.352  
~nuke==11.3.4       
ellipsanime         
rez_utils-1         
~platform==linux    (implicit)
~arch==x86_64       (implicit)
~os==debian-10      (implicit)

resolved packages:
ellipsanime-0.0.3  /mnt/rnd/PIPELINE/__PACKAGES__/project/ellipsanime/0.0.3  
python-3.7.6       /mnt/rnd/PIPELINE/__PACKAGES__/app/python/3.7.6           
rez_utils-1.0.3    /mnt/rnd/PIPELINE/__PACKAGES__/int/rez_utils/1.0.3        

Invoking custom build system...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/rez/serialise.py", line 71, in open_file_for_write
    f.write(content)
  File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/rez/vendor/atomicwrites/__init__.py", line 154, in _open
    self.commit(f)
  File "/usr/local/lib/python3.7/site-packages/rez/vendor/atomicwrites/__init__.py", line 185, in commit
    replace_atomic(f.name, self._path)
  File "/usr/local/lib/python3.7/site-packages/rez/vendor/atomicwrites/__init__.py", line 92, in replace_atomic
    return _replace_atomic(src, dst)
  File "/usr/local/lib/python3.7/site-packages/rez/vendor/atomicwrites/__init__.py", line 50, in _replace_atomic
    _sync_directory(os.path.normpath(os.path.dirname(dst)))
  File "/usr/local/lib/python3.7/site-packages/rez/vendor/atomicwrites/__init__.py", line 44, in _sync_directory
    _proper_fsync(fd)
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/rez", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.7/site-packages/rez/cli/_main.py", line 153, in run
    returncode = run_cmd()
  File "/usr/local/lib/python3.7/site-packages/rez/cli/_main.py", line 145, in run_cmd
    return opts.func(opts, opts.parser, extra_arg_groups)
  File "/usr/local/lib/python3.7/site-packages/rez/cli/build.py", line 170, in command
    variants=opts.variants)
  File "/usr/local/lib/python3.7/site-packages/rezplugins/build_process/local.py", line 41, in build
    install=install)
  File "/usr/local/lib/python3.7/site-packages/rez/build_process_.py", line 203, in visit_variants
    result = func(variant, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/rezplugins/build_process/local.py", line 280, in _build_variant
    variant.install(install_path)
  File "/usr/local/lib/python3.7/site-packages/rez/packages_.py", line 402, in install
    overrides=overrides)
  File "/usr/local/lib/python3.7/site-packages/rezplugins/package_repository/filesystem.py", line 625, in install_variant
    variant = _create_variant()
  File "/usr/local/lib/python3.7/site-packages/rezplugins/package_repository/filesystem.py", line 618, in _create_variant
    overrides=overrides
  File "/usr/local/lib/python3.7/site-packages/rezplugins/package_repository/filesystem.py", line 1077, in _create_variant
    dump_package_data(package_data, buf=f, format_=package_format)
  File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/rez/serialise.py", line 73, in open_file_for_write
    except WindowsError as e:
NameError: name 'WindowsError' is not defined
ERROR: Job failed: exit code 1

The text was updated successfully, but these errors were encountered:

nerdvegas · 2020-02-24T22:08:23Z

I haven't seen this error, have you determined that it's related to running within a container?

The second stacktrace looks like it's simply caused by a missing import for the WindowsError, I will look into that. In the meantime perhaps if you could narrow down the repro, if possible.

Cheers!
A

alexxbb · 2020-09-21T23:56:18Z

Hit a simillar problem: NameError: name 'WindowsError' is not defined

https://github.com/nerdvegas/rez/blob/master/src/rez/serialise.py#L76

nerdvegas · 2020-09-22T00:18:22Z

The first thing I'd investigate is whether a newer version of `atomicwrites` is necessary. A

…

On Tue, Sep 22, 2020 at 9:56 AM alexxbb ***@***.***> wrote: Hit a simillar problem: NameError: name 'WindowsError' is not defined https://github.com/nerdvegas/rez/blob/master/src/rez/serialise.py#L76 <http://url> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#858 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMOUSVPYAXLA4MRSP5J4JDSG7R25ANCNFSM4K2MWD2Q> .

alexxbb · 2020-09-22T05:50:58Z

Wonder why is this windows-specific WindowsError in the code in the first place? I'm getting errors on Linux with Python2.

nerdvegas · 2020-09-22T21:17:50Z

Oh, I haven't seen failures with atomiwrites on linux before, I'd assumed you were on Windows.

Would it be possible to isolate this to a small repro using atomicwrites only? I am assuming this is an edge case we've not seen before - if this were failing in the general case, I'd have gotten loads of bug reports.

alexxbb · 2020-09-23T14:13:08Z

rez-pip --install --release --python-version=2.7 python-gitlab

Traceback (most recent call last):
  File "/usr/bin/rez-pip", line 8, in <module>
    sys.exit(run_rez_pip())
  File "/opt/rez/lib/python2.7/site-packages/rez/cli/_entry_points.py", line 188, in run_rez_pip
    return run("pip")
  File "/opt/rez/lib/python2.7/site-packages/rez/cli/_main.py", line 152, in run
    returncode = run_cmd()
  File "/opt/rez/lib/python2.7/site-packages/rez/cli/_main.py", line 144, in run_cmd
    return opts.func(opts, opts.parser, extra_arg_groups)
  File "/opt/rez/lib/python2.7/site-packages/rez/cli/pip.py", line 67, in command
    extra_args=opts.extra)
  File "/opt/rez/lib/python2.7/site-packages/rez/pip.py", line 404, in pip_install_package
    pkg.is_pure_python = metadata["is_pure_python"]
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/opt/rez/lib/python2.7/site-packages/rez/package_maker.py", line 217, in make_package
    variant_ = variant.install(path)
  File "/opt/rez/lib/python2.7/site-packages/rez/packages.py", line 413, in install
    overrides=overrides)
  File "/opt/rez/lib/python2.7/site-packages/rezplugins/package_repository/filesystem.py", line 655, in install_variant
    variant = _create_variant()
  File "/opt/rez/lib/python2.7/site-packages/rezplugins/package_repository/filesystem.py", line 648, in _create_variant
    overrides=overrides
  File "/opt/rez/lib/python2.7/site-packages/rezplugins/package_repository/filesystem.py", line 1114, in _create_variant
    dump_package_data(package_data, buf=f, format_=package_format)
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/opt/rez/lib/python2.7/site-packages/rez/serialise.py", line 74, in open_file_for_write
    except WindowsError as e:
NameError: global name 'WindowsError' is not defined

Without --release it installed locally just fine. Hope this helps.

Thanks.

jlorrain · 2020-12-15T14:56:22Z

I have encountered the same issue using debian.
I tried to install a pip package on a rez repository mounted from my network.

This seems to happen because some fs don't handle sync for directories (used to ensure atomicity) : untitaker/python-atomicwrites#17

my current workaround is to call rez pip on a local directory and use rsync before and after the call to transfert the newly installed package

Since atomicity can't be guaranteed within Network repositories, it would be nice to use a simple (non atomic) method if atomicwrites fail with this error (OSError: [Errno 22])

Fix issue AcademySoftwareFoundation#858

nerdvegas · 2020-12-29T02:59:49Z

fixed in #998

JeanChristopheMorinPerso added the bug label Mar 29, 2020

jlorrain pushed a commit to jlorrain/rez that referenced this issue Dec 16, 2020

Handle non atomic write method when writing on network fs

d22f97f

Fix issue AcademySoftwareFoundation#858

jlorrain mentioned this issue Dec 16, 2020

Handle non atomic write method when writing on network fs with linux #991

Closed

nerdvegas mentioned this issue Dec 29, 2020

update open_file_for_write with simplified error handling #998

Merged

nerdvegas closed this as completed Dec 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'rez build --install --prefix' error #858

'rez build --install --prefix' error #858

ClementHector commented Feb 24, 2020

nerdvegas commented Feb 24, 2020

alexxbb commented Sep 21, 2020

nerdvegas commented Sep 22, 2020 via email

alexxbb commented Sep 22, 2020

nerdvegas commented Sep 22, 2020

alexxbb commented Sep 23, 2020

jlorrain commented Dec 15, 2020

nerdvegas commented Dec 29, 2020

'rez build --install --prefix' error #858

'rez build --install --prefix' error #858

Comments

ClementHector commented Feb 24, 2020

nerdvegas commented Feb 24, 2020

alexxbb commented Sep 21, 2020

nerdvegas commented Sep 22, 2020 via email

alexxbb commented Sep 22, 2020

nerdvegas commented Sep 22, 2020

alexxbb commented Sep 23, 2020

jlorrain commented Dec 15, 2020

nerdvegas commented Dec 29, 2020