Add link libtensorflow_framework.so => libtensorflow_framework.so.1

[mdorier]

System information

• TensorFlow version (you are using): 1.14.0
• Are you willing to contribute it (Yes/No): No

Describe the feature and the current behavior/state.

Starting from TensorFlow 1.14.0, the pip packages come with a libtensorflow_framework.so.1 instead of a libtensorflow_framework.so. This breaks the build process of codes that extend tensorflow. Adding a symbolic link libtensorflow_framework.so => libtensorflow_framework.so.1 would greatly help.

[mihaimaruseac]

Unfortunately a wheel file is just a fancy zip archive so a symbolic link gets translated into a new copy of the file. So the file gets included twice, bloating the size of the pip package.

We will slowly evolve into a state where we can have the symlink in place, but it will take some time, unfortunately.

[byronyi]

The recommended approach for extending TF is to use tf.sysconfig.get_link_flags() instead of relying on manually specifying shared object path.

[mdorier]

Thanks for the quick reply. In the mean time, do you have any solution for a distutils-based package that has the following:

Extension('_my_custome_operations', source_files,
        libraries=['tensorflow_framework'],
        library_dirs=...,
        ...)

Because of the .1 suffix, the linker is not finding libtensorflow_framework. Putting the full name (libtensorflow_framework.so.1) doesn't work either. I had to either revert to Tensorflow 1.13.2 or add the symlink manually.

[mdorier]

Inspired by @byronyi 's comment, I tried what tf.sysconfig.get_link_flags() returns, i.e. I put libraries=[':libtensorflow_framework.so.1'] and it works. I should probably find a way to just pass what tf.sysconfig.get_link_flags() to distutils, though.

[gadagashwini-zz]

@mdorier Can you please let us know if you are happy to close if no issue persists

[mathlf2015]

Inspired by @byronyi 's comment, I tried what tf.sysconfig.get_link_flags() returns, i.e. I put libraries=[':libtensorflow_framework.so.1'] and it works. I should probably find a way to just pass what tf.sysconfig.get_link_flags() to distutils, though.

hello mdorier , can you show the code which help you solved this issue, i met the same problem

[mdorier]

@mathlf2015 Right now I still sort of hard-code libraries=[':libtensorflow_framework.so.2'] when creating the Extension object. My code is a bit particular, it runs on a supercomputer and I cannot import tensorflow on the node where I'm building it (tensorflow is cross-compiler for compute nodes), so I ended up having my setup.py script load a small JSON file indicating the library name (:libtensorflow_framework.so.2), library dir, and include dir, as well as possible cxx_flags to add.

[mathlf2015]

Right now I still sort of hard-code libraries=[':libtensorflow_framework.so.2'] when creating the Extension object. My code is a bit particular, it runs on a supercomputer and I cannot import tensorflow on the node where I'm building it (tensorflow is cross-compiler for compute nodes), so I ended up having my setup.py script load a small JSON file indicating the library name (:libtensorflow_framework.so.2), library dir, and include dir, as well as possible cxx_flags to add.

File "../../model/embedding/listing2vec.py", line 247, in __init__ os.path.join(os.path.dirname(os.path.realpath(__file__)), 'utils/word2vec_ops.so')) File "/root/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/load_library.py", line 61, in load_op_library lib_handle = py_tf.TF_LoadLibrary(library_filename) tensorflow.python.framework.errors_impl.NotFoundError: libtensorflow_framework.so: cannot open shared object file: No such file or directory

mdorier ,thank you . when i try to load a xx.so by load_library.py, i will always get the error ,i don't know how to us the "tf.sysconfig.get_link_flags()" to solve this . could you give me some advices.

[mdorier]

Not sure what you've done so far but on my laptop on Debian with tensorflow 2.1.0 installed via pip with the --user flag, tf.sysconfig.get_link_flags() returns ['-L/home/mdorier/.local/lib/python3.7/site-packages/tensorflow_core', '-l:libtensorflow_framework.so.2']. From this, you should find out which one is a library directory (the one with -L) and which one is the library (-l), then remove these -L and -l and pass these as the libraries and library_dirs arguments to your Extension object in setup.py. That should build fine.

Then if when trying to load the library you get a No such file or directory, try setting your LD_LIBRARY_PATH environment variable to point to the directory with the libtensorflow_framework.so library is located. I can't help much more; from experience this kind of problem can be very different from a platform to another.

[mathlf2015]

Not sure what you've done so far but on my laptop on Debian with tensorflow 2.1.0 installed via pip with the --user flag, tf.sysconfig.get_link_flags() returns ['-L/home/mdorier/.local/lib/python3.7/site-packages/tensorflow_core', '-l:libtensorflow_framework.so.2']. From this, you should find out which one is a library directory (the one with -L) and which one is the library (-l), then remove these -L and -l and pass these as the libraries and library_dirs arguments to your Extension object in setup.py. That should build fine.

Then if when trying to load the library you get a No such file or directory, try setting your LD_LIBRARY_PATH environment variable to point to the directory with the libtensorflow_framework.so library is located. I can't help much more; from experience this kind of problem can be very different from a platform to another.

thank you very much ,i solve my problem by a and b and c