Failed to include caffe_pb2 #12

tianyuandu · 2017-09-07T11:45:50Z

Hi Yihui,

I tried your code and met some problems.

After make -j8 and make pycaffe, I tried to python3 train.py, but found something wrong with protobuf.
So I change the protobuf version but the problem was still not solved.

Here is the problem:
When I tried protobuf 3.0.0(b1,b2,b3,b4) or 3.1.0 , the error message is:

Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 1087, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 1109, in InternalParse
    (tag_bytes, new_pos) = local_ReadTag(buffer, pos)
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/decoder.py", line 181, in ReadTag
    while six.indexbytes(buffer, pos) & 0x80:
TypeError: unsupported operand type(s) for &: 'str' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 19, in <module>
    from lib.net import Net, load_layer, caffe_test
  File "/mnt/lustre/dutianyuan/channel-pruning/lib/net.py", line 7, in <module>
    import caffe
  File "/mnt/lustre/dutianyuan/channel-pruning/caffe/python/caffe/__init__.py", line 4, in <module>
    from .proto.caffe_pb2 import TRAIN, TEST
  File "/mnt/lustre/dutianyuan/channel-pruning/caffe/python/caffe/proto/caffe_pb2.py", line 799, in <module>
    options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), '\020\001')),
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/descriptor.py", line 869, in _ParseOptions
    message.ParseFromString(string)
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/message.py", line 185, in ParseFromString
    self.MergeFromString(serialized)
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 1093, in MergeFromString
    raise message_mod.DecodeError('Truncated message.')
google.protobuf.message.DecodeError: Truncated message.

and when I change protobuf to 3.2.0 / 3.3.0 / 3.4.0, the error message is

Traceback (most recent call last):
File "train.py", line 19, in <module>
  from lib.net import Net, load_layer, caffe_test 
File "/mnt/lustre/dutianyuan/channel-pruning/lib/net.py", line 7, in <module>
  import caffe
File "/mnt/lustre/dutianyuan/channel-pruning/caffe/python/caffe/__init__.py", line 4, in <module>
  from .proto.caffe_pb2 import TRAIN, TEST
File "/mnt/lustre/dutianyuan/channel-pruning/caffe/python/caffe/proto/caffe_pb2.py", line 17, in <module>
  serialized_pb='\n\x0b\x63\x61\x66\x66\x65.proto\x12\x05\x63\x61\x66\x66\x65\"\x1c\n\tBlobShape\x12\x0f\n\x03\x64im\x18\x01 
.....with a lot of \x......
  File "/home/dutianyuan/anaconda3/lib/python3.5/site-packages/google/protobuf/descriptor.py", line 824, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: expected bytes, str found

When I first met this problem, my Python version is 3.4 and your default setting is with Python 3.5. So I install Python3.5 and Python3.6 by Anaconda, Python 3.6 by apt-get. No matter what version is, the problem was still not solved.

Hope you can show me the specific version of your coding environment!
Thanks!

The text was updated successfully, but these errors were encountered:

ethanhe42 · 2017-09-07T13:33:59Z

I encountered this before, however solved it with upgrade to protobuf 3.2
my version:

$ pip3 show protobuf
Name: protobuf
Version: 3.2.0
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: [email protected]
Author-email: [email protected]
License: New BSD License
Location: /usr/local/lib/python3.5/dist-packages
Requires: setuptools, six

If you still suffer this problem, the alternative is install my ImageData layer to your own Caffe.
Replace the following parts:

/src/caffe/layers/image_data_layer.cpp
/include/caffe/layers/image_data_layer.hpp
/src/caffe/util/io.cpp
/include/caffe/util/io.hpp
src/caffe/data_transformer.cpp

and https://github.com/yihui-he/caffe-pro/blob/master/src/caffe/proto/caffe.proto#L783-L814

ethanhe42 · 2017-09-07T13:35:29Z

whether these parts still works well in current caffe is not granted

ethanhe42 · 2017-09-18T15:59:03Z

According to BVLC/caffe#2464, this problem is solved in latest Caffe version. Since I can't reproduce this problem, I've merge my Caffe to the latest BVLC/caffe. Please update and try remake.

tianyuandu · 2017-09-20T04:04:03Z

@yihui-he Thanks for your suggestion.
I've tried the same setting you mentioned with caffe in your repository but still didn't work.
Then I cloned the latest Caffe (before official Caffe solved this problem) and replaced the modified files, and built with protobuf 3.2 and python3.5, I can successfully include caffe_pb2 and run your code.

LearnerInGithub · 2017-09-27T06:23:08Z

@tianyuandu I clone the official caffe, and replaced the files you mentioned, but still face this problem. Does anything I'm missing? I installed the protobuf 3.2 and python3 is Python3.5.

ethanhe42 · 2017-09-27T06:29:57Z

Can you built official caffe successfully in the first place? If not, please ask this question in Caffe.

LearnerInGithub · 2017-09-27T06:33:23Z

@yihui-he Yes, I can compile caffe successfully, problem will happy while I run the command:
python3 train.py -action c3 -caffe 0

tianyuandu · 2017-09-27T08:15:14Z

@LearnerInGithub
I'm not sure what your error message is. You can paste error message here.
After built official caffe, can you import caffe and import caffe.proto?
If import caffe worked and import caffe.proto failed, you can try to add these codes at the start of train.py after from __future__ import print_function:
import sys
sys.path.insert(0, "path/to/caffe/python")

Hope that can help you.

LearnerInGithub · 2017-09-27T08:34:47Z

@tianyuandu Thanks for your reply, when I import caffe, error will occurs:

Traceback (most recent call last):
File "", line 1, in
File "/channel-pruning/caffe/python/caffe/init.py", line 1, in
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver, NCCL, Timer
File "/cephfs/person//channel-pruning/caffe/python/caffe/pycaffe.py", line 15, in
import caffe.io
File "/channel-pruning/caffe/python/caffe/io.py", line 2, in
import skimage.io
File "/usr/local/lib/python3.5/site-packages/skimage/io/init.py", line 15, in
reset_plugins()
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 93, in reset_plugins
_load_preferred_plugins()
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 73, in _load_preferred_plugins
_set_plugin(p_type, preferred_plugins['all'])
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 85, in _set_plugin
use_plugin(plugin, kind=plugin_type)
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 255, in use_plugin
_load(name)
File "/usr/local/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 299, in _load
fromlist=[modname])
File "/usr/local/lib/python3.5/site-packages/skimage/io/_plugins/matplotlib_plugin.py", line 3, in
import matplotlib.pyplot as plt
File "/usr/local/lib/python3.5/site-packages/matplotlib/pyplot.py", line 37, in
from matplotlib.figure import Figure, figaspect
File "/usr/local/lib/python3.5/site-packages/matplotlib/figure.py", line 40, in
from matplotlib.axes import Axes, SubplotBase, subplot_class_factory
File "/usr/local/lib/python3.5/site-packages/matplotlib/axes/init.py", line 4, in
from ._subplots import *
File "/usr/local/lib/python3.5/site-packages/matplotlib/axes/_subplots.py", line 10, in
from matplotlib.axes._axes import Axes
File "/usr/local/lib/python3.5/site-packages/matplotlib/axes/_axes.py", line 24, in
import matplotlib.dates as _ # <-registers a date unit converter
File "/usr/local/lib/python3.5/site-packages/matplotlib/dates.py", line 125, in
from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY,
File "/usr/local/lib/python3.5/site-packages/dateutil/rrule.py", line 55
raise ValueError, "Can't create weekday with n == 0"
^
SyntaxError: invalid syntax

tianyuandu · 2017-09-27T08:48:50Z

@LearnerInGithub
Actually I haven't meet this problem.
I just found a similar error in http://blog.csdn.net/yzf0011/article/details/73250107
You can try this:sudo pip3 install --upgrade python-util

LearnerInGithub · 2017-09-27T08:52:56Z

@tianyuandu I execute this command: sudo pip3 install --upgrade python-util:
Requirement already up-to-date: python-util in /usr/local/lib/python3.5/site-packages
Requirement already up-to-date: docopt in /usr/local/lib/python3.5/site-packages (from python-util)

However, the import caffe still have problem:

import caffe
Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
File "", line 1, in
File "/cephfs/person/mingliu/channel-pruning/caffe/python/caffe/init.py", line 4, in
from .proto.caffe_pb2 import TRAIN, TEST
File "/cephfs/person/mingliu/channel-pruning/caffe/python/caffe/proto/caffe_pb2.py", line 831, in
options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), _b('\020\001')), file=DESCRIPTOR),
TypeError: init() got an unexpected keyword argument 'file'

tianyuandu · 2017-09-27T09:04:22Z

@LearnerInGithub Seem like caffe_pb2.py has some problem. Is your protobuf 3 install by pip3?
And when you are building caffe, the protobuf in system was chosen and maybe the version is mismatch with pip's protobuf.
I used cmake to build caffe so that I can see the exact version of protobuf, boost and cuda. You can also try this.

LearnerInGithub · 2017-09-27T09:36:32Z

@tianyuandu Sounds a good idea, I will try it. Thanks you!

LearnerInGithub · 2017-09-27T13:08:06Z

@tianyuandu @yihui-he protobuf problem fixed, because of the c++ version pb not matching the python version, reinstall them has fixed. Thanks two guys!

yingning · 2017-12-22T02:51:20Z

@LearnerInGithub ,you mentioned you reinstall protobuf and python ? which version of protobuf and python you used? I had the same problem and my protobuf is 3.5.0 ,python version is 2.7.13 of anaconda. what should I do to fix this problem?Thanks .

zhongcl-thu · 2018-04-11T02:36:00Z

@tianyuandu @LearnerInGithub Hello， I downloaded the latest version of caffe, installed anaconda, and installed both python 2.7 and 3.5. The protoc in usr/lib is version 2.5. If I don't install(conda install) protoc 3.2 (or later) in python and use protoc2.5 , then caffe can be made all. But once protoc3.2 (or later) is installed in python, i cannot make all caffe. I hope you can provide some advice, thank you.

cateweb · 2018-07-09T05:39:31Z

solved with sudo pip3 install sklearn-contrib-lightning

LongruiDong · 2018-11-03T11:47:13Z

@tianyuandu @LearnerInGithub Hello， I downloaded the latest version of caffe, installed anaconda, and installed both python 2.7 and 3.5. The protoc in usr/lib is version 2.5. If I don't install(conda install) protoc 3.2 (or later) in python and use protoc2.5 , then caffe can be made all. But once protoc3.2 (or later) is installed in python, i cannot make all caffe. I hope you can provide some advice, thank you.

i am facing same problem,have you solved the problem?

SophieFeng · 2019-02-24T12:52:33Z

I have solved this problem. I update protobuf from v2.5.0 to v3.2.0, it seems work! ps: I use python 3.5.0.

ethanhe42 mentioned this issue Sep 18, 2017

some kind of issue when python3 train.py -action c3 -caffe 0 #13

Closed

ethanhe42 closed this as completed Sep 20, 2017

ethanhe42 mentioned this issue Sep 27, 2017

trian error #22

Closed

ethanhe42 mentioned this issue Oct 16, 2017

make pruning error #37

Closed

ethanhe42 pinned this issue Jan 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to include caffe_pb2 #12

Failed to include caffe_pb2 #12

tianyuandu commented Sep 7, 2017

ethanhe42 commented Sep 7, 2017 •

edited

Loading

ethanhe42 commented Sep 7, 2017

ethanhe42 commented Sep 18, 2017

tianyuandu commented Sep 20, 2017

LearnerInGithub commented Sep 27, 2017

ethanhe42 commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017

tianyuandu commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017 •

edited

Loading

tianyuandu commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017

tianyuandu commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017

yingning commented Dec 22, 2017 •

edited

Loading

zhongcl-thu commented Apr 11, 2018

cateweb commented Jul 9, 2018

LongruiDong commented Nov 3, 2018

SophieFeng commented Feb 24, 2019

Failed to include caffe_pb2 #12

Failed to include caffe_pb2 #12

Comments

tianyuandu commented Sep 7, 2017

ethanhe42 commented Sep 7, 2017 • edited Loading

ethanhe42 commented Sep 7, 2017

ethanhe42 commented Sep 18, 2017

tianyuandu commented Sep 20, 2017

LearnerInGithub commented Sep 27, 2017

ethanhe42 commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017

tianyuandu commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017 • edited Loading

tianyuandu commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017

tianyuandu commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017

LearnerInGithub commented Sep 27, 2017

yingning commented Dec 22, 2017 • edited Loading

zhongcl-thu commented Apr 11, 2018

cateweb commented Jul 9, 2018

LongruiDong commented Nov 3, 2018

SophieFeng commented Feb 24, 2019

ethanhe42 commented Sep 7, 2017 •

edited

Loading

LearnerInGithub commented Sep 27, 2017 •

edited

Loading

yingning commented Dec 22, 2017 •

edited

Loading