CaffeonSpark生成错误caffe.proto需要'src/main/java/caffe/Caffe.java

Question

我正在尝试让CaffeOnSpark在本地运行，并在CaffeOnSpark维基上遵循以下步骤：https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_local

当我在步骤5中进行Make构建时，前4个项目对me.However是有好处的。

整个错误堆栈：

jiaqi@jiaqi-ThinkPad-L450:~/github/CaffeOnSpark$ make build
cd caffe-public; make proto; make -j4 -e distribute; cd ..
make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
make[1]: Nothing to be done for 'proto'.
make[1]: Leaving directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
CXX/LD -o .build_release/tools/convert_imageset.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_binary.bin
CXX/LD -o .build_release/tools/compute_image_mean.bin
CXX/LD -o .build_release/tools/caffe.bin
.build_release/tools/compute_image_mean.o: In function `std::string* google::MakeCheckOpString<int, int>(int const&, int const&, char const*)':
compute_image_mean.cpp:(.text._ZN6google17MakeCheckOpStringIiiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringIiiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/compute_image_mean.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
compute_image_mean.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/compute_image_mean.o: In function `main':
compute_image_mean.cpp:(.text.startup+0x168): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/compute_image_mean.bin' failed
make[1]: *** [.build_release/tools/compute_image_mean.bin] Error 1
make[1]: *** Waiting for unfinished jobs....
.build_release/tools/convert_imageset.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
convert_imageset.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/convert_imageset.o: In function `main':
convert_imageset.cpp:(.text.startup+0x347): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/convert_imageset.bin' failed
make[1]: *** [.build_release/tools/convert_imageset.bin] Error 1
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/upgrade_net_proto_binary.bin' failed
make[1]: *** [.build_release/tools/upgrade_net_proto_binary.bin] Error 1
.build_release/tools/caffe.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
caffe.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/caffe.o: In function `main':
caffe.cpp:(.text.startup+0x50): undefined reference to `google::SetVersionString(std::string const&)'
caffe.cpp:(.text.startup+0x83): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/caffe.bin' failed
make[1]: *** [.build_release/tools/caffe.bin] Error 1
make[1]: Leaving directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
export LD_LIBRARY_PATH="/home/y/lib64:/home/y/lib64/mkl/intel64:/home/jiaqi/github/CaffeOnSpark/caffe-public/distribute/lib:/home/jiaqi/github/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; mvn -B package
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT
[WARNING] The expression ${version} is deprecated. Please use ${project.version} instead.
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] caffe
[INFO] caffe-distri
[INFO] caffe-grid
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-distri 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (proto) @ caffe-distri ---
[INFO] Executing tasks

protoc:
     [exec] make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-distri'
     [exec] make[1]: Leaving directory '/home/jiaqi/github/CaffeOmake[1]: *** No rule to make target '../caffe-public/distribute/proto/caffe.proto', needed by 'src/main/java/caffe/Caffe.java'. nSpark/caffe-distri'
     [exec]  Stop.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] caffe .............................................. SUCCESS [  0.019 s]
[INFO] caffe-distri ....................................... FAILURE [  2.425 s]
[INFO] caffe-grid ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.144 s
[INFO] Finished at: 2016-05-13T12:33:40+08:00
[INFO] Final Memory: 9M/238M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (proto) on project caffe-distri: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...<exec failonerror="true" dir="/home/jiaqi/github/CaffeOnSpark/caffe-distri" executable="make">... @ 5:97 in /home/jiaqi/github/CaffeOnSpark/caffe-distri/target/antrun/build-protoc.xml
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :caffe-distri
Makefile:9: recipe for target 'build' failed
make: *** [build] Error 1

protoc似乎找不到它的规则。你有没有讨论过这个问题？请帮帮我!

我的protoc的版本是2.5.0。

Answer 1

我以前遇到过CaffeOnSpark的构建错误。我知道这篇文章很陈旧，但不管怎样，它可能会对某些人有所帮助。

从日志中，我认为你有一些依赖问题。CaffeOnSpark需要Caffe met的所有依赖项。所以我想问的第一个问题是Caffe是否正常工作(如果安装了)。

1. 如果没有安装Caffe，您可以尝试安装Caffe并通过打开python并输入import caffe来验证它的工作情况。如果Caffe没有运行，请转到步骤5.
2. 如果caffe运行正常，请验证您执行的命令是否与下载git的命令完全相同：

git clone https://github.com/yahoo/CaffeOnSpark.git --recursive

• Then请在CaffeOnSpark/caffe-public中检查您的Makefile.config。确保正确地给出了所有标志。检查后，运行以下命令：

cd CaffeOnSpark make clean make build

• If您会遇到以下错误：

make1:离开`/home/das/CaffeOnSpark/caffe-public‘导出LD_LIBRARY_PATH="/usr/local/cuda-7.5/lib64:/home/das/CaffeOnSpark/caffe-public/distribute/lib:/home/das/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64“目录；mvn -B包-DskipTests /bin/sh: 1: mvn:未找到make：* build Error 127

运行以下命令来安装maven：

sudo apt-get install maven

• 如果你仍然收到错误，你可能需要重新安装/检查依赖项。我会把说明书贴出来以防万一。请注意，其中许多实际上并不是必需的。我安装这些来设置我的Deep Learning机器：

sudo apt-get update sudo apt-get install -y build-essential cmake g++ gfortran git pkg-config python-dev software-properties-common sudo apt-get install -y python-pip python-dev python-numpy python-scipy python-numpy h5py python-skimage python-matplotlib python-pandas python-sklearn python- sudo apt-get install --no- -y推荐libboost all-dev sudo apt-get install libprotobuf-dev libleveldb dev libsnappy-dev libhdf5--y-protobuf dev-编译器libgflags dev libgoogle-glog-dev liblmdb-dev

并从Caffe根文件夹安装要求：

sudo pip install -r python/requirements.txt

• If存在BLAS问题，请正确安装：

install

• Please克隆https://github.com/xianyi/OpenBLAS.git cd OpenBLAS make FC=gfortran PREFIX 4 sudo make PREFIX=/usr/local -j确保.bashrc文件的所有路径都是正确的。

导出LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH导出CAFFE_ROOT=/home/das/Caffe export PYTHONPATH=$CAFFE_ROOT/python:$PYTHONPATH export CAFFE_ON_SPARK=/home/das/CaffeOnSpark export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export HADOOP_HOME=/usr/local/hadoop export SPARK_HOME=/usr/local/spark export LD_LIBRARY_PATH=${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib

..and如果仍然有错误，请将其发布，以便我们可以查看。

谢谢。

Answer 2

早些时候应该会有一条错误消息，说找不到blas库。默认情况下，Caffe使用地图库，可以在Makefile.config：BLAS := atlas中找到，如果你安装了其他blas库，比如mkl，你应该使用BLAS := mkl。

Answer 3

我尝试在Ubuntu16.04上编译CaffeOnSpark，在GetStarted_standalone参考之前，我从相应的网站安装了maven-3.0.5、hadoop-2.7.1和spark-2.0.0-bin-hadoop2.7。我遇到了类似的问题，错误如下：

[INFO] Compiling 10 source files to /home/bdg/usr/CaffeOnSpark/caffe-distri/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/bdg/usr/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[9919,31] cannot find symbol
  symbol:   class ProtocolStringList
  location: package com.google.protobuf
[ERROR] /home/bdg/usr/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[8280,24] cannot find symbol
  symbol:   class ProtocolStringList
  location: package com.google.protobuf
...
[ERROR] /home/bdg/usr/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[5655,26] cannot find symbol
  symbol:   method getUnmodifiableView()
  location: variable input_ of type com.google.protobuf.LazyStringList
[ERROR] /home/bdg/usr/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[6395,26] cannot find symbol
  symbol:   method getUnmodifiableView()
  location: variable input_ of type com.google.protobuf.LazyStringList
...
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :caffe-distri
Makefile:19: recipe for target 'build' failed
make: *** [build] Error 1