在linux批处理模式下运行Rcpp时出错

Question

我试图运行一个非常简单的模型与rstan在批处理模式，并有一些错误。当我尝试在批处理模式下运行相同的代码时，我没有任何错误。

来自斯坦论坛的社区帮助我理解，问题是Rcpp而不是STAN没有在我的大学服务器上正常工作。我们是从一个尝试在批处理模式下而不是在批处理模式下运行一个非常简单的Rcpp代码的测试中获得这一理解的。Rcpp代码是：

### Libraries
library(Rcpp)

Rcpp::sourceCpp(code='
  #include <Rcpp.h>

  // [[Rcpp::export]]
  int fibonacci(const int x) {
    if (x == 0) return(0);
    if (x == 1) return(1);
    return (fibonacci(x - 1)) + fibonacci(x - 2);
  }',
                verbose = TRUE,
                rebuild = TRUE
)

结果如下。

非批处理模式：

Generated R functions
-------------------------------------------------------

`.sourceCpp_1_DLLInfo` <- dyn.load('/tmp/RtmpDNL6of/sourceCpp-x86_64-pc-linux-gnu-1.0.7/sourcecpp_178b21d435d91/sourceCpp_4.so')

fibonacci <- Rcpp:::sourceCppFunction(function(x) {}, FALSE, `.sourceCpp_1_DLLInfo`, 'sourceCpp_1_fibonacci')

rm(`.sourceCpp_1_DLLInfo`)

Building shared library
--------------------------------------------------------

DIR: /tmp/RtmpDNL6of/sourceCpp-x86_64-pc-linux-gnu-1.0.7/sourcecpp_178b21d435d91

/hpc/apps/R/4.1.2/lib64/R/bin/R CMD SHLIB --preclean -o 'sourceCpp_4.so' 'file178b25628f5ea.cpp'
g++ -std=gnu++14 -I"/hpc/apps/R/4.1.2/lib64/R/include" -DNDEBUG   -I"/hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include" -I"/tmp/RtmpDNL6of/sourceCpp-x86_64-pc-linux-gnu-1.0.7" -I/hpc/apps/zlib/1.2.8/include -I/hpc/apps/pcre/8.38/include -I/hpc/apps/bzip2/1.0.6/include -I/hpc/apps/curl/7.54.1/include -I/hpc/apps/cairo/1.14.12/include   -fpic  -g -O2  -c file178b25628f5ea.cpp -o file178b25628f5ea.o
g++ -std=gnu++14 -shared -L/hpc/apps/R/4.1.2/lib64/R/lib -L/hpc/apps/zlib/1.2.8/lib -L/hpc/apps/pcre/8.38/lib -L/hpc/apps/bzip2/1.0.6/lib -L/hpc/apps/curl/7.54.1/lib -L/hpc/apps/cairo/1.14.12/lib -o sourceCpp_4.so file178b25628f5ea.o -L/hpc/apps/R/4.1.2/lib64/R/lib -lR

批处理模式：

Generated R functions 
-------------------------------------------------------

`.sourceCpp_1_DLLInfo` <- dyn.load('/localscratch/250948.1.ragatkolab.q/RtmpDlK5Sy/sourceCpp-x86_64-pc-linux-gnu-1.0.7/sourcecpp_142bd68d660f1/sourceCpp_2.so')

fibonacci <- Rcpp:::sourceCppFunction(function(x) {}, FALSE, `.sourceCpp_1_DLLInfo`, 'sourceCpp_1_fibonacci')

rm(`.sourceCpp_1_DLLInfo`)

Building shared library
--------------------------------------------------------

DIR: /localscratch/250948.1.mylab.q/RtmpDlK5Sy/sourceCpp-x86_64-pc-linux-gnu-1.0.7/sourcecpp_142bd68d660f1

/hpc/apps/R/4.1.2/lib64/R/bin/R CMD SHLIB -o 'sourceCpp_2.so' 'file142bd64ebe902.cpp' 
g++ -std=gnu++14 -I"/hpc/apps/R/4.1.2/lib64/R/include" -DNDEBUG   -I"/hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include" -I"/localscratch/250948.1.ragatkolab.q/RtmpDlK5Sy/sourceCpp-x86_64-pc-linux-gnu-1.0.7" -I/hpc/apps/zlib/1.2.8/include -I/hpc/apps/pcre/8.38/include -I/hpc/apps/bzip2/1.0.6/include -I/hpc/apps/curl/7.54.1/include -I/hpc/apps/cairo/1.14.12/include   -fpic  -g -O2  -c file142bd64ebe902.cpp -o file142bd64ebe902.o
In file included from /hpc/apps/gcc/9.1.0/include/c++/9.1.0/x86_64-pc-linux-gnu/bits/c++config.h:524,
                 from /hpc/apps/gcc/9.1.0/include/c++/9.1.0/cmath:41,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp/platform/compiler.h:100,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp/r/headers.h:66,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/RcppCommon.h:30,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp.h:27,
                 from file142bd64ebe902.cpp:2:
/hpc/apps/gcc/9.1.0/include/c++/9.1.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: fatal error: features.h: No such file or directory
   39 | #include <features.h>
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [file142bd64ebe902.o] Error 1
Error in Rcpp::sourceCpp(code = "\n  #include <Rcpp.h>\n\n  // [[Rcpp::export]]\n  int fibonacci(const int x) {\n    if (x == 0) return(0);\n    if (x == 1) return(1);\n    return (fibonacci(x - 1)) + fibonacci(x - 2);\n  }",  : 
  Error 1 occurred building shared library.

WARNING: The tools required to build C++ code for R were not found.

Please install GNU development tools including a C++ compiler.

Execution halted

然后，他们让我修改我的makevars文件--我使用R包usethis::edit_r_makevars() --添加了行：

CXXFLAGS += -I"/hpc/apps/gcc/9.1.0/include/c++/9.1.0/parallel/"
CXX14FLAGS += -I"/hpc/apps/gcc/9.1.0/include/c++/9.1.0/parallel/"

一旦我做了这个修改(我希望我做的正确)，我的错误消息将更改为：

Generated extern "C" functions 
--------------------------------------------------------


#include <Rcpp.h>
#ifdef RCPP_USE_GLOBAL_ROSTREAM
Rcpp::Rostream<true>&  Rcpp::Rcout = Rcpp::Rcpp_cout_get();
Rcpp::Rostream<false>& Rcpp::Rcerr = Rcpp::Rcpp_cerr_get();
#endif

// fibonacci
int fibonacci(const int x);
RcppExport SEXP sourceCpp_1_fibonacci(SEXP xSEXP) {
BEGIN_RCPP
    Rcpp::RObject rcpp_result_gen;
    Rcpp::RNGScope rcpp_rngScope_gen;
    Rcpp::traits::input_parameter< const int >::type x(xSEXP);
    rcpp_result_gen = Rcpp::wrap(fibonacci(x));
    return rcpp_result_gen;
END_RCPP
}

Generated R functions 
-------------------------------------------------------

`.sourceCpp_1_DLLInfo` <- dyn.load('/localscratch/251578.1.mylablab.q/RtmpCHyEAd/sourceCpp-x86_64-pc-linux-gnu-1.0.7/sourcecpp_1e50364a03315/sourceCpp_2.so')

fibonacci <- Rcpp:::sourceCppFunction(function(x) {}, FALSE, `.sourceCpp_1_DLLInfo`, 'sourceCpp_1_fibonacci')

rm(`.sourceCpp_1_DLLInfo`)

Building shared library
--------------------------------------------------------

DIR: /localscratch/251578.1.mylablab.q/RtmpCHyEAd/sourceCpp-x86_64-pc-linux-gnu-1.0.7/sourcecpp_1e50364a03315

/hpc/apps/R/4.1.2/lib64/R/bin/R CMD SHLIB -o 'sourceCpp_2.so' 'file1e50325e7686.cpp' 
g++ -std=gnu++14 -I"/hpc/apps/R/4.1.2/lib64/R/include" -DNDEBUG   -I"/hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include" -I"/localscratch/251578.1.mylablab.q/RtmpCHyEAd/sourceCpp-x86_64-pc-linux-gnu-1.0.7" -I/hpc/apps/zlib/1.2.8/include -I/hpc/apps/pcre/8.38/include -I/hpc/apps/bzip2/1.0.6/include -I/hpc/apps/curl/7.54.1/include -I/hpc/apps/cairo/1.14.12/include   -fpic  -g -O2  -I"/hpc/apps/gcc/9.1.0/include/c++/9.1.0/parallel/" -c file1e50325e7686.cpp -o file1e50325e7686.o
In file included from /hpc/apps/gcc/11.1.0/include/c++/11.1.0/x86_64-pc-linux-gnu/bits/c++config.h:571,
                 from /hpc/apps/gcc/11.1.0/include/c++/11.1.0/cmath:41,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp/platform/compiler.h:100,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp/r/headers.h:66,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/RcppCommon.h:30,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp.h:27,
                 from file1e50325e7686.cpp:2:
/hpc/apps/gcc/11.1.0/include/c++/11.1.0/x86_64-pc-linux-gnu/bits/os_defines.h:44:19: error: missing binary operator before token "("
   44 | #if __GLIBC_PREREQ(2,15) && defined(_GNU_SOURCE)
      |                   ^
/hpc/apps/gcc/11.1.0/include/c++/11.1.0/x86_64-pc-linux-gnu/bits/os_defines.h:52:19: error: missing binary operator before token "("
   52 | #if __GLIBC_PREREQ(2, 27)
      |                   ^
In file included from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp/platform/compiler.h:100,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp/r/headers.h:66,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/RcppCommon.h:30,
                 from /hpc/home/user/R/x86_64-pc-linux-gnu-library/4.1/Rcpp/include/Rcpp.h:27,
                 from file1e50325e7686.cpp:2:
/hpc/apps/gcc/11.1.0/include/c++/11.1.0/cmath:45:15: fatal error: math.h: No such file or directory
   45 | #include_next <math.h>
      |               ^~~~~~~~
compilation terminated.
make: *** [file1e50325e7686.o] Error 1
Error in Rcpp::sourceCpp(code = "\n  #include <Rcpp.h>\n\n  // [[Rcpp::export]]\n  int fibonacci(const int x) {\n    if (x == 0) return(0);\n    if (x == 1) return(1);\n    return (fibonacci(x - 1)) + fibonacci(x - 2);\n  }",  : 
  Error 1 occurred building shared library.

WARNING: The tools required to build C++ code for R were not found.

Please install GNU development tools including a C++ compiler.

Execution halted

我在Linux方面的知识是有限的，我恐怕我的问题超出了斯坦论坛的范围。有什么想法怎么继续吗？

Answer 1

恐怕你的问题还在本地。在您使用的计算中，可能有人以不同的方式设置了R，从而区别于交互式R使用和您所称的批处理使用。

但这是不标准的，而且在你的结尾。一般来说，它只是起作用了。使用您的程序的最小修改版本(我仍然感到困惑:它加载doParallel，注册内核，但是.触发并行代码--无论如何--它工作得很好：

使用Rscript

edd@rob:~/git/stackoverflow/70470842(master)$ Rscript answer.R  
Loading required package: foreach        
Loading required package: iterators      
Loading required package: parallel       
Fib(10) is  55                           
edd@rob:~/git/stackoverflow/70470842(master)$ 

在批处理模式中使用R

edd@rob:~/git/stackoverflow/70470842(master)$ R -s -f answer.R 
Loading required package: foreach     
Loading required package: iterators   
Loading required package: parallel    
Fib(10) is  55                        
edd@rob:~/git/stackoverflow/70470842(master)$  

利用我们小小的前卫

edd@rob:~/git/stackoverflow/70470842(master)$ r answer.R     
Loading required package: foreach               
Loading required package: iterators             
Loading required package: utils                 
Loading required package: parallel              
Fib(10) is  55                                  
edd@rob:~/git/stackoverflow/70470842(master)$   

修改源代码

library(doParallel)             # Library        

ncores <- 5                     # Simulation parameters
n_sim <- 5                  
registerDoParallel(ncores)  

Rcpp::sourceCpp(code='  
  #include <Rcpp.h>     
  // [[Rcpp::export]]            
  int fibonacci(const int x) {   
    if (x == 0) return(0);       
    if (x == 1) return(1);       
    return (fibonacci(x - 1)) + fibonacci(x - 2); 
  }'          
  )           
                                                                                                                                                                                                                   
cat("Fib(10) is ", fibonacci(10), "\n")  

Answer 2

作为将来的参考，在收到这里和斯坦论坛的反馈后，我与我所在机构的IT团队进行了互动，他们最终找到了解决方案。用他们的话说：

计算节点通常不用于开发。因此某些开发包在默认情况下不会安装。我们在计算节点上安装了“glibc”和“glibc”。

因此，当以交互方式运行作业时，作业似乎运行在submit节点上(该节点安装了这些库)，而从调度器运行时，作业运行在计算节点上(没有安装库)。造成这种差异的原因是，有关图书馆传统上只在开发和汇编期间使用。这种使用构成了一个独特的用例。“