使用Armadillo和OpenBLAS多线程时性能不一致

Question

我用Armadillo编写了矩阵向量乘法和线性系统求解.Armadillo是从源代码编译的，使用的是OpenBLAS，也是从源代码编译的。不幸的是，我得到的结果不一致的单线程和多线程运行。矩阵向量乘法在单线程上运行得更快，而线性系统求解时在多线程时运行得更快。我希望有人能给我指点我做错了什么。

见下文：

• 源代码
• 编译和运行bash脚本
• 结果
• 系统信息

matmul_armadillo.cpp

#include <armadillo>

using namespace arma;

int main(int argc, char *argv[])
{
    const int n = atoi(argv[1]);

    mat A = randu<mat>(n, n);
    vec x = randu<vec>(n);

    A*x;

    return 0;
}

solve_armadillo.cpp

#include <armadillo>

using namespace arma;

int main(int argc, char *argv[])
{
    const int n = atoi(argv[1]);

    mat A = randu<mat>(n, n);
    vec b = randu<vec>(n);
    vec x;

    x = solve(A, b);

    return 0;
}

benchmark.sh

#!/bin/bash

g++ matmul_armadillo.cpp -o matmul_armadillo -O3 -march=native -std=c++11 -larmadillo
g++ solve_armadillo.cpp -o solve_armadillo -O3 -march=native -std=c++11 -larmadillo

N=7500

export OPENBLAS_NUM_THREADS=1
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=2
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=3
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=4
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=5
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=6
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=7
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N
echo ''

export OPENBLAS_NUM_THREADS=8
echo 'Running matmul_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./matmul_armadillo $N
echo ''
echo 'Running solve_armadillo on' $OPENBLAS_NUM_THREADS 'threads'
time ./solve_armadillo $N

结果

$ ./benchmark.sh 
Running matmul_armadillo on 1 threads

real    0m0.943s
user    0m0.628s
sys     0m0.159s

Running solve_armadillo on 1 threads

real    0m13.910s
user    0m13.553s
sys     0m0.300s

Running matmul_armadillo on 2 threads

real    0m1.528s
user    0m1.361s
sys     0m0.402s

Running solve_armadillo on 2 threads

real    0m15.815s
user    0m29.097s
sys     0m1.083s

Running matmul_armadillo on 3 threads

real    0m1.534s
user    0m1.480s
sys     0m0.533s

Running solve_armadillo on 3 threads

real    0m11.729s
user    0m31.022s
sys     0m1.290s

Running matmul_armadillo on 4 threads

real    0m1.543s
user    0m1.619s
sys     0m0.674s

Running solve_armadillo on 4 threads

real    0m10.013s
user    0m34.055s
sys     0m1.696s

Running matmul_armadillo on 5 threads

real    0m1.545s
user    0m1.620s
sys     0m0.664s

Running solve_armadillo on 5 threads

real    0m9.945s
user    0m33.803s
sys     0m1.669s

Running matmul_armadillo on 6 threads

real    0m1.543s
user    0m1.607s
sys     0m0.684s

Running solve_armadillo on 6 threads

real    0m10.069s
user    0m34.283s
sys     0m1.699s

Running matmul_armadillo on 7 threads

real    0m1.542s
user    0m1.622s
sys     0m0.661s

Running solve_armadillo on 7 threads

real    0m10.041s
user    0m34.154s
sys     0m1.704s

Running matmul_armadillo on 8 threads

real    0m1.546s
user    0m1.576s
sys     0m0.712s

Running solve_armadillo on 8 threads

real    0m10.123s
user    0m34.492s
sys     0m1.697s

系统信息

• openSUSE 13.1 64位
• 鲤鱼4.100.2 (来源汇编)
• OpenBLAS 0.2.8 (从源代码编译)

Answer 1

我怀疑

A*x;

可能是因为您没有对结果做任何操作而进行了优化。Armadillo中乘法运算的延迟评估模板魔术可以很容易地导致计算的Lapack例程被调用。因此，如果启用线程，您只需要测量设置该线程的开销。因此，在禁用线程的情况下，程序执行得更快。

使用

x = solve(A, b);

这是不同的，因为这会直接导致相应的Lapack调用，这可能无法被优化，因为编译器不能排除副作用，并且实际上将结果分配给一个变量。对于这样大的矩阵，solve调用从多处理中获益。

要修复基准测试，您应该做两件事：

• 利用计算结果来阻止优化器做太多事情。
• 多次重复计算，以获得更好的统计量，减少初始设置成本的影响。

下面是一个未经测试的示例：

#include <iostream>
#include <armadillo>

using namespace arma;

int main(int argc, char *argv[])
{
    const int n = atoi(argv[1]);

    mat A = randu<mat>(n, n);
    vec x = randu<vec>(n);

    for (int i = 0; i < 100; ++i) {
        x = A*x;
    }
    x.print(std::cout);

    return 0;
}

print调用可能没有必要。