如何在Mac上安装带有homebrew的apache-spark 2.2.0

Question

"$brew安装apache-spark‘得到2.3.x版本. '$brew搜索apache-spark’和'$brew info apache-spark‘不提供一个选项来安装一个不同的版本.有没有可能得到一个不同的版本与自制软件？

Answer 1

运行这些命令(假设你已经通过Homebrew安装了apache-spark )

cd "$(brew --repo homebrew/core)"
git log Formula/apache-spark.rb

例如：2.2.0版本：

...

提交bdf68bd79ebd16a70b7a747e027afbe5831f9cc3

作者: ilovezfs

日期: Tue Jul 11 22:19:12 2017 -0700

apache-spark 2.2.0 (#15507)

……

git checkout -b  apache-spark-2.2.0 bdf68bd79ebd16a70b7a747e027afbe5831f9cc3
brew unlink apache-spark
HOMEBREW_NO_AUTO_UPDATE=1 brew install apache-spark

清理

git checkout master
git branch -d apache-spark-2.2.0 

检查/切换：

brew list apache-spark --versions
brew switch apache-spark 2.2.0

Answer 2

我也有同样的问题，当我通过自制软件安装时，默认情况下它只能找到apache-spark 2.3.0公式，即使删除了repos也找不到2.2.0。

因此，我备份了path: /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula中现有的apache-spark.rb (版本2.3.0)，然后改写为：

​

class ApacheSpark < Formula
  desc "Engine for large-scale data processing"
  homepage "https://spark.apache.org/"
  url "https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz"
  version "2.2.0"
  sha256 "97fd2cc58e08975d9c4e4ffa8d7f8012c0ac2792bcd9945ce2a561cf937aebcc"
  head "https://github.com/apache/spark.git"

  bottle :unneeded

  def install
    # Rename beeline to distinguish it from hive's beeline
    mv "bin/beeline", "bin/spark-beeline"

    rm_f Dir["bin/*.cmd"]
    libexec.install Dir["*"]
    bin.write_exec_script Dir["#{libexec}/bin/*"]
  end

  test do
    assert_match "Long = 1000", pipe_output(bin/"spark-shell", "sc.parallelize(1 to 1000).count()")
  end
end

​

然后按照上面的流程重新安装，我有2.2.0和2.3.0的开关设施。

希望能有所帮助。

Answer 3

我需要在我的MacBook上特别安装Apache Spark 2.4.0版本。但在Brew列表中不再可用，但您仍然可以使用它。

安装最新的Spark by brew install apache-spark。假设它安装了apache-spark-3.0.1

完成后，执行brew edit apache-spark并编辑Pachecos-spark.rb，如下所示

class ApacheSpark < Formula
  desc "Engine for large-scale data processing"
  homepage "https://spark.apache.org/"
  url "https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz"
  mirror "https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz"
  version "2.4.0"
  sha256 "c93c096c8d64062345b26b34c85127a6848cff95a4bb829333a06b83222a5cfa"
  license "Apache-2.0"
  head "https://github.com/apache/spark.git"

  bottle :unneeded

  depends_on "openjdk@8"

  def install
    # Rename beeline to distinguish it from hive's beeline
    mv "bin/beeline", "bin/spark-beeline"

    rm_f Dir["bin/*.cmd"]
    libexec.install Dir["*"]
    bin.install Dir[libexec/"bin/*"]
    bin.env_script_all_files(libexec/"bin", JAVA_HOME: Formula["openjdk@8"].opt_prefix)
  end

  test do
    assert_match "Long = 1000",
      pipe_output(bin/"spark-shell --conf spark.driver.bindAddress=127.0.0.1",
                  "sc.parallelize(1 to 1000).count()")
  end
end

现在使用brew uninstall apache-spark再次卸载spark，使用brew install apache-spark再次安装它

结果

% spark-shell
2021-02-09 19:27:11 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.0.17:4040
Spark context available as 'sc' (master = local[*], app id = local-1612927640472).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/
         
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.