Spark-Shell启动错误

Question

在使用spark-1.6.0-bin-hadoop2.6启动spark-shell时，我看到了错误。这是刚刚出现的新行为。

以下日志消息中显示的故障的结果是sqlContext不可用(但sc可用)。

有没有可以释放的Derby锁？Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.

<console>:16: error: not found: value sqlContext
         import sqlContext.implicits._
                ^
<console>:16: error: not found: value sqlContext
         import sqlContext.sql

16/05/25 11:00:00 ERROR Schema: Failed initialising database.
Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.
org.datanucleus.exceptions.NucleusDataStoreException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.


16/05/25 11:06:02 WARN Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient


16/05/25 11:06:02 ERROR Schema: Failed initialising database.
Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@372e972d, see the next exception for details.
org.datanucleus.exceptions.NucleusDataStoreException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@372e972d, see the next exception for details.

Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        ... 134 more
Caused by: java.sql.SQLException: Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        ... 131 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.

Answer 1

当我尝试创建DataFrame时，我在Spark2.0.0 shell中遇到了类似的问题，只需删除metastore_db/dbex.lck，问题就解决了。

Answer 2

和安迪的答案类似。我在windows中也遇到了同样的问题，下面是解决方案：

1. 以管理员身份在windows上运行cmd
2. 导航到spark主目录
3. 打开spark-shell

c:\park\bin> spark-shell

Answer 3

解决这个问题的最好方法是首先重新启动系统，然后转到spark主目录，并在那里使用sudo用户运行spark-shell。

sudo bin/spark-shell

或者，如果要使用pyspark实例，则类型

sudo bin/pyspark

出现这个问题的主要原因是mertstore_db的权限不足。