星星之火ETL和火花节约服务器

Question

一些细节：

• Spark (版本3.2.1)
• 驱动程序: Hive (Version2.3.9)

ThriftCLIService:使用5.500个工作线程在端口10000上启动ThriftBinaryCLIService

BI工具通过odbc驱动程序连接。

激活火花放电服务器后，我无法使用submit运行火花放电脚本，因为它们都使用相同的metastore_db

error:
Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@3acaa384, see the next exception for details.
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
        ... 140 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /tmp/metastore_db.

我需要能够运行PySpark (Spark )，同时为BI工具查询提供火花节约服务器。有什么解决办法吗？

谢谢!

Answer 1

在我的例子中，解决方案是将metastore_db移动到数据库服务器，比如MySql (在我的例子中)或Postgresql。

您必须配置$SPARK_HOME/conf/hive-site.xml，并将您的jdbc驱动程序包含在$SPARK_HOME/jars路径中。

用于MySQL连接的hivesite.xml示例

<configuration>
  <!-- Hive Execution Parameters -->
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://XXX.XXX.XXX.XXX:3306/metastore?createDatabaseIfNotExist=true&amp;useSSL=FALSE&amp;autoReconnect=true&amp;nullCatalogMeansCurrent=true</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>YOUR_USER</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>YOUR_PASSWORD</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.cj.jdbc.Driver</value>
</property>
<property>
  <name>hive.server2.transport.mode</name>
  <value>http</value>
</property>
<property>
  <name>hive.server2.thrift.http.port</name>
  <value>10000</value>
</property>
<property>
  <name>hive.server2.http.endpoint</name>
  <value>cliservice</value>
</property>


<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description/>
</property>
  <property>
    <name>datanucleus.autoCreateSchema</name>
    <value>true</value>
  </property>
  <property>
    <name>datanucleus.fixedDatastore</name>
    <value>true</value>
  </property>
  <property>
    <name>datanucleus.autoCreateTables</name>
    <value>true</value>
  </property>
  <property>
    <name>datanucleus.schema.autoCreateTables</name>
    <value>true</value>
  </property>

</configuration>