spring data hive与hive模板的集成

Question

我正在尝试使用spring data hadoop将hive集成到我的应用程序中，但遇到了一些问题。首先我不确定的是<hdp:hive-server host="some-other-host" port="10001" />是连接到现有的配置单元服务器，还是像创建一个新的配置单元服务器这样的东西，然后能够连接到它。其次，我的配置没有抛出任何错误，所以它看起来不错，甚至hiveTemplate自动装配也工作得很好，但是当我执行查询时，似乎没有得到任何响应。应用程序在这一点上有点卡住了。

以下是配置

<hive-client-factory host="${hive-${env}.server}" port="${hive-${env}.port}" />

<hive-template />

下面是我是如何使用它的

log.debug("before hive query");

for(String result : hiveTemplate.query("show tables;")){
    log.debug("=> " + result);
}

log.debug("after hive query");

我在日志输出中看到的都是before hive query。在那之后什么都不会发生。如果有任何帮助，我将不胜感激。我可能做错了什么。

Answer 1

端口number.

• Usually Thrift服务器使用端口号10000部署

1. Try 10000。检查您的安装是使用HiveServer2还是HiveServer。我能够让Spring Batch工作流与HiveServer一起工作，但我还没有成功地使用HiveServer2.
2. Ensure在运行您的程序

之前，您已经启动并运行了HiveServer/HiveServer

Answer 2

如果我们正在使用spring，请确保您的所有依赖项都与spring version.After兼容，这些all使用以下命令提供读写权限。

hadoop fs -mkdir /tmp
hadoop fs -chmod a+w /tmp
hadoop fs -mkdir -p /user/hive/warehouse
hadoop fs -chmod a+w /user/hive/warehouse

我在pom.xml中使用了以下版本的依赖项

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <artifactId>spring-hadoop-samples-hive</artifactId>

    <name>Spring Hadoop Samples - Hive</name>

    <parent>
        <groupId>org.springframework.samples</groupId>
        <artifactId>spring-hadoop-samples</artifactId>
        <version>1.0.0.BUILD-SNAPSHOT</version>
        <relativePath>../parent/pom.xml</relativePath>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <spring.hadoop.version>2.3.0.M1</spring.hadoop.version>
        <hadoop.version>2.7.1</hadoop.version>
        <hive.version>1.2.1</hive.version>
        <!-- <hive.version>2.1.1</hive.version> -->
    </properties>

    <dependencies>

        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-hadoop</artifactId>
            <version>${spring.hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework</groupId>
                    <artifactId>spring-context-support</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-jdbc</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-tx</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-metastore</artifactId>
            <version>${hive.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-service</artifactId>
            <version>${hive.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.thrift</groupId>
            <artifactId>libfb303</artifactId>
            <version>0.9.1</version>
        </dependency>

        <!-- runtime Hive deps start -->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-common</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>


        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-shims</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-serde</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-contrib</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>

        <!-- runtime Hive deps end -->

        <dependency>
            <groupId>org.codehaus.groovy</groupId>
            <artifactId>groovy</artifactId>
            <version>1.8.5</version>
            <scope>runtime</scope>
        </dependency>

    </dependencies>

    <repositories>
        <repository>
            <id>spring-milestone</id>
            <url>http://repo.spring.io/libs-milestone</url>
        </repository>
    </repositories>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>appassembler-maven-plugin</artifactId>
                <version>1.2.2</version>
                <configuration>
                    <repositoryLayout>flat</repositoryLayout>
                    <configurationSourceDirectory>src/main/config</configurationSourceDirectory>
                    <copyConfigurationDirectory>true</copyConfigurationDirectory>
                    <!-- Extra JVM arguments that will be included in the bin scripts -->
                    <extraJvmArguments>-Xms512m -Xmx1024m -Dhive.version=${hive.version}</extraJvmArguments>
                    <programs>
                        <program>
                            <mainClass>org.springframework.samples.hadoop.hive.HiveApp</mainClass>
                            <name>hiveApp</name>
                        </program>
                        <program>
                            <mainClass>org.springframework.samples.hadoop.hive.HiveClientApp</mainClass>
                            <name>hiveClientApp</name>
                        </program>
                        <program>
                            <mainClass>org.springframework.samples.hadoop.hive.HiveAppWithApacheLogs</mainClass>
                            <name>hiveAppWithApacheLogs</name>
                        </program>
                    </programs>
                </configuration>
                <executions>
                    <execution>
                        <id>package</id>
                        <goals>
                            <goal>assemble</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <executions>
                    <execution>
                        <id>config</id>
                        <phase>package</phase>
                        <configuration>
                            <tasks>
                                <copy todir="target/appassembler/data">
                                    <fileset dir="data"/>
                                </copy>
                            </tasks>
                        </configuration>
                        <goals>
                            <goal>run</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>