如何从bash shell脚本运行beeline和hive查询

Question

登录unix bash shell后，我可以手动执行以下步骤。

echo "Connecting beeline" 

beeline
!connect jdbc:hive2://a301-1234-1234.stm.XXX.com:10000/default;;ssl=true;sslTrustStore=/app/bds/cloudera_truststore.jks;sslTrustPassword=;principal=hive/_HOST@BDS.XXXX.COM

INSERT OVERWRITE DIRECTORY "/dev/ref/HIVE_EXPORT/" ROW FORMAT DELIMITED FIELDS TERMINATED BY "," ESCAPED BY "\\" SELECT * FROM test_ref_st.Daily_report  limit 10;

hadoop fs -get('/dev/ref/HIVE_EXPORT/000000_0', '/user/rj/hiveExtract.csv')
echo "Query result extracted "

我需要通过shell脚本test1.sh按顺序运行上述所有步骤，如下所示

bash-4.2$ sh -x test1.sh

然后它只运行到直线，其余的命令不会运行。当前输出：

bash-4.2$ sh test1.sh

Picked up JAVA_TOOL_OPTIONS:
Beeline version 1.1.0-cdh5.16.2 by Apache Hive
beeline>

Answer 1

Bash正在逐行处理您的脚本。它运行beeline并等待您的输入。您可以使用heredoc从脚本写入标准输入：

beeline <<EOF
!connect jdbc:hive2://a301-1234-1234.stm.XXX.com:10000/default;;ssl=true;sslTrustStore=/app/bds/cloudera_truststore.jks;sslTrustPassword=;principal=hive/_HOST@BDS.XXXX.COM
INSERT OVERWRITE DIRECTORY "/dev/ref/HIVE_EXPORT/" ROW FORMAT DELIMITED FIELDS TERMINATED BY "," ESCAPED BY "\\" SELECT * FROM test_ref_st.Daily_report  limit 10;
EOF

Answer 2

使用!connect将打开beeline外壳控制台。要使用beeline CLI命令，您可以在SHELL中执行以下操作：

#!/bin/bash

HIVE_CONN=jdbc:hive2://a301-1234-1234.stm.XXX.com:10000/default;;  ## limited for simplicity

echo "executing query using beeline"
beeline -u $HIVE_CONN -e "INSERT OVERWRITE DIRECTORY "/dev/ref/HIVE_EXPORT/" ROW FORMAT DELIMITED FIELDS TERMINATED BY "," ESCAPED BY "\\" SELECT * FROM test_ref_st.Daily_report  limit 10;"

... 
rest of your code

-e代表要执行的查询

有关beeline CLI的更多信息，请访问此处。