pyspark将数据帧写入hbase，整数值以字节形式加载

Question

当通过pyspark将数据帧写入hbase时，默认情况下，当将数据帧写入hbase表中将整数值转换为字节类型时，我们是否有任何选项将整数值转换为整数？

Below is the code:
    catalog2 = {
        "table": {"namespace": "default","name": "trip_test1"},
        "rowkey": "key1",
        "columns": {
        "serial_no": {"cf": "rowkey","col": "key1","type": "string"},
        "payment_type": {"cf": "sales","col": "payment_type","type":"string"},
        "fare_amount": {"cf": "sales","col": "fare_amount","type": "string"},
        "surcharge": {"cf": "sales","col": "surcharge","type": "string"},
        "mta_tax": {"cf": "sales","col": "mta_tax","type": "string"},
        "tip_amount": {"cf": "sales","col": "tip_amount","type": "string"},
        "tolls_amount": {"cf": "sales","col": "tolls_amount","type":"string"},
        "total_amount": {"cf": "sales","col": "total_amount","type": "string"}
    }
}

import json

cat2=json.dumps(catalog2)

df.write.option("catalog",cat2).option("newtable","5").format("org.apache.spark.sql.execution.datasources.hbase").save()

输出：

\x00\x00\x03\xE7 column=sales:payment_type, timestamp=1529495930994, value=CSH
\x00\x00\x03\xE7 column=sales:surcharge, timestamp=1529495930994, value=\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x03\xE7 column=sales:tip_amount, timestamp=1529495930994, value=\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x03\xE7 column=sales:tolls_amount, timestamp=1529495930994, value=\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x03\xE7 column=sales:total_amount, timestamp=1529495930994, value=@!\x00\x00\x00\x00\x00\x00
\x00\x00\x03\xE8 column=sales:fare_amount, timestamp=1529495930994, value=@\x18\x00\x00\x00\x00\x00\x00
\x00\x00\x03\xE8 column=sales:mta_tax, timestamp=1529495930994, value=?\xE0\x00\x00\x00\x00\x00\x00

预期输出：

999 column=sales:fare_amount, timestamp=1529392479358, value=8.0
999 column=sales:mta_tax, timestamp=1529392479358, value=0.5
999 column=sales:payment_type, timestamp=1529392479358, value=CSH
999 column=sales:surcharge, timestamp=1529392479358, value=0.0
999 column=sales:tip_amount, timestamp=1529392479358, value=0.0
999 column=sales:tolls_amount, timestamp=1529392479358, value=0.0
999 column=sales:total_amount, timestamp=1529392479358, value=8.5

Answer 1

数值将转换为字节，然后存储在Hbase中。您必须使用相同的库(在您的示例中为"org.apache.spark.sql.execution.datasources.hbase")，以便在从Hbase读取数据时获得精确值。

如果你想在Hbase中将值存储为数字，请将列的数据类型转换为字符串类型，并将其存储为库"org.apache.spark.sql.execution.datasources.hbase“不会将字符串转换为字节。

请确保列值的数据类型与目录类型相同，以获得更好的结果。