使用Hbase实现多表输出的Apache Gora Reducer

Question

我有一个小数据在Hbase表中爬行通过Nutch。它使用Apache Gora作为ORM。我在Hbase中找到了很多处理单表数据的例子(mapreduce)。但我的问题是，我必须将数据复制到多个表中(在reducer中)。没有Gora，有一些指南，例如，this question等，但如何为我的情况做它。

Answer 1

我从来没有按你的要求去做，但是你可能会在Gora Tutorial "Constructing the job" section中看到答案。在这里，有一个减速器配置的示例，其内容为：

/* Mappers are initialized with GoraMapper.initMapper() or 
 * GoraInputFormat.setInput()*/
GoraMapper.initMapperJob(job, inStore, TextLong.class, LongWritable.class
    , LogAnalyticsMapper.class, true);

/* Reducers are initialized with GoraReducer#initReducer().
 * If the output is not to be persisted via Gora, any reducer 
 * can be used instead. */
GoraReducer.initReducerJob(job, outStore, LogAnalyticsReducer.class);

然后，您可以只配置自己的缩减程序as told following your link (if it is a correct answer)，而不是使用GoraReducer.initReducerJob()

GoraMapper.initMapperJob(job, inStore, TextLong.class, LongWritable.class
    , LogAnalyticsMapper.class, true);
job.setOutputFormatClass(MultiTableOutputFormat.class);
job.setReducerClass(MyReducer.class);
job.setNumReduceTasks(2);
TableMapReduceUtil.addDependencyJars(job);
TableMapReduceUtil.addDependencyJars(job.getConfiguration());

我知道在前面的示例中，映射器发出(TextLong, LongWritable)键值，因此您的reducer应该类似于from the link you wrote和the answer

public class MyReducer extends TableReducer<TextLong, LongWritable, Put> {

    private static final Logger logger = Logger.getLogger( MyReducer.class );

    @SuppressWarnings( "deprecation" )
    @Override
    protected void reduce( TextLong key, Iterable<LongWritable> data, Context context ) throws IOException, InterruptedException {
        logger.info( "Working on ---> " + key.toString() );
        for ( Result res : data ) {
            Put put = new Put( res.getRow() );
            KeyValue[] raw = res.raw();
            for ( KeyValue kv : raw ) {
                put.add( kv );
            }

        ImmutableBytesWritable key = new ImmutableBytesWritable(Bytes.toBytes("tableName"));
        context.write(key, put);    

        }
    }
}

再说一次，我从来没这么做过..。所以可能不起作用:\