java.io.IOException:没有用于方案的FileSystem : maprfs

Question

我正在尝试执行/运行一个web应用程序(使用Spring工具套件)，通过它可以将作业提交给hadoop。我使用的是apache-tomcat-6.0.36服务器和mapr客户端。然而，当我运行我的应用程序时，我会发现以下错误：

java.io.IOException: No FileSystem for scheme: maprfs
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1534)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:69)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1571)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1553)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:232)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
at com.aexp.lookalike.PreInitialize.createVarFiles(PreInitialize.java:50)
at com.aexp.lookalike.PreInitialize.run(PreInitialize.java:98)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.springframework.data.hadoop.mapreduce.ToolExecutor.invokeTargetObject(ToolExecutor.java:31)
at org.springframework.data.hadoop.mapreduce.ToolExecutor.invokeTargetObject(ToolExecutor.java:27)
at org.springframework.data.hadoop.mapreduce.HadoopCodeExecutor.invokeTarget(HadoopCodeExecutor.java:185)
at org.springframework.data.hadoop.mapreduce.HadoopCodeExecutor.runCode(HadoopCodeExecutor.java:102)
at org.springframework.data.hadoop.mapreduce.ToolTasklet.execute(ToolTasklet.java:33)
at org.springframework.data.hadoop.mapreduce.ToolTasklet$$FastClassByCGLIB$$317c6986.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at org.springframework.data.hadoop.mapreduce.ToolTasklet$$EnhancerByCGLIB$$4b232693.execute(<generated>)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:281)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

我在配置时遗漏了什么吗？造成这一错误的原因是什么？我是否错过了在应用程序的类路径中添加任何一个jars？

Answer 1

问题在于您的类路径没有mapr jars。确保CLASSPATH和LIBRARY_PATH设置是正确的

CLASSPATH应该包含$mapr_HOME/lib/maprfs-1.0.3MapR-3.0.jar“LIB_PATH应该包含/opt/mapr/lib

Answer 2

我得到了类似的例外，但"hdfs“。我发现我使用带有依赖项的maven创建的应用程序jar中没有hadoop附带的hdfs的“服务”规范。我使用maven阴影插件(maven- shade plugin )解决了它，使用它内置的变压器"org.apache.maven.plugins.shade.resource.ServicesResourceTransformer".。

我认为没有阴影插件的maven不会为hdfs FileSystem所需的"org.apache.hadoop.fs.FileSystem“提供hdfs服务规范。我猜"maprfs“是另一个需要在METAINF/services文件夹中指定的文件系统。试着手动提供。您可以从hadoop获得它。

希望这能有所帮助。