谷歌云DataProc是否提供了一个带有默认JVM和YARN设置的网页？

Question

作为Apache Hadoop on AWS Elastic-Map-Reduce (EMR)服务的前用户，我习惯于从静态页面here1获取有关EMR集群中各种大小的VM的默认部署设置的信息。这些设置包括JVM最大内存大小、YARN调度器最小/最大内存分配、映射和减少最大内存等。

是否有类似的网页包含Google Cloud (GCP) DataProc服务的相应信息？我找过了，但找不到...

1

Answer 1

这类信息可通过DataProc命令行界面工具获得。如果需要查找默认值，请启动配置操作，然后使用该操作的ID执行

    gcloud dataproc operations describe ID

有关该命令的详细信息，请访问此处：https://cloud.google.com/sdk/gcloud/reference/dataproc/operations/describe

您将获得您所描述的设置类型以及更多信息。例如：

    capacity-scheduler:yarn.scheduler.capacity.root.default.ordering-policy: fair
    core:fs.gs.block.size: '134217728'
    core:fs.gs.metadata.cache.enable: 'false'
    distcp:mapreduce.map.java.opts: -Xmx768m
    distcp:mapreduce.map.memory.mb: '1024'
    distcp:mapreduce.reduce.java.opts: -Xmx768m
    distcp:mapreduce.reduce.memory.mb: '1024'
    hdfs:dfs.datanode.address: 0.0.0.0:9866
    hdfs:dfs.datanode.http.address: 0.0.0.0:9864
    hdfs:dfs.datanode.https.address: 0.0.0.0:9865
    hdfs:dfs.datanode.ipc.address: 0.0.0.0:9867
    hdfs:dfs.namenode.handler.count: '20'
    hdfs:dfs.namenode.http-address: 0.0.0.0:9870
    hdfs:dfs.namenode.https-address: 0.0.0.0:9871
    hdfs:dfs.namenode.lifeline.rpc-address: three-node-cluster-j6q2al2mkkqck-m:8050
    hdfs:dfs.namenode.secondary.http-address: 0.0.0.0:9868
    hdfs:dfs.namenode.secondary.https-address: 0.0.0.0:9869
    hdfs:dfs.namenode.service.handler.count: '10'
    hdfs:dfs.namenode.servicerpc-address: three-node-cluster-j6q2al2mkkqck-m:8051
    mapred-env:HADOOP_JOB_HISTORYSERVER_HEAPSIZE: '3840'
    mapred:mapreduce.job.maps: '21'
    mapred:mapreduce.job.reduce.slowstart.completedmaps: '0.95'
    mapred:mapreduce.job.reduces: '7'
    mapred:mapreduce.map.cpu.vcores: '1'
    mapred:mapreduce.map.java.opts: -Xmx2457m
    mapred:mapreduce.map.memory.mb: '3072'
    mapred:mapreduce.reduce.cpu.vcores: '1'
    mapred:mapreduce.reduce.java.opts: -Xmx2457m
    mapred:mapreduce.reduce.memory.mb: '3072'
    mapred:mapreduce.task.io.sort.mb: '256'
    mapred:yarn.app.mapreduce.am.command-opts: -Xmx2457m
    mapred:yarn.app.mapreduce.am.resource.cpu-vcores: '1'
    mapred:yarn.app.mapreduce.am.resource.mb: '3072'
    presto-jvm:MaxHeapSize: 12288m
    presto:query.max-memory-per-node: 7372MB
    presto:query.max-total-memory-per-node: 7372MB
    spark-env:SPARK_DAEMON_MEMORY: 3840m
    spark:spark.driver.maxResultSize: 1920m
    spark:spark.driver.memory: 3840m
    spark:spark.executor.cores: '2'
    spark:spark.executor.instances: '2'
    spark:spark.executor.memory: 5586m
    spark:spark.executorEnv.OPENBLAS_NUM_THREADS: '1'
    spark:spark.scheduler.mode: FAIR
    spark:spark.sql.cbo.enabled: 'true'
    spark:spark.yarn.am.memory: 640m
    yarn-env:YARN_TIMELINESERVER_HEAPSIZE: '3840'
    yarn:yarn.nodemanager.resource.memory-mb: '12288'
    yarn:yarn.scheduler.maximum-allocation-mb: '12288'
    yarn:yarn.scheduler.minimum-allocation-mb: '1024'