为什么orientdb oetl import给我这个错误

Question

我正在尝试将csv文件导入OrientDB3.0，我已经创建并测试了json文件，它可以处理较小的数据集。但是我想导入的数据集大约有10亿行(6列)

下面是我在导入oetl时使用的user.json文件

{
    "source": { "file": { "path": "d1.csv" } },
    "extractor": { "csv": {} },
    "transformers": [
      { "vertex": { "class": "User" } }
    ],
    "loader": {
      "orientdb": {
         "dbURL": "plocal:/databases/magriwebdoc",
         "dbType": "graph",
         "classes": [
           {"name": "User", "extends": "V"}
         ], "indexes": [
           {"class":"User", "fields":["id:string"], "type":"UNIQUE" }
         ]
      }
    }
  }

以下是oetl命令的控制台输出：

2019-05-22 14:31:15:484 INFO  Windows OS is detected, 262144 limit of open files will be set for the disk cache. [ONative]
2019-05-22 14:31:15:647 INFO  8261029888 B/7878 MB/7 GB of physical memory were detected on machine [ONative]
2019-05-22 14:31:15:647 INFO  Detected memory limit for current process is 8261029888 B/7878 MB/7 GB [ONative]
2019-05-22 14:31:15:649 INFO  JVM can use maximum 455MB of heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2019-05-22 14:31:15:649 INFO  Because OrientDB is running outside a container 12% of memory will be left unallocated according to the setting 'memory.leftToOS' not taking into account heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2019-05-22 14:31:15:650 INFO  OrientDB auto-config DISKCACHE=6,477MB (heap=455MB os=7,878MB) [orientechnologies]
2019-05-22 14:31:15:652 INFO  System is started under an effective user : `lenovo` [OEngineLocalPaginated]
2019-05-22 14:31:15:670 INFO  WAL maximum segment size is set to 6,144 MB [OrientDBEmbedded]
2019-05-22 14:31:15:701 INFO  BEGIN ETL PROCESSOR [OETLProcessor]
2019-05-22 14:31:15:703 INFO  [file] Reading from file d1.csv with encoding UTF-8 [OETLFileSource]
2019-05-22 14:31:15:703 INFO  Started execution with 1 worker threads [OETLProcessor]
2019-05-22 14:31:16:008 INFO  Page size for WAL located in D:\databases\magriwebdoc is set to 4096 bytes. [OCASDiskWriteAheadLog]
2019-05-22 14:31:16:703 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 1001ms [0 warnings, 0 errors] [OETLProcessor]
2019-05-22 14:31:16:770 INFO  Storage 'plocal:D:\databases/magriwebdoc' is opened under OrientDB distribution : 3.0.18 - Veloce (build 747595e790a081371496f3bb9c57cec395644d82, branch 3.0.x) [OLocalPaginatedStorage]
2019-05-22 14:31:17:703 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 2001ms [0 warnings, 0 errors] [OETLProcessor]
2019-05-22 14:31:17:954 SEVER ETL process has problem:  [OETLProcessor]
2019-05-22 14:31:17:956 INFO  END ETL PROCESSOR [OETLProcessor]
2019-05-22 14:31:17:957 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 2255ms [0 warnings, 0 errors] [OETLProcessor]D:\orientserver\bin>

我知道代码是正确的，但我认为这更多的是内存问题！

请告诉我该怎么做。

Answer 1

您是否尝试过根据要处理的数据大小来改进内存设置？

在文档中，您可以自定义以下属性：

• Configuration Environmental Variables (请参见parameter)
• Performance-Tuning - Memory Settings

$ORIENTDB_OPTS_MEMORY

也许能帮到你

Answer 2

您的json脚本似乎没有问题，但您可以尝试删除索引部分。由于索引错误，我也遇到了同样的问题。这可能是因为唯一索引的限制。您可以尝试：

1. 删除json脚本的索引部分。
2. 如果需要此索引，请确保在导入数据集之前清除数据库。