使用新4j导入工具导入40M+数据时出错

Question

我使用neo4j-import导入4000万个节点，下面是我的shell：

[luning@pinnacle bin]$ ./neo4j-import --into ../data/weibo.db --nodes:User "/data/weibo/user-header.csv,/data/weibo/users/000000_0.csv,/data/weibo/users/000001_0.csv,/data/weibo/users/000002_0.csv,/data/weibo/users/000003_0.csv,/data/weibo/users/000004_0.csv,/data/weibo/users/000005_0.csv,/data/weibo/users/000006_0.csv,/data/weibo/users/000007_0.csv,/data/weibo/users/000008_0.csv,/data/weibo/users/000009_0.csv,/data/weibo/users/000010_0.csv,/data/weibo/users/000011_0.csv,/data/weibo/users/000012_0.csv,/data/weibo/users/000013_0.csv,/data/weibo/users/000014_0.csv,/data/weibo/users/000015_0.csv,/data/weibo/users/000016_0.csv,/data/weibo/users/000017_0.csv,/data/weibo/users/000018_0.csv,/data/weibo/users/000019_0.csv,/data/weibo/users/000020_0.csv,/data/weibo/users/000021_0.csv,/data/weibo/users/000022_0.csv,/data/weibo/users/000023_1.csv,/data/weibo/users/000024_0.csv,/data/weibo/users/000025_0.csv" --delimiter "TAB"

Nodes
[*>:87.20 MB/s---------------------------|PROPERTIES(2)===============|NOD|v:227.03 MB/s(2)====] 48MImport error: Panic called, so exiting

Neo4j Import Tool
    neo4j-import is used to create a new Neo4j database from data in CSV files. See 
    the chapter "Import Tool" in the Neo4j Manual for details on the CSV file format 
    - a special kind of header is required.
Usage:
--into <store-dir>
    Database directory to import into. Must not contain existing database.
--nodes [:Label1:Label2] "<file1>,<file2>,..."
    Node CSV header and data. Multiple files will be logically seen as one big file 
    from the perspective of the importer. The first line must contain the header. 
    Multiple data sources like these can be specified in one import, where each data 
    source has its own header. Note that file groups must be enclosed in quotation 
    marks.
--relationships [:RELATIONSHIP_TYPE] "<file1>,<file2>,..."
    Relationship CSV header and data. Multiple files will be logically seen as one 
    big file from the perspective of the importer. The first line must contain the 
    header. Multiple data sources like these can be specified in one import, where 
    each data source has its own header. Note that file groups must be enclosed in 
    quotation marks.
--delimiter <delimiter-character>
    Delimiter character, or 'TAB', between values in CSV data. The default option is 
    ,.
--array-delimiter <array-delimiter-character>
    Delimiter character, or 'TAB', between array elements within a value in CSV

我检查了他们的模式。它们都是一致的。它显示了

导入错误:惊慌调用，因此退出

有人知道怎么解决吗？

下面是我的工作记录：

    java.lang.RuntimeException: Panic called, so exiting
    at org.neo4j.unsafe.impl.batchimport.staging.StageExecution.stillExecuting(StageExecution.java:63)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisor.anyStillExecuting(ExecutionSupervisor.java:79)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisor.finishAwareSleep(ExecutionSupervisor.java:102)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisor.supervise(ExecutionSupervisor.java:64)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisors.superviseDynamicExecution(ExecutionSupervisors.java:65)
    at org.neo4j.unsafe.impl.batchimport.ParallelBatchImporter.executeStages(ParallelBatchImporter.java:226)
    at org.neo4j.unsafe.impl.batchimport.ParallelBatchImporter.doImport(ParallelBatchImporter.java:151)
    at org.neo4j.tooling.ImportTool.main(ImportTool.java:263)
Caused by: java.lang.RuntimeException: Panic called, so exiting
    at org.neo4j.unsafe.impl.batchimport.staging.AbstractStep.assertHealthy(AbstractStep.java:189)
    at org.neo4j.unsafe.impl.batchimport.staging.AbstractStep.await(AbstractStep.java:180)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutorServiceStep.receive(ExecutorServiceStep.java:82)
    at org.neo4j.unsafe.impl.batchimport.staging.AbstractStep.sendDownstream(AbstractStep.java:226)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutorServiceStep$2.call(ExecutorServiceStep.java:103)
    at org.neo4j.unsafe.impl.batchimport.staging.ExecutorServiceStep$2.call(ExecutorServiceStep.java:87)
    at org.neo4j.unsafe.impl.batchimport.executor.DynamicTaskExecutor$Processor.run(DynamicTaskExecutor.java:217)
Caused by: java.lang.RuntimeException: Panic called, so exiting
    ... 7 more
Caused by: java.lang.RuntimeException: Panic called, so exiting
    ... 7 more
Caused by: org.neo4j.unsafe.impl.batchimport.input.InputException: ERROR in input
  data source: BufferedCharSeeker[buffer:org.neo4j.csv.reader.SectionedCharBuffer@4ac5af5c, seekPos:2764030, line:2882236]
  in field: descriptions:string:4
  for header: [id:string, screenname:string, locations:string, descriptions:string, :IGNORE, profileimageurl:string, gender:string, followerscount:string, friendscount:string, statusescount:string, favouritescount:string, verified:string, verifiedreason:string, :IGNORE, :IGNORE, :IGNORE, :IGNORE, :IGNORE, :IGNORE, :IGNORE, darenint:string, :IGNORE, :IGNORE, updateddate:string]
  raw field value: 6:19:
  original error: Tried to read in a value larger than effective buffer size 8388608
    at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:152)
    at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:42)
    at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
    at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
    at org.neo4j.helpers.collection.NestingIterator.fetchNextOrNull(NestingIterator.java:61)
    at org.neo4j.helpers.collection.PrefetchingIterator.peek(PrefetchingIterator.java:60)
    at org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:46)
    at org.neo4j.unsafe.impl.batchimport.staging.IteratorBatcherStep.nextBatchOrNull(IteratorBatcherStep.java:54)
    at org.neo4j.unsafe.impl.batchimport.staging.InputIteratorBatcherStep.nextBatchOrNull(InputIteratorBatcherStep.java:42)
    at org.neo4j.unsafe.impl.batchimport.staging.ProducerStep.process(ProducerStep.java:73)
    at org.neo4j.unsafe.impl.batchimport.staging.ProducerStep$1.run(ProducerStep.java:54)
Caused by: java.lang.IllegalStateException: Tried to read in a value larger than effective buffer size 8388608
    at org.neo4j.csv.reader.BufferedCharSeeker.fillBufferIfWeHaveExhaustedIt(BufferedCharSeeker.java:258)
    at org.neo4j.csv.reader.BufferedCharSeeker.nextChar(BufferedCharSeeker.java:231)
    at org.neo4j.csv.reader.BufferedCharSeeker.seek(BufferedCharSeeker.java:109)
    at org.neo4j.unsafe.impl.batchimport.input.csv.InputEntityDeserializer.fetchNextOrNull(InputEntityDeserializer.java:81)
    ... 10 more

Answer 1

其中一个字段可能有一段引文没有结尾.因此CSV解析器将读取和读取，直到找到下一个引号。你不太可能有一个8米大的领域，所以我就是这么想的。

Answer 2

我也犯了同样的错误，并删除了"*“、"&”、"/“等特殊字符，但是保持单引号足以消除错误。

Answer 3

当我试图使用这种方法将数据导入到图形中时，我还得到了"Import : Executor已被关闭“和"Import :恐慌调用，所以退出”错误。

当我得到这些错误时，我的数据没有引号字符(“和')。

解决我问题的办法是摆脱所有其他特殊角色。

我可能忽略了文档中的一些内容，因为我认为节点属性中的所有文本都会作为字符串读入。原来neo4j-import不喜欢"&“和"/”这样的字符！

当我编辑我的数据时(yay sed!)为了只包含字母数字字符，导入工具工作得很好。