Elasticsearch，Kibana看到速度慢，如何拆分索引？
Elasticsearch，Kibana看到速度慢，如何拆分索引？
提问于 2019-04-13 13:23:56
我正在为日志数据创建索引，每天大约有10万个文档被索引。
我发现通过kibana查看数据有点慢。
每天，像log-2019-04-11' is created, and I'm usinglog*`这样的索引作为索引模式来查看kibana中的数据。
我使用单节点，并且只更改了ES配置的Xms8g Xmm8g。
**编辑**
我知道有分片的概念，但我还没有接触(配置)任何与分片重新相关的东西(比如在elasticsearch.yml中)
GET /analytics-prod*/_stats/
{
"_shards" : {
"total" : 550,
"successful" : 275,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 5128749,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 4396388356
},总有一天数据大小大约是..
GET /analytics-prod-2019.04.14/_stats/ [55/1894]
{
"_shards" : {
"total" : 10,
"successful" : 5,
"failed" : 0
},
"_all" : {
"primaries" : {
"docs" : {
"count" : 68912,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 67331653
},我有两个比较，第一个是较快的一个，但如果可能的话，我想在较慢的一个上运行ES。
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 6
On-line CPU(s) list: 0-5
Thread(s) per core: 1
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz
Stepping: 10
CPU MHz: 800.041
CPU max MHz: 4100.0000
CPU min MHz: 800.0000
BogoMIPS: 5808.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 9216K
NUMA node0 CPU(s): 0-5
=== START OF INFORMATION SECTION ===
Device Model: ST2000DM006-2DM164
Serial Number: Z560A76X
LU WWN Device Id: 5 000c50 09286b712
Firmware Version: CC26
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Apr 15 19:01:32 2019 KST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled另一台计算机具有以下cpu / hdd
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
Stepping: 9
CPU MHz: 1694.098
CPU max MHz: 3400.0000
CPU min MHz: 1600.0000
BogoMIPS: 6799.84
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-3
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Blue
Device Model: WDC WD10EZEX-00RKKA0
Serial Number: WD-WMC1S5459395
LU WWN Device Id: 5 0014ee 0ae471bd6
Firmware Version: 80.00A80
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Apr 15 19:02:20 2019 KST
SMART support is: Available - device has SMART capability.
SMART support is: EnabledGET _cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.0.57 72 88 4 0.60 0.72 0.61 mdi * 3tf0hMb获取分析-prod-2019.04.01/_映射
它太大了，不能粘贴在这里，我保存了输出并打印了行数
$ cat a.json | wc -l 772
{
"analytics-prod-2019.04.01" : {
"mappings" : {
"doc" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"batch_no" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"data" : {
"properties" : {
"anonymousId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
},
"channel" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"context" : {
"properties" : {
"app" : {
"properties" : {
"build" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"namespace" : {
"type" : "text",
"namespace" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"device" : {
"properties" : {
"adTrackingEnabled" : {
"type" : "boolean"
},
"advertisingId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"id" : {
"type" : "text",
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"manufacturer" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"model" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"library" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"locale" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"network" : {。。它一直在运行，我粘贴了大约1/4的输出。
回答 1
Stack Overflow用户
发布于 2019-04-23 16:36:45
您的日常数据非常小(大约64MB)。对于这么小的索引，您不需要5个分片。分片的建议大小是10-40GB，这意味着您正在对节点进行过度分片。当对5个分片的10个索引执行搜索请求时，您需要转到总共50个分片(据我所见，您只有一个节点)。如果您将其减少到1个分片，那么它将只有10个分片。
如果您希望对数据执行更多的并行搜索请求，请尝试放置更多副本(如果您有6个节点，请尝试添加最多5个副本)。也许你可以尝试放入月度索引而不是每日索引，因为每天的100k文档非常少。
Java heap应该占你的内存的50% (如果你有内存的64GB，放Xms30g Xmm30g，不要放更多，因为Java指针- more info)。
此外，您还需要正确设置映射或模板。Elasticsearch为您做到了这一点，但我猜您没有对大多数字段进行任何全文搜索分析，因此您可以将它们作为keyword放入预定义的映射- more info中
最后，您应该使用SSD而不是旋转磁盘来执行更多、更快的I/O操作。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/55662312
复制相关文章
相似问题
腾讯云开发者
Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有
深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 
粤公网安备44030502008569号
腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号