与cqlsh连接到在Kubernetes上运行的节点的间歇性问题
与cqlsh连接到在Kubernetes上运行的节点的间歇性问题
提问于 2022-10-26 18:01:58
在尝试使用cqlsh登录时，我的身份验证遇到了一些问题。我正在运行一个由3个节点组成的集群，它位于Kubernetes中的三个不同的物理节点上。在过去的一个月里，它一直像一种魅力一样滚动，但大约一周左右，它开始下降。下面您可以看到，当我尝试登录到不同的节点和响应。(a FYI -并不总是节点0有问题，我也看到节点1有同样的问题，但是节点0工作得很好。这个问题似乎类似于Cassandra PasswordAuthenticator causing timeout，但那里的建议没有帮助。
I have no name!@cassandra-0:/$ cqlsh -u cassandra -p abc123
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': OperationTimedOut('errors=Timed out creating connection (5 seconds), last_host=None',)})
I have no name!@cassandra-1:/$ cqlsh -u cassandra -p abc123
Python 2.7 support is deprecated. Install Python 3.6+ or set CQLSH_NO_WARN_PY2 to suppress this message.Connected to mycluster at 127.0.0.1:9042
[cqlsh 6.0.0 | Cassandra 4.0.0 | CQL spec 3.4.5 | Native protocol v5]
Use HELP for help.
cassandra@cqlsh>
I have no name!@cassandra-2:/$ cqlsh -u cassandra -p abc123
Python 2.7 support is deprecated. Install Python 3.6+ or set CQLSH_NO_WARN_PY2 to suppress this message.Connected to mycluster at 127.0.0.1:9042
[cqlsh 6.0.0 | Cassandra 4.0.0 | CQL spec 3.4.5 | Native protocol v5]
Use HELP for help.
cassandra@cqlsh>以下是节点0的日志
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 MessagingMetrics.java:206 - HINT_RSP messages were dropped in last 5000 ms: 0 internal and 4 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 8582 ms
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 MessagingMetrics.java:206 - HINT_REQ messages were dropped in last 5000 ms: 0 internal and 2 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 8582 ms
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:65 - Pool Name Active Pending Completed Blocked All Time Blocked
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - ReadStage 0 0 17086 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - CompactionExecutor 0 0 195522 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,090 StatusLogger.java:69 - MutationStage 0 0 88515 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - MemtableReclaimMemory 0 0 210 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - PendingRangeCalculator 0 0 13 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - GossipStage 0 0 660680 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - SecondaryIndexManagement 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - HintsDispatcher 2 0 37776 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - Repair-Task 0 0 5 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - Native-Transport-Requests 0 0 17111 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - RequestResponseStage 0 0 20 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - MemtableFlushWriter 0 0 210 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - PerDiskMemtableFlushWriter_0 0 0 210 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - MemtablePostFlush 0 0 255 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - Sampler 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - ValidationExecutor 0 0 41 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - ViewBuildExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,091 StatusLogger.java:69 - InternalResponseStage 0 0 151676 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:69 - AntiEntropyStage 0 0 249 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:69 - CacheCleanupExecutor 0 0 0 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:79 - CompactionManager 0 0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:91 - MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:101 - Cache Type Size Capacity KeysToSave
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:103 - KeyCache 11452 75497472 all
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:109 - RowCache 0 0 all
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:116 - Table Memtable ops,data
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,092 StatusLogger.java:119 - system_schema.columns 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.types 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.indexes 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.keyspaces 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.dropped_columns 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.aggregates 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.triggers 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.tables 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.views 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system_schema.functions 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.compaction_history 3,634
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.repairs 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.size_estimates 49344,1087744
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.table_estimates 98688,2339296
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.paxos 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.built_views 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.peer_events 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.peers_v2 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.peers 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.peer_events_v2 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.batches 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,093 StatusLogger.java:119 - system.transferred_ranges 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.transferred_ranges_v2 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.view_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.local 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.sstable_activity 229,2816
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.available_ranges_v2 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.available_ranges 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,094 StatusLogger.java:119 - system.prepared_statements 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.roles 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.role_members 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.resource_role_permissons_index 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.network_permissions 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,095 StatusLogger.java:119 - system_auth.role_permissions 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - system_distributed.parent_repair_history 10,418916
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - system_distributed.repair_history 29495,15967
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,100 StatusLogger.java:119 - system_distributed.view_build_status 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,101 StatusLogger.java:119 - system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2022-10-26 17:39:00,101 StatusLogger.java:119 - system_traces.events 0,0键空间复制
cassandra@cqlsh> describe keyspace system_auth;
CREATE KEYSPACE system_auth WITH replication = {'class':
'NetworkTopologyStrategy', 'datacenter1': '3'} AND durable_writes = true; Nodetool状态
I have no name!@cassandra-prod-2:/$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.233.92.33 3.63 MiB 256 100.0% xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx rack1
UN 10.233.96.184 4.2 MiB 256 100.0% xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx rack1
UN 10.233.90.48 3.24 MiB 256 100.0% xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx rack1我试着做了一些未成功的工具修复。
有人知道发生了什么事吗？
我尝试重新启动所有c*-节点(K8-集群中的底层节点)，运行nodetool修复，但没有成功。
回答 1
Stack Overflow用户
回答已采纳
发布于 2022-10-27 03:51:49
您描述的有时无法用cqlsh连接到集群的症状与其说是身份验证问题，不如说是节点没有响应的问题。
从您发布的日志条目中，它显示节点正在删除提示消息。如果您还记得，当副本不承认write_request_timeout_in_ms中的写入时，协调员(负责通过向所有副本发送突变来协调写入请求)存储“提示”。
“提示”包含没有写入的副本的IP以及突变有效负载。当副本恢复联机时，协调器从提示(在Cassandra中称为暗示切换)“重放”突变到副本。
丢弃的信息是一个更大问题的症状。节点丢弃消息，因为它们是重载的，这是一种在不能接受任何请求时进行负载释放的机制。
寻找GC暂停，这是另一个过载的症状。如果豆荚受到资源的限制，请考虑分配更多的RAM，这样就可以增大堆的大小。此外，还可以通过添加更多的节点(pods)来增加集群的容量。干杯!
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/74212276
复制相关文章
相似问题
腾讯云开发者
Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有
深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 
粤公网安备44030502008569号
腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号