在批量加载数据后不可用查询表,mmappedrigions.java返回assertionError

问题描述 投票:0回答:1
cqlsh> select * from article_infos limit 5; ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures: UNKNOWN from /10.1.1.93:7000" info={'consistency': 'ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 1, 'error_code_map': {'10.1.1.93': '0x0000'}}

当执行此类查询时,Cassandra日志打印:

ERROR [ReadStage-2] 2025-03-14 10:03:29,385 JVMStabilityInspector.java:70 - Exception in thread Thread[ReadStage-2,5,SharedPool] java.lang.AssertionError: 7671074 > 7340032 at org.apache.cassandra.io.util.MmappedRegions$State.floor(MmappedRegions.java:363) at org.apache.cassandra.io.util.MmappedRegions.floor(MmappedRegions.java:242) at org.apache.cassandra.io.util.MmapRebufferer.rebuffer(MmapRebufferer.java:40) at org.apache.cassandra.io.tries.Walker.<init>(Walker.java:75) at org.apache.cassandra.io.tries.ValueIterator.<init>(ValueIterator.java:96) at org.apache.cassandra.io.tries.ValueIterator.<init>(ValueIterator.java:80) at org.apache.cassandra.io.sstable.format.bti.PartitionIndex$IndexPosIterator.<init>(PartitionIndex.java:407) at org.apache.cassandra.io.sstable.format.bti.PartitionIterator.<init>(PartitionIterator.java:113) at org.apache.cassandra.io.sstable.format.bti.PartitionIterator.create(PartitionIterator.java:75) at org.apache.cassandra.io.sstable.format.bti.BtiTableReader.coveredKeysIterator(BtiTableReader.java:295) at org.apache.cassandra.io.sstable.format.bti.BtiTableScanner$BtiScanningIterator.prepareToIterateRow(BtiTableScanner.java:114) at org.apache.cassandra.io.sstable.format.SSTableScanner$BaseKeyScanningIterator.computeNext(SSTableScanner.java:264) at org.apache.cassandra.io.sstable.format.SSTableScanner$BaseKeyScanningIterator.computeNext(SSTableScanner.java:244) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.io.sstable.format.SSTableScanner.hasNext(SSTableScanner.java:206) at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:90) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:375) at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:187) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$4.hasNext(UnfilteredPartitionIterators.java:264) at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:90) at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:334) at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:201) at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:186) at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:48) at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:372) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2210) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2607) at org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) ERROR [Reference-Reaper] 2025-03-14 10:03:55,267 Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@1883489551:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Data.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1883489551:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Data.db was not released before the reference was garbage collected ERROR [Reference-Reaper] 2025-03-14 10:03:55,268 Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@34244591:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Partitions.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@34244591:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Partitions.db was not released before the reference was garbage collected ERROR [Reference-Reaper] 2025-03-14 10:03:55,268 Ref.java:243 - LEAK DETECTED: a reference (class org.apache.cassandra.io.util.FileHandle$Cleanup@713715721:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Rows.db) to class org.apache.cassandra.io.util.FileHandle$Cleanup@713715721:/scratch/USER/cassandra/data/PROJECT/article_infos-0e9822e0ff5111ef8c7267a1b8a131a2/da-3gok_1b58_3yj682gtxbd0q6ceck-bti-Rows.db was not released before the reference was garbage collected
我当前使用Kafka和DataStax Connector接收器将数据写入Cassandra,但是即使我直接使用插入语句,也不会浏览kafka。
不确定的时间后,错误通常会消失,我可以查询数据库。 我想知道是否有人知道这里发生了什么。我很好奇,如果有人知道:

如果可以避免此错误
如何在编写大量数据后如何知道数据库何时处于“良好”状态。即使假设上述查询成功,我已经观察到这个问题很多次,以至于它对基本数据完整性的信念动摇了。我知道卡桑德拉(Cassandra)即使完成数据编写完成后也会运行压实步骤,但是在上面的断言错误是令人不安的。

您描述的症状表明该节点已过载,并且无反应。这就是您在加载大量数据后不能查询它的原因。

如果您的群集的尺寸不正确,则在磁盘所能维持的吞吐量上加载数据就像进行DDOS攻击一样。例如,如果

commitlog/
    磁盘只能处理每秒10,000个IO(IOPS),但是您正在以15,000 IOPS插入数据,那么请求将在磁盘上排队,直到最终可以赶上。 同时,数据库在等待磁盘再次开始运行的同时变得无响应。这就是为什么它最终似乎再次运行的原因,或者正如您所说的那样:“错误通常会消失[不确定的时间后]。
  • 使用类比,想象一家快餐店,工人可以从每分钟5人服用订单。如果您是客户,如果队列中有5个或更少,等待时间是可以忍受的。但是,如果队列中有10个人,等待时间开始变得烦人。解决方案是让餐厅经理至少再增加一名工人来增加他们接受订单的能力。
  • 同样的事情也适用于您的群集。如果通常可以处理10,000个IOP的吞吐量,但正在发送15K IOPS,则有两个选择:
  • throttle负载,因此每秒仅发送10K请求。
通过添加50%的节点(例如,将5个节点添加到10个节点群集中)。
cassandra
1个回答
0
投票
没有魔力。您不能最大化磁盘。您无需进行任何调整即可击败物理。欢呼!

最新问题
© www.soinside.com 2019 - 2025. All rights reserved.