Milvus 查询操作失败并出现“DEADLINE_EXCEEDED”错误

问题描述 投票：0回答：1

我在 Milvus 实例上运行查询时遇到 

DEADLINE_EXCEEDED
错误。简单查询操作时间歇性出现该错误。

这是重现该问题的最小代码：

from pymilvus import MilvusClient

client = MilvusClient(uri="http://localhost:19530")

res = client.query(
    collection_name="db0207",
    filter="",
    output_fields=["count(*)"]
)

操作失败并出现以下错误：

[2024-09-24 16:56:01,424] ERROR in decorators: grpc RpcError: [query], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-09-24 16:55:59.364782', 'gRPC error': '2024-09-24 16:56:01.424031'}>
[2024-09-24 16:56:01,424] ERROR in milvus_client: Failed to query collection: db0207

grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.DEADLINE_EXCEEDED
        details = "Deadline Exceeded"
        debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2024-09-24T16:56:01.365234577+07:00", grpc_status:4}"
>

我还有这个警告：

milvus-standalone  | [2024/09/24 11:54:00.288 +00:00] [WARN] [proxy/impl.go:3112] ["Query failed to WaitToFinish"] [traceID=bf9098c3e8473c3478f68e8c8af0152e] [role=proxy] [db=default] [collection=db0207] [partitions="[]"] [error="proxy TaskCondition context Done: context deadline exceeded"] [errorVerbose="proxy TaskCondition context Done: context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*TaskCondition).WaitToFinish\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/condition.go:45\n  | github.com/milvus-io/milvus/internal/proxy.(*Proxy).query\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3111\n  | github.com/milvus-io/milvus/internal/proxy.(*Proxy).Query\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:3171\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Query\n  | \t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:922\n  | github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Query_Handler.func1\n  | \t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/[email protected]/milvuspb/milvus.pb.go:12730\n  | github.com/milvus-io/milvus/internal/proxy/connection.KeepActiveInterceptor\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/connection/util.go:60\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.TraceLogInterceptor\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/trace_log_interceptor.go:36\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryUpdateAccessInfoInterceptor\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:47\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.RateLimitInterceptor.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/rate_limit_interceptor.go:55\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.UnaryServerInterceptor.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/privilege_interceptor.go:82\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.UnaryServerHookInterceptor.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:118\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy.DatabaseInterceptor.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/database_interceptor.go:15\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/auth/auth.go:47\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n  | \t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:342\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLogInterceptor\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:38\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\n  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n  | \t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34\n  | github.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Query_Handler\n  | \t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/[email protected]/milvuspb/milvus.pb.go:12732\n  | google.golang.org/grpc.(*Server).processUnaryRPC\n  | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:1360\n  | google.golang.org/grpc.(*Server).handleStream\n  | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:1737\n  | google.golang.org/grpc.(*Server).serveStreams.func1.1\n  | \t/go/pkg/mod/google.golang.org/[email protected]/server.go:982\nWraps: (2) proxy TaskCondition context Done\nWraps: (3) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) context.deadlineExceededError"]

问题：

  • 出现此
    DEADLINE_EXCEEDED
    错误的可能原因是什么？
  • 如何在我的应用程序中防止或处理此错误？
  • 我应该调整任何配置设置来避免这种超时吗？

其他上下文：此错误间歇性发生。该查询是一个简单的计数操作，错误发生在查询开始后大约 2 秒。

任何帮助或见解将不胜感激。谢谢！

vector-database milvus
1个回答
0
投票

此错误通常发生在需要扫描所有数据的查询操作期间，需要检查的重要事项：

查询超时设置：

  • 在计数操作扫描所有数据时检查超时配置

调试过程：

要正确解决此问题，请检查 Milvus 日志，例如：对于 Docker 安装：

docker-compose logs > milvus.log

此外，日志中的警告表明上下文截止日期问题：

proxy TaskCondition context Done: context deadline exceeded

为了尝试解决这个问题，我会说：

  • 检查超时设置：考虑使用更具体的查询，而不是计算所有记录。

该错误可能与查询执行时间超过默认超时有关，即使数据集很小也是如此。这可能是由于系统配置或资源限制造成的。

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.