#111678

Critical: UNKNOWN gRPC failure in frontend service call causing runtime exception

CRITICALError

Error Frequency

Error Details

First seen:3 days ago

Last seen:16 mins ago

Stack:nodejs

Service:frontend

Stack TraceError: 2 UNKNOWN: Exception calling application: <_InactiveRpcError of RPC that terminated with:

Error message

2 UNKNOWN: Exception calling application: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14}"
>

Error: 2 UNKNOWN: Exception calling application: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14}"
>
    at callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:32:19)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:193:76)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:361:141)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:324:181)
    at /app/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
    at process.processTicksAndRejections (node:internal/process/task_queues:85:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/app/node_modules/@grpc/grpc-js/build/src/client.js:161:32)
    at ServiceClientImpl.<anonymous> (/app/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at /app/node_modules/@opentelemetry/instrumentation-grpc/build/src/clientUtils.js:130:31
    at /app/node_modules/@opentelemetry/instrumentation-grpc/build/src/instrumentation.js:215:209
    at AsyncLocalStorage.run (node:internal/async_local_storage/async_hooks:91:14)
    at AsyncLocalStorageContextManager.with (/app/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at ContextAPI.with (/app/node_modules/@opentelemetry/api/build/src/api/context.js:60:46)
    at ServiceClientImpl.clientMethodTrace [as listRecommendations] (/app/node_modules/@opentelemetry/instrumentation-grpc/build/src/instrumentation.js:215:42)
    at /app/.next/server/chunks/[root-of-the-server]__b9429a7a._.js:1:1931
    at new Promise (<anonymous>)

Activity

User journey22 events before error

Jul 8, 20:09:27GET

+0suser_get_recommendations

+0sGET

+0srouter frontend egress

+0sGET /api/recommendations

+0sexecuting api route (pages) /api/recommendations

+0soteldemo.RecommendationService/ListRecommendations

+0sdns.lookup

+0stcp.connect

+0sget_product_list

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)" debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14}" >

+0soteldemo.RecommendationService/ListRecommendations

+0sflagd.evaluation.v1.Service/ResolveBoolean

+0smessage

+0sresolveBoolean

+0sfeature_flag.evaluation

+0sexception

Issue events

Investigation

First Recent

Trace Breakdown

Jul 08, 20:09:27 UTC - Jul 08, 20:15:16 UTC

17Spans

0Errors

5.8 mDuration

Service / Span

load-generatoruser_get_recommendations

load-generatorGET /

frontend-proxyGET /

frontend-proxyrouter frontend egress

frontendGET /api/recommendations

!frontendexecuting api route (pages) /api/recommendations

frontendoteldemo.RecommendationService/ListRecommendations

frontenddns.lookup

frontendtcp.connect

!recommendationoteldemo.RecommendationService/ListRecommendations

!recommendationget_product_list

recommendationflagd.evaluation.v1.Service/ResolveBoolean

flagdflagd.evaluation.v1.Service/ResolveBoolean

flagdresolveBoolean

!recommendationoteldemo.ProductCatalogService/ListProducts

Resource

Spans

Errors

Avg. Duration

Exec. Time

%Exec. Time

load-generator

—

229.4 ms

458.9 ms

24%

frontend-proxy

—

218.4 ms

436.8 ms

23%

frontend

105.7 ms

634.4 ms

34%

Time	Span name	Event type	Span kind	Exec. time
Jul 08, 20:09:27 UTC	GET /api/recommendations	HTTP		159.8 ms
Jul 08, 20:09:27 UTC	GET /api/recommendations	HTTP		158.1 ms
Jul 08, 20:09:27 UTC	executing api route (pages) /api/recommendations	HTTP		152.4 ms
Jul 08, 20:09:27 UTC	oteldemo.RecommendationService/ListRecommendations	GRPC		145.7 ms
Jul 08, 20:09:27 UTC	dns.lookup			16.3 ms
Jul 08, 20:09:27 UTC	tcp.connect			2.2 ms

recommendation

79.6 ms

318.6 ms

17%

Time	Span name	Event type	Span kind	Exec. time
Jul 08, 20:09:27 UTC	oteldemo.RecommendationService/ListRecommendations	GRPC		109.1 ms
Jul 08, 20:09:27 UTC	get_product_list			107.2 ms
Jul 08, 20:09:27 UTC	flagd.evaluation.v1.Service/ResolveBoolean	GRPC		18.3 ms
Jul 08, 20:09:27 UTC	oteldemo.ProductCatalogService/ListProducts	GRPC		84.1 ms

flagd

—

36.3 µs

108.9 µs

Critical: UNKNOWN gRPC failure in frontend service call causing runtime exception

Investigation

Trace Breakdown

ServicesExec Time %