Skip to main content
Demo Project
DocsBook DemoStart Free Trial

Critical: grpc UNAVAILABLE in recommendation service RPC fails with inactive channel

CRITICALError
Error Frequency
Error Details
First seen:3 days ago
Last seen:15 mins ago
Stack:python
Service:recommendation
Stack TraceTraceback (most recent call last):
Error message
<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14}"
>
Traceback (most recent call last):
  File "/venv/lib/python3.12/site-packages/opentelemetry/trace/__init__.py", line 589, in use_span
    yield span
  File "/venv/lib/python3.12/site-packages/opentelemetry/sdk/trace/__init__.py", line 1105, in start_as_current_span
    yield span
  File "/app/recommendation_server.py", line 95, in get_product_list
    cat_response = product_catalog_stub.ListProducts(demo_pb2.Empty())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 68, in __call__
    return self._interceptor.intercept_unary(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 190, in intercept_unary
    return self._intercept(request, metadata, client_info, invoker)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 171, in _intercept
    raise exc
  File "/venv/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/_client.py", line 157, in _intercept
    result = invoker(request, metadata)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/grpcext/_interceptor.py", line 58, in invoker
    return self._base_callable(
           ^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1166, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/grpc/_channel.py", line 996, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14}"
>
Activity
User journey22 events before error
Jul 8, 20:09:27GET
+0suser_get_recommendations
+0sGET
+0srouter frontend egress
+0sGET /api/recommendations
+0sGET /api/recommendations
+0sexecuting api route (pages) /api/recommendations
+0soteldemo.RecommendationService/ListRecommendations
+0sdns.lookup
+0stcp.connect
+0sget_product_list
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)" debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.63.167:8080: Failed to connect to remote host: connect: Connection refused (111)", grpc_status:14}" >
+0soteldemo.RecommendationService/ListRecommendations
+0sflagd.evaluation.v1.Service/ResolveBoolean
+0sflagd.evaluation.v1.Service/ResolveBoolean
+0smessage
+0sresolveBoolean
+0sfeature_flag.evaluation
+0sexception
+0sexception
+0sexception
+0sexception
+0sexception
Issue events

Investigation

FirstRecent

Trace Breakdown

Jul 08, 20:09:27 UTC - Jul 08, 20:15:16 UTC
17Spans
0Errors
5.8 mDuration
Service / Span
load-generatoruser_get_recommendations
load-generatorGET /
frontend-proxyGET /
frontend-proxyrouter frontend egress
frontendGET /api/recommendations
frontendGET /api/recommendations
!frontendexecuting api route (pages) /api/recommendations
frontendoteldemo.RecommendationService/ListRecommendations
frontenddns.lookup
frontendtcp.connect
!recommendationoteldemo.RecommendationService/ListRecommendations
!recommendationget_product_list
recommendationflagd.evaluation.v1.Service/ResolveBoolean
flagdflagd.evaluation.v1.Service/ResolveBoolean
flagdresolveBoolean
flagdresolveBoolean
!recommendationoteldemo.ProductCatalogService/ListProducts