Routine schedule
PausedEvery day · No upcoming run
Execution policy: Observe
Routine instructions
Changes apply to future runs. Past runs keep the instructions they used.
check if there any any faulty metrics and notify us
Runs
SucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededSucceededRecent results
check if there any any faulty metrics and notify us
I can identify faulty telemetry from error-level events or error status codes, but notifying you requires a specific channel and explicit notification action. No notification was sent.
Found 221 error-level or error-status telemetry events. SYSTEM accounts for 186, followed by flagd with 13; recommendation, payment, fraud-detection, and ad each have 5, and otelcol-contrib has 6. Five events include grpc._channel._MultiThreadedRendezvous exceptions. No notification was sent because no notification channel or explicit send action was provided.
Error telemetry by service7
Error telemetry over time62
Exceptions in faulty telemetry2
7
3 tool calls
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 6.0
fraud-detection | 5.0
flagd | 13.0
ad | 5.0
SYSTEM | 186.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by attributes.exception.type"
→ Results (2 rows):
null | 221.0
grpc._channel._MultiThreadedRendezvous | 5.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (62 rows):
1.7898624e9 | 6.0
1.78986237e9 | 3.0
1.78986234e9 | 2.0
1.78986228e9 | 4.0
1.78986222e9 | 4.0
1.78986216e9 | 4.0
1.7898621e9 | 5.0
1.78986198e9 | 4.0
1.78986192e9 | 4.0
1.78986186e9 | 4.0
1.78986174e9 | 5.0
1.78986168e9 | 4.0
1.78986165e9 | 1.0
1.78986162e9 | 4.0
1.78986156e9 | 4.0
1.78986147e9 | 2.0
1.78986144e9 | 7.0
1.78986141e9 | 2.0
1.78986138e9 | 4.0
1.78986132e9 | 4.0
... +42 more
Five active incidents are currently emitting events, with the latest activity at 2026-09-20 15:30 UTC. There are also 132 open issues, including critical payment, checkout, frontend, accounting, product-catalog, and recommendation-service failures. Recent telemetry contains ERROR-level monitor.alert.triggered events from SYSTEM.
Recent error telemetry
4 tool calls
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-19T03:27:37.094818Z","environment":null,"id":"3df518a2-4b0f-449c-a5bf-6420326e28ce","recommended_action":"Review the changes and update your integration accordingly.","service":"payment","severity":"critical","title":"Critical: Payment Service Deadline Exceeded After 600 Seconds","type":"runtime_exception","updated_at":"2026-09-19T04:00:31.180255Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-17T20:01:50.948767Z","environment":null,"id":"0135838d-0c75-42ce-a2c2-0169121a10d7","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Checkout Service Logs POST /get-quote Request Errors","type":"log_pattern","updated_at":"2026-09-17T21:01:30.495109Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-13T03:54:27.36485Z","environment":null,"id":"27d072f1-74c4-49bc-a55b-c3d70b9c2092","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"Frontend GetAds RPC Fails with gRPC UNAVAILABLE Connection Error","type":"log_pattern","updated_at":"2026-09-13T08:05:58.672684Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-12T22:34:19.297676Z","environment":null,"id":"b5a10b8c-4459-4c0e-86ea-faa3c4541e15","recommended_action":"Review the changes and update your integration accordingly.","service":"accounting","severity":"critical","title":"Database Connection Timeout in Accounting Service to PostgreSQL","type":"log_pattern","updated_at":"2026-09-13T03:03:44.628946Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-11T00:13:23.873753Z","environment":null,"id":"ef19c43c-43a7
list_incidents
→ {"incidents":[{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-20T15:30:26.723785Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-20T15:30:24.102468Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-20T15:30:07.996988Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-20T15:30:20.951552Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"},{"closedAt":null,"id":"ac88ec9e-aac1-4618-a92b-a17ac5dae60d","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-20T15:30:02.957744Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:35.794135Z"}],"phase":"active"}
get_facets
→ No facet data available
sample_logsString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Sample logs:
- [ERROR] monitor.alert.triggered (SYSTEM): {..} d00cae46-195d-5a2f-bb3e-1ea06ab59411 2026-09-20T15:30:26.723786Z null monitor.alert.triggered 1
- [ERROR] monitor.alert.triggered (SYSTEM): {..} c3d00d93-dbcf-55d8-8d5a-207febd63392 2026-09-20T15:30:24.102469Z null monitor.alert.triggered 2
- [ERROR] monitor.alert.triggered (SYSTEM): {..} bffb0520-7ebf-5bda-8452-e03ce039d436 2026-09-20T15:30:20.951552Z null monitor.alert.triggered 3
- [ERROR] monitor.alert.triggered (SYSTEM): {..} 1007f476-0db2-58c8-8145-11a41695133b 2026-09-20T15:30:07.996988Z null monitor.alert.triggered 1
- [ERROR] monitor.alert.triggered (SYSTEM): {..} cd7194f4-4701-5242-8382-605cb7f600f4 2026-09-20T15:30:02.957745Z null monitor.alert.triggered 5
Active faults are present: 5 active incidents, 132 open issues, and 206 ERROR-level or ERROR-status telemetry events. Error volume is dominated by SYSTEM (168) and flagd (16), followed by recommendation, payment, fraud-detection, and ad (5 each). Critical open issues include payment deadline exceedances, checkout request errors, frontend gRPC/TCP/HTTP 500 failures, accounting database timeouts, and product-catalog database/RPC errors. The latest active incident activity was observed at 2026-09-21 15:31 UTC.
Errors by service7
Errors over time62
Recent error telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"b12f2834-cab9-45ea-bf0f-e31c686d8521","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-21T15:30:04.151512Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-21T10:45:02.686562Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-21T15:31:03.456938Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-21T15:30:04.775529Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-21T15:29:02.764141Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-21T15:30:11.404559Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-19T03:27:37.094818Z","environment":null,"id":"3df518a2-4b0f-449c-a5bf-6420326e28ce","recommended_action":"Review the changes and update your integration accordingly.","service":"payment","severity":"critical","title":"Critical: Payment Service Deadline Exceeded After 600 Seconds","type":"runtime_exception","updated_at":"2026-09-19T04:00:31.180255Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-17T20:01:50.948767Z","environment":null,"id":"0135838d-0c75-42ce-a2c2-0169121a10d7","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Checkout Service Logs POST /get-quote Request Errors","type":"log_pattern","updated_at":"2026-09-17T21:01:30.495109Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-13T03:54:27.36485Z","environment":null,"id":"27d072f1-74c4-49bc-a55b-c3d70b9c2092","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"Frontend GetAds RPC Fails with gRPC UNAVAILABLE Connection Error","type":"log_pattern","updated_at":"2026-09-13T08:05:58.672684Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-12T22:34:19.297676Z","environment":null,"id":"b5a10b8c-4459-4c0e-86ea-faa3c4541e15","recommended_action":"Review the changes and update your integration accordingly.","service":"accounting","severity":"critical","title":"Database Connection Timeout in Accounting Service to PostgreSQL","type":"log_pattern","updated_at":"2026-09-13T03:03:44.628946Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-11T00:13:23.873753Z","environment":null,"id":"ef19c43c-43a7
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
frontend-proxy | 1.0
fraud-detection | 5.0
flagd | 16.0
ad | 5.0
SYSTEM | 168.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (62 rows):
1.79000466e9 | 1.0
1.7900046e9 | 3.0
1.79000454e9 | 4.0
1.79000448e9 | 2.0
1.79000445e9 | 1.0
1.79000442e9 | 2.0
1.79000436e9 | 2.0
1.7900043e9 | 4.0
1.79000424e9 | 2.0
1.79000418e9 | 3.0
1.79000412e9 | 4.0
1.79000403e9 | 2.0
1.790004e9 | 6.0
1.79000388e9 | 5.0
1.79000382e9 | 5.0
1.79000376e9 | 4.0
1.79000367e9 | 1.0
1.79000364e9 | 4.0
1.79000358e9 | 5.0
1.79000346e9 | 4.0
... +42 more
Active faults are present: 6 active incidents, 129 open issues, and multiple critical issues updated today. The latest incident activity was at 2026-09-22 15:31 UTC. Notable active problems include payment deadline exceedances, checkout request errors, frontend gRPC failures, accounting database timeouts, and product-catalog database/RPC errors. Telemetry aggregation returned no rows, so current error volume by service and over time could not be confirmed from the query results.
Errors by service
Errors over time0
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"441d089c-dd09-44e1-acc3-200ebdd5c608","issueId":"adedd1ec-0951-497d-a504-4cd37e4b3056","lastEventAt":"2026-09-22T15:29:06.296532Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T15:27:05.877996Z"},{"closedAt":null,"id":"59c344f1-c526-42bf-81d2-cf0c4a6690e3","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-22T15:31:03.573471Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T15:15:03.74456Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-22T15:30:10.221206Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-22T15:31:09.976378Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-22T15:29:06.5599Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-22T15:31:06.77485Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-19T03:27:37.094818Z","environment":null,"id":"3df518a2-4b0f-449c-a5bf-6420326e28ce","recommended_action":"Review the changes and update your integration accordingly.","service":"payment","severity":"critical","title":"Critical: Payment Service Deadline Exceeded After 600 Seconds","type":"runtime_exception","updated_at":"2026-09-19T04:00:31.180255Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-17T20:01:50.948767Z","environment":null,"id":"0135838d-0c75-42ce-a2c2-0169121a10d7","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Checkout Service Logs POST /get-quote Request Errors","type":"log_pattern","updated_at":"2026-09-17T21:01:30.495109Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-13T03:54:27.36485Z","environment":null,"id":"27d072f1-74c4-49bc-a55b-c3d70b9c2092","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"Frontend GetAds RPC Fails with gRPC UNAVAILABLE Connection Error","type":"log_pattern","updated_at":"2026-09-13T08:05:58.672684Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-12T22:34:19.297676Z","environment":null,"id":"b5a10b8c-4459-4c0e-86ea-faa3c4541e15","recommended_action":"Review the changes and update your integration accordingly.","service":"accounting","severity":"critical","title":"Database Connection Timeout in Accounting Service to PostgreSQL","type":"log_pattern","updated_at":"2026-09-13T03:03:44.628946Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-11T00:13:23.873753Z","environment":null,"id":"ef19c43c-43a7
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (0 rows):
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (0 rows):
Five active incidents are present as of 2026-09-23 15:31 UTC. The critical “error alerts” incident affects 72 clients and 72 requests and is still receiving observations. Telemetry also contains 302 ERROR-level or ERROR-status events, led by SYSTEM (168) and otelcol-contrib (102); flagd has 12, while recommendation, payment, fraud-detection, and ad each have 5. Several active incidents are titled “Test alert” and may represent monitoring noise, but they remain unresolved.
Error telemetry by service
Recent error telemetry
7 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-23T15:30:04.452718Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-23T15:29:37.847914Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-23T15:31:02.57028Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-23T15:29:32.928622Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-23T15:28:48.187511Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
get_issue
→ {"activity":[{"created_at":"2026-09-23T15:30:04.452718Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:29:02.973272Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:28:01.947349Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:26:04.825755Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:25:03.329943Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:24:01.947026Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:22:04.520827Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:21:03.564236Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:20:02.130733Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:18:04.684199Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:17:03.066394Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:16:01.467109Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:14:04.574522Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:13:03.563597Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:12:02.080189Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:10:04.476215Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:09:03.3239Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:08:02.172923Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:06:04.318428Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:05:02.96056Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:04:01.895195Z","created_by":null,"event":"Episode EKObservati
get_issue
→ {"activity":[{"created_at":"2026-09-23T15:31:48.026612Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:29:37.847914Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:29:23.858455Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:26:13.934518Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:24:10.047287Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:22:15.176232Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:21:13.061625Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:21:13.019588Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:18:12.467986Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:17:11.498665Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:15:32.424411Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:13:11.301137Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:12:07.346417Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:10:09.512313Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:08:06.883846Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:06:09.528094Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:04:10.935702Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:02:07.350922Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:00:43.220913Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:59:05.681893Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:57:05.461566Z","created_by":null,"event":"Episode EKObserv
get_issue
→ {"activity":[{"created_at":"2026-09-23T15:31:02.57028Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:30:51.212983Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:28:37.728152Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:26:12.245409Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:24:09.08622Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:22:14.082773Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:21:11.652122Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:20:41.809434Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:18:11.763368Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:17:10.446152Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:16:07.031847Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:15:02.572665Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:13:10.505311Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:12:06.771782Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:10:08.33293Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:08:05.291274Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:06:08.173819Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:04:09.64539Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:02:05.583906Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:00:37.784585Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:59:04.732255Z","created_by":null,"event":"Episode EKObservatio
get_issue
→ {"activity":[{"created_at":"2026-09-23T15:29:32.928622Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:24:07.564685Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:18:08.908764Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:13:08.770156Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:08:03.813126Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:02:03.570122Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:57:03.2467Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:51:05.691989Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:45:33.793625Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:39:07.322946Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:34:03.150209Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:28:05.280906Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:22:07.062334Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:17:01.70856Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:11:06.380151Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:05:06.931534Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T13:59:06.342612Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T13:54:04.178468Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T13:49:01.052059Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T13:43:05.810205Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T13:37:31.625053Z","created_by":null,"event":"Episode EKObservati
get_issue
→ {"activity":[{"created_at":"2026-09-23T15:31:18.496461Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:28:48.187511Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:26:13.373046Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:24:09.632796Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:22:14.62497Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:21:12.394731Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:21:11.143293Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:18:10.810897Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:17:09.495338Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:16:04.418744Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:15:00.766685Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:13:10.106948Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:12:06.104276Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:10:07.895128Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:08:06.471647Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:06:08.836787Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:04:10.472846Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:02:06.863492Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T15:00:39.096211Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:59:05.127193Z","created_by":null,"event":"Episode EKObservation"},{"created_at":"2026-09-23T14:57:05.105453Z","created_by":null,"event":"Episode EKObserva
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 102.0
fraud-detection | 5.0
flagd | 12.0
ad | 5.0
SYSTEM | 168.0
Active faults are present. There are 5 active incidents, 133 open issues, and 20 open critical issues in the current result set. The newest critical findings are frontend-proxy 500 errors in router egress requests and 503 errors on GET /api/products/{id}. Ongoing high-impact issues also include payment requests exceeding 600 seconds, checkout quote failures, frontend gRPC failures, accounting database timeouts, and product-catalog database/RPC errors. Telemetry aggregation could not be completed because the telemetry database was starting up, so current error volume and trend counts are unavailable.
Error telemetry by service
Error telemetry over time
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-24T15:30:08.676914Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-24T15:30:08.71886Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-24T15:30:08.700411Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-24T15:28:29.591461Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-24T15:30:08.690971Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T13:52:08.098113Z","environment":null,"id":"26e1e48c-cb72-4e1c-87a1-9bd1125742c3","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New 500 Errors in Frontend Proxy Router Egress Requests","type":"log_pattern","updated_at":"2026-09-24T15:01:37.395341Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T04:10:22.877402Z","environment":null,"id":"5d81745a-01da-4203-8694-3e0e522bf35a","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New 503 Errors on GET /api/products/{id} in Frontend Proxy","type":"log_pattern","updated_at":"2026-09-24T05:01:15.079034Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-23T19:50:20.682037Z","environment":null,"id":"b6140b84-3e3c-4592-bf19-f2977d449432","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New 503 Errors on GET /api/products/{id} in Frontend Proxy","type":"log_pattern","updated_at":"2026-09-23T20:01:07.647643Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-23T06:54:39.821736Z","environment":null,"id":"7260738b-3f12-43a9-82e1-d51ef309e5ff","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New 503 Errors on GET /icons/Chevron.svg in Frontend Proxy","type":"log_pattern","updated_at":"2026-09-23T08:00:50.422338Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-19T03:27:37.094818Z","environment":null,"id":"3df518
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Error: Query execution failed - ! AnnotatedException !
Underlying exception type: HasqlException
Hasql connection error: Connection error while connecting to the database
reason: connection to server at "timefusion.s.past3.tech" (57.129.86.138), port 5432 failed: FATAL: the database system is starting up
Annotations:
* Annotation @([Char],Text) ("selectLogTable","SELECT jsonb_build_array(resource___service___name,count(*)::float, count(*) OVER()) FROM otel_logs_and_spans\n WHERE project_id='00000000-0000-0000-0000-000000000000' and timestamp BETWEEN '2026-09-24T14:31:48.548692965Z' AND '2026-09-24T15:31:48.548692965Z' and ((level = 'ERROR' OR status_code = 'ERROR'))\n GROUP BY resource___service___name ORDER BY resource___service___name desc limit 20 ")
CallStack (from HasCallStack):
checkpoint, called at src/Models/Apis/LogQueries.hs:312:9 in monoscope-0.1.0.0-inplace:Models.Apis.LogQueries
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Error: Query execution failed - ! AnnotatedException !
Underlying exception type: HasqlException
Hasql connection error: Connection error while connecting to the database
reason: connection to server at "timefusion.s.past3.tech" (57.129.86.138), port 5432 failed: FATAL: the database system is starting up
Annotations:
* Annotation @([Char],Text) ("selectLogTable","SELECT jsonb_build_array(extract(epoch from time_bucket('30 seconds', timestamp))::integer, count(*)::float, count(*) OVER())\n FROM otel_logs_and_spans\n WHERE project_id='00000000-0000-0000-0000-000000000000' and timestamp BETWEEN '2026-09-24T14:31:48.860083577Z' AND '2026-09-24T15:31:48.860083577Z' and ((level = 'ERROR' OR status_code = 'ERROR'))\n GROUP BY time_bucket('30 seconds', timestamp) \n ORDER BY time_bucket('30 seconds', timestamp) DESC\n limit 20 ")
CallStack (from HasCallStack):
checkpoint, called at src/Models/Apis/LogQueries.hs:312:9 in monoscope-0.1.0.0-inplace:Models.Apis.LogQueries
Active faults are present. There are 5 active incidents, 140 open issues, and 280 ERROR-level or ERROR-status telemetry events. Error telemetry is dominated by SYSTEM (165) and otelcol-contrib (79), followed by flagd (16), with recommendation, payment, fraud-detection, and ad at 5 each. Notable open critical issues include frontend 500/503 failures, product-catalog NOT_FOUND errors, checkout product lookup failures, payment requests exceeding 600 seconds, and accounting database timeouts. The newest active incident activity was observed at 2026-09-25 15:31 UTC.
Error events by service7
3 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-25T15:30:06.238726Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-25T15:28:41.270646Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-25T15:28:34.515895Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-25T15:26:10.110331Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-25T15:31:08.671641Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.636521Z","environment":null,"id":"515181da-58d3-4709-ae18-0ae828a6efb6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on Frontend Product Endpoint at 13.5s Duration","type":"log_pattern","updated_at":"2026-09-25T01:01:58.600215Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.590732Z","environment":null,"id":"791477bb-24e2-4d58-aa7b-c97b92a41bc0","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"ProductCatalogService GetProduct RPC Returns NOT_FOUND Errors","type":"log_pattern","updated_at":"2026-09-25T01:01:43.356739Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:51.404049Z","environment":null,"id":"a43e41fa-0935-4a88-81b6-ae1f7feaabdf","recommended_action":"Review the changes and update your integration accordingly.","service":"product-catalog","severity":"critical","title":"Product Catalog gRPC GetProduct Returns NOT_FOUND Error","type":"log_pattern","updated_at":"2026-09-25T01:01:30.137145Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:51.402598Z","environment":null,"id":"19cb5959-02bb-44fa-87d7-f411eaa7c893","recommended_action":"Review the changes and update your integration accordingly.","service":"product-catalog","severity":"critical","title":"New ERROR Pattern in Product Catalog ListProducts RPC","type":"log_pattern","updated_at":"2026-09-25T01:01:23.250451Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:48.218034Z","environment":null,"id":"ac87fc84-5c19-4
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 79.0
fraud-detection | 5.0
flagd | 16.0
ad | 5.0
SYSTEM | 165.0
Active faults are present: 5 active incidents and 141 open issues. The most recent incident activity was 2026-09-26 13:09 UTC. Notable open critical issues include frontend 500/503 failures, product-catalog and checkout product lookup failures, payment requests exceeding 600 seconds, and accounting database timeouts. The telemetry error query returned no rows, so current error volume could not be confirmed from telemetry.
Current error telemetry
Error-status telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.636521Z","environment":null,"id":"515181da-58d3-4709-ae18-0ae828a6efb6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on Frontend Product Endpoint at 13.5s Duration","type":"log_pattern","updated_at":"2026-09-25T01:01:58.600215Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.590732Z","environment":null,"id":"791477bb-24e2-4d58-aa7b-c97b92a41bc0","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"ProductCatalogService GetProduct RPC Returns NOT_FOUND Errors","type":"log_pattern","updated_at":"2026-09-25T01:01:43.356739Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:51.404049Z","environment":null,"id":"a43e41fa-0935-4a88-81b6-ae1f7feaabdf","recommended_action":"Review the changes and update your integration accordingly.","service":"product-catalog","severity":"critical","title":"Product Catalog gRPC GetProduct Returns NOT_FOUND Error","type":"log_pattern","updated_at":"2026-09-25T01:01:30.137145Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:51.402598Z","environment":null,"id":"19cb5959-02bb-44fa-87d7-f411eaa7c893","recommended_action":"Review the changes and update your integration accordingly.","service":"product-catalog","severity":"critical","title":"New ERROR Pattern in Product Catalog ListProducts RPC","type":"log_pattern","updated_at":"2026-09-25T01:01:23.250451Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:48.218034Z","environment":null,"id":"ac87fc84-5c19-4
count_queryString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Query 'level == "ERROR" OR status_code == "ERROR"' matches 0 entries
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (0 rows):
Active faults remain: 5 active incidents and 139 open issues, including critical frontend 500/503 errors, product-catalog and checkout product lookup failures, payment deadline exceedances after approximately 600 seconds, and accounting/database failures. Current telemetry shows ERROR or ERROR-status events from recommendation (4), payment (4), otelcol-contrib (14), fraud-detection (3), flagd (11), and ad (3). The latest active incident activity was observed on 2026-09-26 at 13:09 UTC. No notification was sent because the conversation does not specify a notification channel or explicitly request an external send action.
Current error events by service
Error events over time22
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.636521Z","environment":null,"id":"515181da-58d3-4709-ae18-0ae828a6efb6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on Frontend Product Endpoint at 13.5s Duration","type":"log_pattern","updated_at":"2026-09-26T20:19:21.971924Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.590732Z","environment":null,"id":"791477bb-24e2-4d58-aa7b-c97b92a41bc0","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"ProductCatalogService GetProduct RPC Returns NOT_FOUND Errors","type":"log_pattern","updated_at":"2026-09-25T01:01:43.356739Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:51.404049Z","environment":null,"id":"a43e41fa-0935-4a88-81b6-ae1f7feaabdf","recommended_action":"Review the changes and update your integration accordingly.","service":"product-catalog","severity":"critical","title":"Product Catalog gRPC GetProduct Returns NOT_FOUND Error","type":"log_pattern","updated_at":"2026-09-26T20:19:21.432914Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:51.402598Z","environment":null,"id":"19cb5959-02bb-44fa-87d7-f411eaa7c893","recommended_action":"Review the changes and update your integration accordingly.","service":"product-catalog","severity":"critical","title":"New ERROR Pattern in Product Catalog ListProducts RPC","type":"log_pattern","updated_at":"2026-09-26T20:19:21.098121Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:53:48.218034Z","environment":null,"id":"ac87fc84-5c19-4
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 4.0
payment | 4.0
otelcol-contrib | 14.0
fraud-detection | 3.0
flagd | 11.0
ad | 3.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (22 rows):
1.79052237e9 | 2.0
1.79052231e9 | 1.0
1.79052189e9 | 1.0
1.79052171e9 | 1.0
1.7905215e9 | 1.0
1.79052129e9 | 1.0
1.79052111e9 | 2.0
1.7905209e9 | 2.0
1.79052066e9 | 2.0
1.79052054e9 | 2.0
1.79052048e9 | 2.0
1.7905203e9 | 1.0
1.79052006e9 | 2.0
1.79051994e9 | 2.0
1.79051988e9 | 2.0
1.7905197e9 | 1.0
1.79051967e9 | 3.0
1.79051964e9 | 3.0
1.79051961e9 | 3.0
1.79051958e9 | 3.0
... +2 more
list_log_patterns
→ [{"first_seen_at":"2026-08-27T01:50:38.397184Z","id":12966361808,"is_error":false,"last_seen_at":"2026-09-27T15:31:50.772899Z","level":null,"occurrence_count":3917155,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> ms user <*>","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct attributes;text-textWeak⇒{\"demo\":{\"product\":{\"id\":\"L9ECAV7KIM\",\"name\":\"Lens Cleaning Kit\"}},\"rpc\":{\"method\":\"oteldemo.ProductCatalogService/GetProduct\",\"response\":{\"status_code\":\"OK\"},\"system\":{\"name\":\"grpc\"}},\"server\":{\"address\":\"10.42.0.175\",\"port\":8080},\"user\":{\"id\":\"49bc7e32-c888-4acf-b0a5-b16413a42f8e\"}} protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒3.4 ms user id;right-badge-neutral⇒49bc7e32-c888-4acf-b0a5-b16413a42f8e","service":"product-catalog","state":"acknowledged"},{"first_seen_at":"2026-08-31T15:03:04.078645Z","id":13484651015,"is_error":false,"last_seen_at":"2026-09-27T15:31:50.772899Z","level":null,"occurrence_count":203126,"pattern":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒ SELECT p.id, p.name, p.description, p.picture, p.price_currency_code, p.price_units, p.price_nanos, p.categories FROM catalog.products p ORDER BY p.id span_name;neutral⇒sql.conn.query db.system;right-badge-postgres⇒postgres duration;right-badge-neutral⇒<*> <*> <*> user <*>","sample_message":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒\n\t\tSELECT p.id, p.name, p.description, p.picture, \n\t\t p.price_currency_code, p.price_units, p.price_nanos, p.categories\n\t\tFROM catalog.products p\n\t\tORDER BY p.id\n\t span_name;neutral⇒sql.conn.query attributes;text-textWeak⇒{\"db\"
Active faults remain: 5 active incidents and 141 open issues, including 20 critical issues in the returned set. New critical findings today include a frontend-web 503 on GET /api/shipping and a payment deadline exceeded after 600 seconds. Telemetry contains 36 ERROR-level or ERROR-status events in the queried result, led by flagd (15), recommendation (5), payment (5), fraud-detection (5), and ad (4). Error activity is recurring across the returned time buckets rather than isolated to a single event.
Errors by service7
Error activity over time20
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T12:34:53.572247Z","environment":null,"id":"744c899d-3e7f-48d9-a4ee-d15ec8e94d5c","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on GET /api/shipping Endpoint in Frontend Web","type":"log_pattern","updated_at":"2026-09-28T13:01:02.385109Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T08:29:11.492244Z","environment":null,"id":"6996a9f9-55bc-4521-8a44-d5a70f08a39d","recommended_action":"Review the changes and update your integration accordingly.","service":"payment","severity":"critical","title":"Deadline Exceeded After 600s in Payment Service","type":"runtime_exception","updated_at":"2026-09-28T09:00:35.24767Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T07:23:41.024674Z","environment":null,"id":"1d7bd72a-19c2-4d9d-8eda-041bb7ab4051","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"HTTP 500 Errors on GET /api/data?contextKeys=accessories,telescopes,assembly","type":"log_pattern","updated_at":"2026-09-28T08:00:56.085413Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.636521Z","environment":null,"id":"515181da-58d3-4709-ae18-0ae828a6efb6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on Frontend Product Endpoint at 13.5s Duration","type":"log_pattern","updated_at":"2026-09-26T20:19:21.971924Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.590732Z","environment":null,"id":"791477b
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
frontend-web | 1.0
frontend-proxy | 1.0
fraud-detection | 5.0
flagd | 15.0
ad | 4.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (20 rows):
1.79060883e9 | 1.0
1.79060868e9 | 2.0
1.7906085e9 | 2.0
1.79060829e9 | 2.0
1.79060823e9 | 2.0
1.79060805e9 | 2.0
1.7906079e9 | 2.0
1.79060769e9 | 2.0
1.7906076e9 | 2.0
1.79060751e9 | 2.0
1.79060745e9 | 2.0
1.7906073e9 | 2.0
1.79060709e9 | 2.0
1.790607e9 | 1.0
1.79060682e9 | 2.0
1.7906067e9 | 1.0
1.79060646e9 | 2.0
1.7906064e9 | 2.0
1.79060622e9 | 2.0
1.7906061e9 | 1.0
Active faults remain: 5 active incidents, 144 open issues, and 22 open critical issues in the returned set. Newly updated critical findings include frontend-proxy HTTP 500 egress errors, frontend-web 503 errors on GET /api/shipping, frontend HTTP 500 errors, and a payment deadline exceeded after 600 seconds. Error telemetry is present for recommendation (5), payment (5), fraud-detection (5), flagd (13), and ad (4). An error log pattern in product-catalog has 2,779,538 occurrences and was observed as recently as 2026-09-29 15:31 UTC.
Error telemetry by service
Recent error telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T23:53:11.65099Z","environment":null,"id":"b0c8040e-f199-4a3d-9d6b-3e95636870b6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New HTTP 500 Egress Error Pattern in Frontend Proxy","type":"log_pattern","updated_at":"2026-09-29T01:00:52.42471Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T12:34:53.572247Z","environment":null,"id":"744c899d-3e7f-48d9-a4ee-d15ec8e94d5c","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on GET /api/shipping Endpoint in Frontend Web","type":"log_pattern","updated_at":"2026-09-28T13:01:02.385109Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T08:29:11.492244Z","environment":null,"id":"6996a9f9-55bc-4521-8a44-d5a70f08a39d","recommended_action":"Review the changes and update your integration accordingly.","service":"payment","severity":"critical","title":"Deadline Exceeded After 600s in Payment Service","type":"runtime_exception","updated_at":"2026-09-28T09:00:35.24767Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T07:23:41.024674Z","environment":null,"id":"1d7bd72a-19c2-4d9d-8eda-041bb7ab4051","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"HTTP 500 Errors on GET /api/data?contextKeys=accessories,telescopes,assembly","type":"log_pattern","updated_at":"2026-09-28T08:00:56.085413Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.636521Z","environment":null,"id":"515181da-58d3
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (5 rows):
recommendation | 5.0
payment | 5.0
fraud-detection | 5.0
flagd | 13.0
ad | 4.0
list_log_patterns
→ [{"first_seen_at":"2026-09-24T13:52:54.450813Z","id":14092192408,"is_error":false,"last_seen_at":"2026-09-29T15:31:44.761155Z","level":null,"occurrence_count":113502,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> <*> <*>","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct attributes;text-textWeak⇒{\"demo\":{\"product\":{\"id\":\"1YMWWN1N4O\",\"name\":\"Eclipsmart Travel Refractor Telescope\"}},\"rpc\":{\"method\":\"oteldemo.ProductCatalogService/GetProduct\",\"response\":{\"status_code\":\"OK\"},\"system\":{\"name\":\"grpc\"}},\"server\":{\"address\":\"10.42.0.175\",\"port\":8080},\"session\":{\"id\":\"0c0cf2bb-7fea-4db3-969c-c2656f4d042b\"}} protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒3.9 ms session;right-badge-neutral⇒0c0cf2bb-7fea-4db3-969c-c2656f4d042b","service":"product-catalog","state":"acknowledged"},{"first_seen_at":"2026-08-27T00:35:45.396324Z","id":12960587661,"is_error":false,"last_seen_at":"2026-09-29T15:31:44.761155Z","level":null,"occurrence_count":3876724,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> ms","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct attributes;text-textWeak⇒{\"demo\":{\"product\":{\"id\":\"66VCHSJNUP\",\"name\":\"Starsense Explorer Refractor Telescope\"}},\"rpc\":{\"method\":\"oteldemo.ProductCatalogService/GetProduct\",\"response\":{\"status_code\":\"OK\"},\"system\":{\"name\":\"grpc\"}},\"server\":{\"address\":\"10.42.0.1
Active faults remain: 5 active incidents, 148 open issues, and recurring ERROR or ERROR-status telemetry. The largest telemetry contributor is otelcol-contrib with 263 events, followed by flagd with 14 and recommendation, payment, fraud-detection, and ad with 5 each. Open critical issues include frontend-proxy HTTP 500 errors, frontend-web HTTP 503 errors on /api/shipping, payment requests exceeding 600 seconds, frontend HTTP 500 errors, and product-catalog/checkout product lookup failures. Error activity is recurring across the observed time buckets, with individual buckets reaching 9 events. The active incidents were last updated on 2026-09-26, so their data may be stale but they remain unresolved.
Error events by service6
Error events over time95
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T23:53:11.65099Z","environment":null,"id":"b0c8040e-f199-4a3d-9d6b-3e95636870b6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New HTTP 500 Egress Error Pattern in Frontend Proxy","type":"log_pattern","updated_at":"2026-09-29T01:00:52.42471Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T12:34:53.572247Z","environment":null,"id":"744c899d-3e7f-48d9-a4ee-d15ec8e94d5c","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on GET /api/shipping Endpoint in Frontend Web","type":"log_pattern","updated_at":"2026-09-28T13:01:02.385109Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T08:29:11.492244Z","environment":null,"id":"6996a9f9-55bc-4521-8a44-d5a70f08a39d","recommended_action":"Review the changes and update your integration accordingly.","service":"payment","severity":"critical","title":"Deadline Exceeded After 600s in Payment Service","type":"runtime_exception","updated_at":"2026-09-28T09:00:35.24767Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T07:23:41.024674Z","environment":null,"id":"1d7bd72a-19c2-4d9d-8eda-041bb7ab4051","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"HTTP 500 Errors on GET /api/data?contextKeys=accessories,telescopes,assembly","type":"log_pattern","updated_at":"2026-09-28T08:00:56.085413Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-24T23:54:03.636521Z","environment":null,"id":"515181da-58d3
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 263.0
fraud-detection | 5.0
flagd | 14.0
ad | 5.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (95 rows):
1.79078181e9 | 2.0
1.79078178e9 | 3.0
1.79078175e9 | 3.0
1.79078172e9 | 3.0
1.79078169e9 | 3.0
1.79078166e9 | 3.0
1.79078163e9 | 3.0
1.7907816e9 | 3.0
1.79078157e9 | 3.0
1.79078154e9 | 3.0
1.79078151e9 | 3.0
1.79078148e9 | 3.0
1.79078145e9 | 3.0
1.79078142e9 | 2.0
1.79078139e9 | 1.0
1.79078136e9 | 3.0
1.79078133e9 | 3.0
1.7907813e9 | 3.0
1.79078127e9 | 3.0
1.79078124e9 | 9.0
... +75 more
Active faults are present: 5 active incidents and 150 open issues. Current ERROR or ERROR-status telemetry is led by otelcol-contrib (138 events), followed by flagd (14), recommendation (5), payment (5), fraud-detection (4), ad (4), and frontend-proxy (1). Newly updated critical issues include ProductCatalogService NOT_FOUND errors in frontend, a recommendation-service error pattern, frontend-proxy HTTP 500 egress errors, frontend-web 503 errors on GET /api/shipping, and payment deadline exceedances after 600 seconds. An accounting database ERROR pattern has 62,518 occurrences and was observed as recently as 2026-10-01 15:31 UTC. No external notification was sent because a notification channel and explicit send request were not provided.
Error telemetry by service7
Current error telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:55.580689Z","environment":null,"id":"655fd4a7-c221-4a6b-b51b-a768469cb64d","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New ProductCatalogService GetProduct NOT_FOUND Error Pattern Detected","type":"log_pattern","updated_at":"2026-10-01T13:00:54.237818Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:45.561698Z","environment":null,"id":"34793dad-3be5-4edd-ba81-e82d0b3aee65","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-01T13:00:43.758522Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T23:53:11.65099Z","environment":null,"id":"b0c8040e-f199-4a3d-9d6b-3e95636870b6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New HTTP 500 Egress Error Pattern in Frontend Proxy","type":"log_pattern","updated_at":"2026-09-29T01:00:52.42471Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T12:34:53.572247Z","environment":null,"id":"744c899d-3e7f-48d9-a4ee-d15ec8e94d5c","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on GET /api/shipping Endpoint in Frontend Web","type":"log_pattern","updated_at":"2026-09-28T13:01:02.385109Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T08:29:11.492244Z","environment":null,"id":"
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 138.0
frontend-proxy | 1.0
fraud-detection | 4.0
flagd | 14.0
ad | 4.0
list_log_patterns
→ [{"first_seen_at":"2026-08-27T11:12:00.467653Z","id":13012905806,"is_error":false,"last_seen_at":"2026-10-01T15:31:44.259648Z","level":null,"occurrence_count":242538,"pattern":"request_type;neutral⇒outgoing status_code;badge-2xx⇒<*> method;badge-POST⇒POST route;neutral⇒/oteldemo.PaymentService/Charge rpc.method;neutral⇒/oteldemo.PaymentService/Charge","sample_message":"request_type;neutral⇒outgoing status_code;badge-2xx⇒200 method;badge-POST⇒POST route;neutral⇒/oteldemo.PaymentService/Charge rpc.method;neutral⇒/oteldemo.PaymentService/Charge attributes;text-textWeak⇒{\"apitoolkit\":{\"errors\":\"[]\",\"msg_id\":\"\",\"parent_id\":\"\",\"sdk_type\":\"JsGrpc\",\"service_version\":\"\",\"tags\":[]},\"demo\":{\"payment\":{\"amount\":\"919.95\"}},\"http\":{\"request\":{\"method\":\"POST\",\"path_params\":{},\"query_params\":{}},\"response\":{\"status_code\":200},\"route\":\"/oteldemo.PaymentService/Charge\"},\"net\":{\"host\":{\"name\":\"\"}},\"rpc\":{\"grpc\":{\"status_code\":0},\"method\":\"/oteldemo.PaymentService/Charge\",\"system\":\"grpc\"},\"server\":{\"address\":\"\"},\"session\":{\"id\":\"2974c0a2-8a2c-48e6-8c20-8690f6721161\"},\"url\":... protocol;right-badge-neutral⇒http protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒3.0 ms session;right-badge-neutral⇒2974c0a2-8a2c-48e6-8c20-8690f6721161 user id;right-badge-neutral⇒f217a563-1449-44c3-ae7a-3db361345394","service":"payment","state":"acknowledged"},{"first_seen_at":"2026-09-05T21:13:45.249057Z","id":13987652524,"is_error":false,"last_seen_at":"2026-10-01T15:31:44.259648Z","level":null,"occurrence_count":69816,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒Charge rpc.service;neutral⇒oteldemo.PaymentService span_name;neutral⇒grpc.oteldemo.PaymentService/Charge protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> <*> <*> user <*>","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒Charge rpc.service;neutral⇒oteldemo.PaymentService span_name;neutral⇒grpc.otelde
Active faults are present: 5 active incidents and 149 open issues, including critical issues updated on 2026-10-01 for frontend ProductCatalogService NOT_FOUND errors and a recommendation-service ERROR pattern. Current ERROR or ERROR-status telemetry includes otelcol-contrib (10), flagd (13), and 5 each from recommendation, payment, fraud-detection, and ad; frontend-proxy also has 1. Persistent error patterns include 65,674 accounting database errors and 728,367 quote /getquote events classified as errors, both observed on 2026-10-02. The active incidents have not emitted events since 2026-09-26 and may be stale, but remain unresolved. No external notification was sent because no channel or explicit send request was provided.
Errors by service7
Recent error events
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:55.580689Z","environment":null,"id":"655fd4a7-c221-4a6b-b51b-a768469cb64d","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New ProductCatalogService GetProduct NOT_FOUND Error Pattern Detected","type":"log_pattern","updated_at":"2026-10-01T13:00:54.237818Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:45.561698Z","environment":null,"id":"34793dad-3be5-4edd-ba81-e82d0b3aee65","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-01T13:00:43.758522Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T23:53:11.65099Z","environment":null,"id":"b0c8040e-f199-4a3d-9d6b-3e95636870b6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New HTTP 500 Egress Error Pattern in Frontend Proxy","type":"log_pattern","updated_at":"2026-09-29T01:00:52.42471Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T12:34:53.572247Z","environment":null,"id":"744c899d-3e7f-48d9-a4ee-d15ec8e94d5c","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on GET /api/shipping Endpoint in Frontend Web","type":"log_pattern","updated_at":"2026-09-28T13:01:02.385109Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T08:29:11.492244Z","environment":null,"id":"
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 10.0
frontend-proxy | 1.0
fraud-detection | 5.0
flagd | 13.0
ad | 5.0
list_log_patterns
→ [{"first_seen_at":"2026-03-09T10:25:00.909541Z","id":692440,"is_error":false,"last_seen_at":"2026-10-02T15:32:22.477933Z","level":"INFO","occurrence_count":971722,"pattern":"severity_text;badge-info⇒INFO Calculated quote","sample_message":"severity_text;badge-info⇒INFO Calculated quote attributes;text-textWeak⇒{\"context\":{\"total\":8.99},\"log\":{\"original\":\"Calculated quote\"}}","service":"quote","state":"acknowledged"},{"first_seen_at":"2026-08-27T01:02:45.615961Z","id":12962681459,"is_error":false,"last_seen_at":"2026-10-02T15:32:22.477933Z","level":null,"occurrence_count":400926,"pattern":"kind;neutral⇒internal <*> duration;right-badge-neutral⇒<*> µs session;right-badge-neutral⇒<*> user name;right-badge-neutral⇒Demo shopper <*> user id;right-badge-neutral⇒<*>","sample_message":"kind;neutral⇒internal span_name;neutral⇒{closure} attributes;text-textWeak⇒{\"code\":{\"file\":{\"path\":\"/var/www/vendor/php-di/slim-bridge/src/ControllerInvoker.php\"},\"function\":{\"name\":\"DI\\\\Bridge\\\\Slim\\\\ControllerInvoker::__invoke\"},\"line\":{\"number\":29}},\"server\":{\"address\":\"quote\"},\"session\":{\"id\":\"1aa4f8bb-0b19-43d4-8514-b0fb7c67b0bc\"},\"user\":{\"full_name\":\"Demo shopper 7bff70dd\",\"id\":\"7bff70dd-8fba-401a-b6b8-695918b4ff65\"}} duration;right-badge-neutral⇒443.0 µs session;right-badge-neutral⇒1aa4f8bb-0b19-43d4-8514-b0fb7c67b0bc user name;right-badge-neutral⇒Demo shopper 7bff70dd user id;right-badge-neutral⇒7bff70dd-8fba-401a-b6b8-695918b4ff65","service":"quote","state":"acknowledged"},{"first_seen_at":"2026-08-27T09:28:18.087592Z","id":13003482019,"is_error":true,"last_seen_at":"2026-10-02T15:32:22.477933Z","level":null,"occurrence_count":728367,"pattern":"request_type;neutral⇒incoming <*> method;badge-POST⇒POST route;neutral⇒/getquote","sample_message":"request_type;neutral⇒incoming status_code;badge-2xx⇒200 method;badge-POST⇒POST route;neutral⇒/getquote attributes;text-textWeak⇒{\"apitoolkit\":{\"errors\":\"[]\",\"msg_id\":\"ba4913c7-b317-
Active faults are present. There are 5 active incidents, 152 open issues, and current ERROR or ERROR-status telemetry across 6 services. The newest critical issue is a frontend-proxy 500 on POST /api/checkout, updated 2026-10-03. Other unresolved critical issues include frontend ProductCatalogService NOT_FOUND errors, recommendation-service errors, frontend-proxy 500/503 failures, frontend-web 503 errors on /api/shipping, and payment requests exceeding 600 seconds. Error telemetry is led by otelcol-contrib (91 events), followed by flagd (15), ad (7), and recommendation, payment, and fraud-detection (5 each). The active incidents are stale: their latest events were on 2026-09-26.
Current error telemetry by service6
Current error and error-status events
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-03T02:45:33.236141Z","environment":null,"id":"8301b93e-b3c7-4a65-8cd8-d53c11fd4e77","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New 500 Errors on POST /api/checkout in Frontend Proxy","type":"log_pattern","updated_at":"2026-10-03T03:00:41.727761Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:55.580689Z","environment":null,"id":"655fd4a7-c221-4a6b-b51b-a768469cb64d","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New ProductCatalogService GetProduct NOT_FOUND Error Pattern Detected","type":"log_pattern","updated_at":"2026-10-01T13:00:54.237818Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:45.561698Z","environment":null,"id":"34793dad-3be5-4edd-ba81-e82d0b3aee65","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-01T13:00:43.758522Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T23:53:11.65099Z","environment":null,"id":"b0c8040e-f199-4a3d-9d6b-3e95636870b6","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New HTTP 500 Egress Error Pattern in Frontend Proxy","type":"log_pattern","updated_at":"2026-09-29T01:00:52.42471Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T12:34:53.572247Z","environment":null,"id":"
run_queryString "(level == \"ERROR\" OR status_code == \"ERROR\") | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 91.0
fraud-detection | 5.0
flagd | 15.0
ad | 7.0
list_log_patterns
→ [{"first_seen_at":"2026-08-28T03:06:30.031018Z","id":13092688259,"is_error":false,"last_seen_at":"2026-10-03T15:31:46.082893Z","level":null,"occurrence_count":960596,"pattern":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒ SELECT p.id, p.name, p.description, p.picture, p.price_currency_code, p.price_units, p.price_nanos, p.categories FROM catalog.products p ORDER BY p.id span_name;neutral⇒sql.conn.query db.system;right-badge-postgres⇒postgres duration;right-badge-neutral⇒<*> <*> <*> user name;right-badge-neutral⇒Demo shopper <*> user <*>","sample_message":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒\n\t\tSELECT p.id, p.name, p.description, p.picture, \n\t\t p.price_currency_code, p.price_units, p.price_nanos, p.categories\n\t\tFROM catalog.products p\n\t\tORDER BY p.id\n\t span_name;neutral⇒sql.conn.query attributes;text-textWeak⇒{\"db\":{\"query\":{\"text\":\"\\n\\t\\tSELECT p.id, p.name, p.description, p.picture, \\n\\t\\t p.price_currency_code, p.price_units, p.price_nanos, p.categories\\n\\t\\tFROM catalog.products p\\n\\t\\tORDER BY p.id\\n\\t\"},\"system\":{\"name\":\"postgresql\"}},\"server\":{\"address\":\"astronomy-db\"},\"session\":{\"id\":\"093a77d6-9a07-4f84-8330-3c3b89103d02\"},\"user\":{\"full_name\":\"Demo shopper d4945d27\",\"id\":\"d4945d27-d746-411c-a148-2e5ae1a65cb6\"}} db.system;right-badge-postgres⇒postgres duration;right-badge-neutral⇒1.2 ms session;right-badge-neutral⇒093a77d6-9a07-4f84-8330-3c3b89103d02 user name;right-badge-neutral⇒Demo shopper d4945d27 user id;right-badge-neutral⇒d4945d27-d746-411c-a148-2e5ae1a65cb6","service":"product-catalog","state":"acknowledged"},{"first_seen_at":"2026-09-03T20:11:20.519479Z","id":13819663138,"is_error":false,"last_seen_at":"2026-10-03T15:31:46.082893Z","level":null,"occurrence_count":153572,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/ListProducts span_name;neutral⇒otelde
Active faults found: 5 active incidents and 154 open issues, including 22 critical issues in the returned set. The newest critical issue is a 503 on POST /api/checkout in frontend-web, updated 2026-10-04 14:00 UTC. Other notable unresolved faults include frontend-proxy 500s on POST /api/checkout, ProductCatalogService NOT_FOUND errors, recommendation-service errors, payment requests exceeding 600 seconds, and frontend-web 503s on /api/shipping. Current ERROR or ERROR-status telemetry totals 351 events across six services, led by otelcol-contrib with 320, followed by flagd with 12, and recommendation, payment, fraud-detection, and ad with 5, 4, 5, and 5 respectively. Active incidents appear stale, with latest events on 2026-09-26.
Error telemetry by service6
Error telemetry over time138
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-04T12:58:57.056216Z","environment":null,"id":"7d90aed5-fa9e-4193-9079-cb81ddaa0d29","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on POST /api/checkout in frontend-web","type":"log_pattern","updated_at":"2026-10-04T14:00:45.827876Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-03T02:45:33.236141Z","environment":null,"id":"8301b93e-b3c7-4a65-8cd8-d53c11fd4e77","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New 500 Errors on POST /api/checkout in Frontend Proxy","type":"log_pattern","updated_at":"2026-10-03T03:00:41.727761Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:55.580689Z","environment":null,"id":"655fd4a7-c221-4a6b-b51b-a768469cb64d","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New ProductCatalogService GetProduct NOT_FOUND Error Pattern Detected","type":"log_pattern","updated_at":"2026-10-01T13:00:54.237818Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:45.561698Z","environment":null,"id":"34793dad-3be5-4edd-ba81-e82d0b3aee65","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-01T13:00:43.758522Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-09-28T23:53:11.65099Z","environment":null,"id":"b0c8
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 5.0
payment | 4.0
otelcol-contrib | 320.0
fraud-detection | 5.0
flagd | 12.0
ad | 5.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp), resource.service.name"
→ Results (138 rows):
1.79112789e9 | otelcol-contrib | 1.0
1.79112786e9 | otelcol-contrib | 3.0
1.79112783e9 | otelcol-contrib | 3.0
1.7911278e9 | otelcol-contrib | 3.0
1.79112777e9 | otelcol-contrib | 3.0
1.79112774e9 | otelcol-contrib | 3.0
1.79112771e9 | otelcol-contrib | 3.0
1.79112768e9 | otelcol-contrib | 3.0
1.79112765e9 | otelcol-contrib | 3.0
1.79112762e9 | otelcol-contrib | 3.0
1.79112759e9 | otelcol-contrib | 3.0
1.79112756e9 | otelcol-contrib | 3.0
1.79112753e9 | otelcol-contrib | 3.0
1.7911275e9 | otelcol-contrib | 3.0
1.79112747e9 | otelcol-contrib | 3.0
1.79112744e9 | otelcol-contrib | 3.0
1.79112741e9 | otelcol-contrib | 3.0
1.79112738e9 | otelcol-contrib | 3.0
1.79112735e9 | otelcol-contrib | 3.0
1.79112732e9 | otelcol-contrib | 3.0
... +118 more
Active faults remain: 5 active incidents and 158 open issues, including newly updated critical checkout GET Quote errors and frontend-web 503 errors on POST /api/checkout. Other unresolved critical issues include frontend-proxy 500s, ProductCatalogService NOT_FOUND errors, recommendation failures, payment deadline exceedances, and shipping failures. Current telemetry contains ERROR or ERROR-status events from recommendation (5), payment (5), fraud-detection (5), flagd (11), and ad (5). The active incidents appear stale: their latest events were on 2026-09-26. No external notification was sent because no notification channel or explicit send action was provided.
Current error telemetry by service
Current error telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-05T10:50:00.050844Z","environment":null,"id":"a93fc1ce-b784-4384-9a4a-af5b466bea58","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"New ERROR Log Pattern on Checkout Service GET Quote Endpoint","type":"log_pattern","updated_at":"2026-10-05T11:01:00.910151Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-04T12:58:57.056216Z","environment":null,"id":"7d90aed5-fa9e-4193-9079-cb81ddaa0d29","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on POST /api/checkout in frontend-web","type":"log_pattern","updated_at":"2026-10-04T14:00:45.827876Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-03T02:45:33.236141Z","environment":null,"id":"8301b93e-b3c7-4a65-8cd8-d53c11fd4e77","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New 500 Errors on POST /api/checkout in Frontend Proxy","type":"log_pattern","updated_at":"2026-10-03T03:00:41.727761Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:55.580689Z","environment":null,"id":"655fd4a7-c221-4a6b-b51b-a768469cb64d","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New ProductCatalogService GetProduct NOT_FOUND Error Pattern Detected","type":"log_pattern","updated_at":"2026-10-01T13:00:54.237818Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:45.561698Z","environment":null,"id":"34793dad-3be5-4edd
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (5 rows):
recommendation | 5.0
payment | 5.0
fraud-detection | 5.0
flagd | 11.0
ad | 5.0
list_log_patterns
→ [{"first_seen_at":"2026-08-14T23:50:00.456683Z","id":11636727101,"is_error":false,"last_seen_at":"2026-10-05T15:31:46.557989Z","level":null,"occurrence_count":237538,"pattern":"request_type;neutral⇒incoming status_code;badge-2xx⇒<*> method;badge-POST⇒POST route;neutral⇒/send_order_confirmation protocol;right-badge-neutral⇒http duration;right-badge-neutral⇒<*> ms user <*>","sample_message":"request_type;neutral⇒incoming status_code;badge-2xx⇒<*> method;badge-POST⇒POST route;neutral⇒/send_order_confirmation protocol;right-badge-neutral⇒http duration;right-badge-neutral⇒<*> ms user <*>","service":"email","state":"acknowledged"},{"first_seen_at":"2026-06-28T08:36:33.809733Z","id":6848243653,"is_error":false,"last_seen_at":"2026-10-05T15:31:46.557989Z","level":null,"occurrence_count":740455,"pattern":"severity_text;badge-info⇒INFO Order confirmation email sent","sample_message":"severity_text;badge-info⇒INFO Order confirmation email sent attributes;text-textWeak⇒{\"demo\":{\"order\":{\"id\":\"d0d29d40-c0d1-11f1-a6ed-4ab050d02f17\"}},\"log\":{\"original\":\"Order confirmation email sent\"}}","service":"email","state":"acknowledged"},{"first_seen_at":"2026-09-07T03:10:52.278062Z","id":14058352685,"is_error":false,"last_seen_at":"2026-10-05T15:31:46.557989Z","level":null,"occurrence_count":100467,"pattern":"kind;neutral⇒internal span_name;neutral⇒sinatra.render_template duration;right-badge-neutral⇒<*> µs <*> user <*>","sample_message":"kind;neutral⇒internal span_name;neutral⇒sinatra.render_template attributes;text-textWeak⇒{\"session\":{\"id\":\"1373167e-0c72-4f11-bc3a-01420a4bfdbe\"},\"sinatra\":{\"template_name\":\"confirmation\"},\"user\":{\"id\":\"b8d4a56c-f56b-470e-b60e-f60315289af7\"}} duration;right-badge-neutral⇒642.8 µs session;right-badge-neutral⇒1373167e-0c72-4f11-bc3a-01420a4bfdbe user id;right-badge-neutral⇒b8d4a56c-f56b-470e-b60e-f60315289af7","service":"email","state":"acknowledged"},{"first_seen_at":"2026-08-31T23:26:39.919555Z","id":13527951595,"is_error":
Active faults remain. There are 5 active incidents, 160 open issues, and 238 ERROR-level or ERROR-status telemetry events across 8 services. Newly updated critical issues include frontend-web 503 errors on POST /api/cart and a checkout ERROR pattern on the GET Quote endpoint, both updated on 2026-10-05. Current error telemetry is dominated by otelcol-contrib (180), followed by frontend (16), flagd (14), frontend-proxy (8), and recommendation, payment, fraud-detection, and ad (5 each). The active incidents appear stale: their latest events were on 2026-09-26. No external notification was sent because no notification channel or explicit send request was provided.
Current error telemetry by service
Error telemetry summary
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-05T15:37:53.612471Z","environment":null,"id":"b207dea7-49fb-4b42-809b-98c9c5b0f386","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"New 503 Errors on POST /api/cart in Frontend Web","type":"log_pattern","updated_at":"2026-10-05T16:00:51.57245Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-05T10:50:00.050844Z","environment":null,"id":"a93fc1ce-b784-4384-9a4a-af5b466bea58","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"New ERROR Log Pattern on Checkout Service GET Quote Endpoint","type":"log_pattern","updated_at":"2026-10-05T11:01:00.910151Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-04T12:58:57.056216Z","environment":null,"id":"7d90aed5-fa9e-4193-9079-cb81ddaa0d29","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on POST /api/checkout in frontend-web","type":"log_pattern","updated_at":"2026-10-04T14:00:45.827876Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-03T02:45:33.236141Z","environment":null,"id":"8301b93e-b3c7-4a65-8cd8-d53c11fd4e77","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"New 500 Errors on POST /api/checkout in Frontend Proxy","type":"log_pattern","updated_at":"2026-10-03T03:00:41.727761Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-01T12:43:55.580689Z","environment":null,"id":"655fd4a7-c221-4a6b-b51b-a768469cb64d
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (8 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 180.0
frontend-proxy | 8.0
frontend | 16.0
fraud-detection | 5.0
flagd | 14.0
ad | 5.0
list_log_patterns
→ [{"first_seen_at":"2026-06-30T14:17:26.096139Z","id":7100943110,"is_error":false,"last_seen_at":"2026-10-06T15:31:44.559384Z","level":null,"occurrence_count":1286024,"pattern":"severity_text;badge-info⇒INFO [ProducerStateManager <*> Wrote producer snapshot at offset <*> with <*> producer ids in <*> ms.","sample_message":"severity_text;badge-info⇒INFO [ProducerStateManager partition=__cluster_metadata-0] Wrote producer snapshot at offset 7017350 with 0 producer ids in 1 ms. attributes;text-textWeak⇒{\"log\":{\"original\":\"[ProducerStateManager partition=__cluster_metadata-0] Wrote producer snapshot at offset 7017350 with 0 producer ids in 1 ms.\"}}","service":"kafka","state":"acknowledged"},{"first_seen_at":"2026-08-27T00:54:18.111368Z","id":12962040456,"is_error":false,"last_seen_at":"2026-10-06T15:31:44.559384Z","level":null,"occurrence_count":378024,"pattern":"[{YYYY-MM-DD} {HH:MM:SS},{integer}] INFO [LocalLog <*> <*> Rolled new log segment at offset <*> in <*> ms. <*>","sample_message":"[2026-10-06 15:31:13,128] INFO [LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Rolled new log segment at offset 7017350 in 1 ms. (org.apache.kafka.storage.internals.log.LocalLog) attributes;text-textWeak⇒{\"log\":{\"file\":{\"path\":\"/var/log/pods/default_kafka-74d75ffb4b-xm42k_0a677465-aafa-4871-82f0-88ea48f72cc1/kafka/0.log\"},\"iostream\":\"stdout\",\"original\":\"[2026-10-06 15:31:13,128] INFO [LocalLog partition=__cluster_metadata-0, dir=/tmp/kafka-logs/__cluster_metadata-0] Rolled new log segment at offset 7017350 in 1 ms. (org.apache.kafka.storage.internals.log.LocalLog)\"},\"logtag\":\"F\"}","service":"kafka","state":"acknowledged"},{"first_seen_at":"2026-05-07T09:41:16.904847Z","id":2238541576,"is_error":false,"last_seen_at":"2026-10-06T15:31:44.559384Z","level":null,"occurrence_count":1234155,"pattern":"[{YYYY-MM-DD} {HH:MM:SS},{integer}] INFO [SnapshotEmitter id={integer}] Successfully wrote snapshot {integer}-{integer} (org.apache.
Active faults remain. There are 5 active incidents, 165 open issues, and 356 ERROR-level or ERROR-status telemetry events across 6 services. The newest critical issue is a frontend-web 503 on POST /v1/traces, updated 2026-10-07 07:09 UTC. Other recent critical issues include frontend-web 503 errors on POST /api/cart, checkout GET Quote errors, and frontend-web/frontend-proxy checkout failures. Error telemetry is dominated by otelcol-contrib (326), followed by flagd (11), recommendation (5), payment (5), ad (5), and fraud-detection (4). A persistent product-catalog database error pattern has 2,144,258 occurrences and was observed at 2026-10-07 15:31 UTC. Active incidents appear stale, with their latest events on 2026-09-26.
Errors by service6
Recent error telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T01:12:53.297999Z","environment":null,"id":"b1b4eb2f-074a-4fe9-a687-8ec4d4ef36ce","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on POST /v1/traces in Frontend Web Service","type":"log_pattern","updated_at":"2026-10-07T07:09:50.315864Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-05T15:37:53.612471Z","environment":null,"id":"b207dea7-49fb-4b42-809b-98c9c5b0f386","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"New 503 Errors on POST /api/cart in Frontend Web","type":"log_pattern","updated_at":"2026-10-05T16:00:51.57245Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-05T10:50:00.050844Z","environment":null,"id":"a93fc1ce-b784-4384-9a4a-af5b466bea58","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"New ERROR Log Pattern on Checkout Service GET Quote Endpoint","type":"log_pattern","updated_at":"2026-10-05T11:01:00.910151Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-04T12:58:57.056216Z","environment":null,"id":"7d90aed5-fa9e-4193-9079-cb81ddaa0d29","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"503 Errors on POST /api/checkout in frontend-web","type":"log_pattern","updated_at":"2026-10-04T14:00:45.827876Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-03T02:45:33.236141Z","environment":null,"id":"8301b93e-b3c7-4a65-8cd8-d53c11fd4e77","
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 326.0
fraud-detection | 4.0
flagd | 11.0
ad | 5.0
list_log_patterns
→ [{"first_seen_at":"2026-09-05T21:17:01.581954Z","id":13987835526,"is_error":false,"last_seen_at":"2026-10-07T15:31:48.061989Z","level":null,"occurrence_count":941094,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> ms <*>","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct attributes;text-textWeak⇒{\"demo\":{\"product\":{\"id\":\"OLJCESPC7Z\",\"name\":\"National Park Foundation Explorascope\"}},\"rpc\":{\"method\":\"oteldemo.ProductCatalogService/GetProduct\",\"response\":{\"status_code\":\"OK\"},\"system\":{\"name\":\"grpc\"}},\"server\":{\"address\":\"10.42.0.175\",\"port\":8080},\"session\":{\"id\":\"3d2c72aa-f415-4b3b-95a8-07dab3ed6408\"}} protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒2.8 ms session;right-badge-neutral⇒3d2c72aa-f415-4b3b-95a8-07dab3ed6408","service":"product-catalog","state":"acknowledged"},{"first_seen_at":"2026-09-05T21:11:54.290724Z","id":13987574690,"is_error":false,"last_seen_at":"2026-10-07T15:31:48.061989Z","level":null,"occurrence_count":172748,"pattern":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒ SELECT p.id, p.name, p.description, p.picture, p.price_currency_code, p.price_units, p.price_nanos, p.categories FROM catalog.products p ORDER BY p.id span_name;neutral⇒sql.conn.query db.system;right-badge-postgres⇒postgres duration;right-badge-neutral⇒<*> <*> <*>","sample_message":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒\n\t\tSELECT p.id, p.name, p.description, p.picture, \n\t\t p.price_currency_code, p.price_units, p.price_nanos, p.categories\n\t\tFROM catalog.products p\n\t\tORDER BY p.id\n\t span_name;neutral⇒sql.conn.query attributes;text-textWeak
Active faults are present as of 2026-10-08 15:31 UTC: 6 active incidents, 181 open issues, and 228 ERROR-level or ERROR-status telemetry events across 7 services. The newest critical issues are a frontend shipping-quote EOF failure, recommendation-service errors, ProductCatalogService NOT_FOUND errors, checkout product lookup failures, and product-catalog INTERNAL/EOF errors, all updated on 2026-10-07–08. Error telemetry is dominated by otelcol-contrib (183), followed by flagd (14), SYSTEM (11), and recommendation, payment, fraud-detection, and ad (5 each). One active incident is still receiving events as of 15:31 UTC; the other five appear stale, with latest events on 2026-09-26. No external notification was sent because no notification channel was specified and no explicit send action was requested.
ERROR telemetry by service
Recent error telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-08T15:31:07.463638Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-08T02:10:48.556602Z","environment":null,"id":"b4079163-d3cf-45ff-9a33-235493ef5428","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New ERROR Pattern: PlaceOrder Fails on Shipping Quote EOF","type":"log_pattern","updated_at":"2026-10-08T03:00:37.474101Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:28.746082Z","environment":null,"id":"2c314505-843b-4909-9485-cb77494581d9","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-07T22:01:41.040459Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:18.342029Z","environment":null,"id":"ca69d65d-7cd9-443e-99e3-1c2966e43a6c","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"ProductCatalogService GetProduct RPC Returns NOT_FOUND Error","type":"log_pattern","updated_at":"2026-10-07T22:01:35.497467Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:18.322638Z","environment":null,"id":"97fac492-fb36-4e75-97b5-a8ec415ac242","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"Product Catalog GetProduct RPC Returns NOT_FOUND Errors in Frontend","type":"log_pattern","updated_at":"2026-10-07T22:01:30.531778Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:35:55.35698Z","environment":null,"id":"a
run_queryString "(level == \"ERROR\" OR status_code == \"ERROR\") | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 183.0
fraud-detection | 5.0
flagd | 14.0
ad | 5.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-08-27T09:18:42.343438Z","id":13002797008,"is_error":false,"last_seen_at":"2026-10-08T15:31:48.951187Z","level":null,"occurrence_count":110757,"pattern":"kind;neutral⇒internal span_name;neutral⇒executing api route (pages) <*> duration;right-badge-neutral⇒<*> ms session;right-badge-neutral⇒<*> user <*>","sample_message":"kind;neutral⇒internal span_name;neutral⇒executing api route (pages) /api/recommendations attributes;text-textWeak⇒{\"next\":{\"span_category\":\"nextjs\",\"span_name\":\"executing api route (pages) /api/recommendations\",\"span_type\":\"Node.runHandler\"},\"server\":{\"address\":\"frontend\"},\"session\":{\"id\":\"450aa77f-4da2-470b-8cc2-063642df508c\"},\"user\":{\"id\":\"f0f89e60-9085-42be-bea9-0ac482a7432a\"}} duration;right-badge-neutral⇒30.5 ms session;right-badge-neutral⇒450aa77f-4da2-470b-8cc2-063642df508c user id;right-badge-neutral⇒f0f89e60-9085-42be-bea9-0ac482a7432a","service":"frontend","state":"acknowledged"},{"first_seen_at":"2026-08-30T05:57:28.012568Z","id":13334876201,"is_error":false,"last_seen_at":"2026-10-08T15:31:48.951187Z","level":null,"occurrence_count":213021,"pattern":"request_type;neutral⇒outgoing rpc.method;neutral⇒PlaceOrder rpc.service;neutral⇒oteldemo.CheckoutService span_name;neutral⇒grpc.oteldemo.CheckoutService/PlaceOrder protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> ms session;right-badge-neutral⇒<*> user <*>","sample_message":"request_type;neutral⇒outgoing rpc.method;neutral⇒PlaceOrder rpc.service;neutral⇒oteldemo.CheckoutService span_name;neutral⇒grpc.oteldemo.CheckoutService/PlaceOrder attributes;text-textWeak⇒{\"rpc\":{\"grpc\":{\"status_code\":0},\"method\":\"PlaceOrder\",\"service\":\"oteldemo.CheckoutService\",\"system\":\"grpc\"},\"server\":{\"address\":\"checkout\",\"port\":8080},\"session\":{\"id\":\"450aa77f-4da2-470b-8cc2-063642df508c\"},\"user\":{\"id\":\"f0f89e60-9085-42be-bea9-0ac482a7432a\"}} protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒51.6
Active faults remain. There are 6 active incidents, including one receiving events as recently as 2026-10-09 15:30 UTC, and 173 open issues. Recent critical issues include shipping-quote EOF failures in frontend, recommendation-service errors, ProductCatalogService NOT_FOUND errors, checkout product lookup failures, and product-catalog INTERNAL/EOF errors. Current telemetry contains 74 ERROR-level or ERROR-status events across 7 services, led by otelcol-contrib (29), flagd (15), SYSTEM (11), recommendation (5), fraud-detection (5), ad (5), and payment (4).
Error telemetry by service7
Error telemetry over time37
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-09T15:30:05.351697Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-08T02:10:48.556602Z","environment":null,"id":"b4079163-d3cf-45ff-9a33-235493ef5428","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New ERROR Pattern: PlaceOrder Fails on Shipping Quote EOF","type":"log_pattern","updated_at":"2026-10-08T03:00:37.474101Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:28.746082Z","environment":null,"id":"2c314505-843b-4909-9485-cb77494581d9","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-07T22:01:41.040459Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:18.342029Z","environment":null,"id":"ca69d65d-7cd9-443e-99e3-1c2966e43a6c","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"ProductCatalogService GetProduct RPC Returns NOT_FOUND Error","type":"log_pattern","updated_at":"2026-10-07T22:01:35.497467Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:18.322638Z","environment":null,"id":"97fac492-fb36-4e75-97b5-a8ec415ac242","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"Product Catalog GetProduct RPC Returns NOT_FOUND Errors in Frontend","type":"log_pattern","updated_at":"2026-10-07T22:01:30.531778Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:35:55.35698Z","environment":null,"id":"a
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 4.0
otelcol-contrib | 29.0
fraud-detection | 5.0
flagd | 15.0
ad | 5.0
SYSTEM | 11.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (37 rows):
1.7915598e9 | 1.0
1.79155944e9 | 1.0
1.79155929e9 | 1.0
1.79155911e9 | 2.0
1.79155908e9 | 1.0
1.79155902e9 | 2.0
1.7915589e9 | 1.0
1.79155872e9 | 1.0
1.79155869e9 | 1.0
1.79155848e9 | 2.0
1.79155842e9 | 2.0
1.79155836e9 | 1.0
1.7915583e9 | 2.0
1.79155806e9 | 3.0
1.79155788e9 | 2.0
1.79155782e9 | 2.0
1.7915577e9 | 3.0
1.79155746e9 | 1.0
1.79155734e9 | 1.0
1.79155725e9 | 2.0
... +17 more
Active faults remain: 6 active incidents, 173 open issues, and 408 ERROR-level or ERROR-status telemetry events across 8 services. The newest active incident emitted an event at 2026-10-10 15:26 UTC. Error telemetry is dominated by otelcol-contrib (360), followed by flagd (13), SYSTEM (10), recommendation (5), payment (5), fraud-detection (5), ad (5), and frontend-proxy (1). Recent critical issues include frontend shipping-quote EOF failures, recommendation-service errors, ProductCatalogService NOT_FOUND errors, checkout product lookup failures, product-catalog INTERNAL/EOF errors, and 503 errors on POST /v1/traces. Five other active incidents appear stale, with latest events on 2026-09-26.
Error telemetry by service
Error telemetry over time165
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-10T15:26:06.86635Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-08T02:10:48.556602Z","environment":null,"id":"b4079163-d3cf-45ff-9a33-235493ef5428","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New ERROR Pattern: PlaceOrder Fails on Shipping Quote EOF","type":"log_pattern","updated_at":"2026-10-08T03:00:37.474101Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:28.746082Z","environment":null,"id":"2c314505-843b-4909-9485-cb77494581d9","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-07T22:01:41.040459Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:18.342029Z","environment":null,"id":"ca69d65d-7cd9-443e-99e3-1c2966e43a6c","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"ProductCatalogService GetProduct RPC Returns NOT_FOUND Error","type":"log_pattern","updated_at":"2026-10-07T22:01:35.497467Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:36:18.322638Z","environment":null,"id":"97fac492-fb36-4e75-97b5-a8ec415ac242","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"Product Catalog GetProduct RPC Returns NOT_FOUND Errors in Frontend","type":"log_pattern","updated_at":"2026-10-07T22:01:30.531778Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-07T21:35:55.35698Z","environment":null,"id":"a
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (8 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 360.0
frontend-proxy | 1.0
fraud-detection | 5.0
flagd | 13.0
ad | 5.0
SYSTEM | 10.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp), resource.service.name"
→ Results (165 rows):
1.79164629e9 | otelcol-contrib | 1.0
1.79164626e9 | otelcol-contrib | 3.0
1.79164623e9 | otelcol-contrib | 3.0
1.7916462e9 | otelcol-contrib | 3.0
1.79164617e9 | otelcol-contrib | 3.0
1.79164614e9 | otelcol-contrib | 3.0
1.79164611e9 | otelcol-contrib | 3.0
1.79164608e9 | otelcol-contrib | 3.0
1.79164605e9 | otelcol-contrib | 3.0
1.79164602e9 | otelcol-contrib | 3.0
1.79164599e9 | otelcol-contrib | 3.0
1.79164596e9 | SYSTEM | 1.0
1.79164596e9 | otelcol-contrib | 3.0
1.79164593e9 | otelcol-contrib | 3.0
1.7916459e9 | otelcol-contrib | 3.0
1.79164587e9 | otelcol-contrib | 3.0
1.79164584e9 | otelcol-contrib | 3.0
1.79164581e9 | otelcol-contrib | 3.0
1.79164578e9 | otelcol-contrib | 3.0
1.79164575e9 | otelcol-contrib | 3.0
... +145 more
Active faults are present. There are 6 active incidents, 171 open issues, and 260 ERROR-level or ERROR-status telemetry events. The newest active incident emitted an event at 2026-10-11 15:27 UTC. Recent critical issues include API shipping and recommendations endpoint changes, a payment deadline exceeded after 600 seconds, shipping-quote EOF failures, recommendation errors, ProductCatalogService NOT_FOUND errors, checkout product lookup failures, and product-catalog INTERNAL/EOF errors. Error telemetry is dominated by otelcol-contrib (217), followed by flagd (12), SYSTEM (11), and recommendation, payment, fraud-detection, and ad (5 each).
Errors by service7
Errors over time80
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-11T15:27:08.090558Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-11T13:53:55.445365Z","environment":null,"id":"663fdcb6-3fb0-490f-b378-a1b530f28230","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: GET /api/shipping Response Field Modified","type":"api_change","updated_at":"2026-10-11T14:00:36.591281Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-11T13:53:55.445365Z","environment":null,"id":"5fb8dfcd-8c8d-479d-bf3b-f184be7cd59d","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Shipping Endpoint Modified with 2 Field Changes","type":"api_change","updated_at":"2026-10-11T14:00:31.816992Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-11T11:33:38.432221Z","environment":null,"id":"12216558-d85f-4504-bb11-c8170363262b","recommended_action":"Review the changes and update your integration accordingly.","service":"payment","severity":"critical","title":"Critical: Payment Service Deadline Exceeded After 600 Seconds","type":"runtime_exception","updated_at":"2026-10-11T12:00:36.776761Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-11T09:59:22.690851Z","environment":null,"id":"adf48e6a-9c30-4b18-8a63-8cbf1969555a","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: Recommendations Endpoint Adds 1 Field and Modifies 1 Field","type":"api_change","updated_at":"2026-10-11T10:00:30.251467Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-08T02:10:48.556602Z","environment":null,"id":"b4079163-d3cf
count_queryString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Query 'level == "ERROR" OR status_code == "ERROR"' matches 260 entries
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 217.0
fraud-detection | 5.0
flagd | 12.0
ad | 5.0
SYSTEM | 11.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (80 rows):
1.79173242e9 | 1.0
1.79173206e9 | 1.0
1.79173197e9 | 1.0
1.79173194e9 | 5.0
1.79173191e9 | 3.0
1.79173188e9 | 3.0
1.79173185e9 | 3.0
1.79173182e9 | 3.0
1.79173179e9 | 3.0
1.79173176e9 | 3.0
1.79173173e9 | 3.0
1.7917317e9 | 7.0
1.79173167e9 | 5.0
1.79173164e9 | 3.0
1.79173161e9 | 3.0
1.79173158e9 | 3.0
1.79173155e9 | 3.0
1.79173152e9 | 3.0
1.79173149e9 | 3.0
1.79173146e9 | 3.0
... +60 more
Active faults are present. There are 6 active incidents, 177 open issues, and 44 observed ERROR-level or ERROR-status telemetry events across 7 services. The newest active incident emitted an event at 2026-10-12 15:31 UTC. Newly updated critical issues include multiple API contract changes in api-service and recommendations endpoints, plus a payment deadline exceeded after 600 seconds. Persistent critical faults include shipping-quote EOF failures, recommendation-service errors, ProductCatalogService NOT_FOUND/INTERNAL errors, checkout product lookup failures, and frontend 503s. Five incidents appear stale, with latest events on 2026-09-26.
ERROR telemetry by service
ERROR telemetry over time44
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-12T15:31:05.951698Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-12T08:29:22.315118Z","environment":null,"id":"bfc0e77a-92cd-49eb-b03e-2d4df0ef09d2","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 2 Fields Modified in Recommendations Endpoint","type":"api_change","updated_at":"2026-10-12T09:00:53.811069Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-12T08:29:22.315118Z","environment":null,"id":"6cf70afc-5945-405f-83e7-a62b66993352","recommended_action":"Review the changes and update your integration accordingly.","service":"{param}-service","severity":"critical","title":"2 Fields Modified in {param}-service GET /{param} API Response","type":"api_change","updated_at":"2026-10-12T09:00:47.509768Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":3,"archived_at":null,"created_at":"2026-10-12T08:23:35.123886Z","environment":null,"id":"37adb399-2d39-4dbd-aec6-226ac0061dfa","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"api-service API Updated with 1 New and 1 Modified Field","type":"api_change","updated_at":"2026-10-12T09:00:35.579369Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-12T08:23:35.123886Z","environment":null,"id":"65e3081e-e3a9-4e37-9236-90c7d9cbadbb","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 4 Fields Modified in api-service Cart Endpoint","type":"api_change","updated_at":"2026-10-12T09:00:41.856537Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-12T07:30:15.142146Z","environment":null,"id":"2452e6ac-9cec-4
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
frontend-proxy | 1.0
fraud-detection | 5.0
flagd | 12.0
ad | 5.0
SYSTEM | 11.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp), resource.service.name"
→ Results (44 rows):
1.79181906e9 | frontend-proxy | 1.0
1.79181906e9 | SYSTEM | 1.0
1.7918187e9 | SYSTEM | 1.0
1.7918184e9 | SYSTEM | 1.0
1.79181825e9 | flagd | 1.0
1.79181825e9 | ad | 1.0
1.79181822e9 | payment | 1.0
1.7918181e9 | recommendation | 1.0
1.79181804e9 | SYSTEM | 1.0
1.79181798e9 | fraud-detection | 1.0
1.79181798e9 | flagd | 1.0
1.79181768e9 | SYSTEM | 1.0
1.79181765e9 | ad | 1.0
1.79181765e9 | flagd | 1.0
1.79181762e9 | payment | 1.0
1.7918175e9 | flagd | 1.0
1.7918175e9 | recommendation | 1.0
1.79181735e9 | flagd | 1.0
1.79181735e9 | fraud-detection | 1.0
1.79181732e9 | SYSTEM | 1.0
... +24 more
Active faults are present. There are 6 active incidents, 174 open issues, and 85 ERROR-level or ERROR-status telemetry events across 7 services. The newest active incident emitted an event at 2026-10-13 15:27 UTC. Recent critical findings include multiple api-service API contract changes, payment requests exceeding 600 seconds, shipping-quote EOF failures, recommendation-service errors, ProductCatalogService NOT_FOUND/INTERNAL errors, checkout product lookup failures, and frontend 503 errors. Error telemetry is dominated by otelcol-contrib (39), followed by flagd (16), SYSTEM (10), and recommendation, payment, fraud-detection, and ad (5 each). Five active incidents appear stale, with latest events on 2026-09-26.
Errors by service7
3 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-13T15:27:06.806838Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-12T08:29:22.315118Z","environment":null,"id":"6cf70afc-5945-405f-83e7-a62b66993352","recommended_action":"Review the changes and update your integration accordingly.","service":"{param}-service","severity":"critical","title":"2 Fields Modified in {param}-service GET /{param} API Response","type":"api_change","updated_at":"2026-10-12T09:00:47.509768Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-12T08:29:22.315118Z","environment":null,"id":"bfc0e77a-92cd-49eb-b03e-2d4df0ef09d2","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 2 Fields Modified in Recommendations Endpoint","type":"api_change","updated_at":"2026-10-12T09:00:53.811069Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-12T08:23:35.123886Z","environment":null,"id":"65e3081e-e3a9-4e37-9236-90c7d9cbadbb","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 4 Fields Modified in api-service Cart Endpoint","type":"api_change","updated_at":"2026-10-12T09:00:41.856537Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":3,"archived_at":null,"created_at":"2026-10-12T08:23:35.123886Z","environment":null,"id":"37adb399-2d39-4dbd-aec6-226ac0061dfa","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"api-service API Updated with 1 New and 1 Modified Field","type":"api_change","updated_at":"2026-10-12T09:00:35.579369Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-12T07:30:15.142146Z","environment":null,"id":"2452e6ac-9cec-4
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 39.0
fraud-detection | 5.0
flagd | 16.0
ad | 5.0
SYSTEM | 10.0
Active faults remain: 6 active incidents, 173 open issues, and 151 ERROR-level or ERROR-status telemetry events across 9 services. The newest active incident emitted an event at 2026-10-14 15:26 UTC. Current error telemetry is dominated by otelcol-contrib (104), followed by flagd (15), SYSTEM (10), and recommendation, payment, fraud-detection, and ad (5 each). Recent critical issues include a new recommendation-service ERROR pattern updated today, multiple API contract changes, payment requests exceeding 600 seconds, shipping-quote EOF failures, ProductCatalogService NOT_FOUND/INTERNAL errors, checkout product lookup failures, and frontend 503 errors. Five active incidents appear stale, with latest events on 2026-09-26.
Errors by service9
Errors over time55
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-14T15:26:09.098317Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T05:32:12.079355Z","environment":null,"id":"113584fd-6bd6-45db-abbe-230925760efb","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-14T06:00:39.702865Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-13T19:53:44.655766Z","environment":null,"id":"fe03ebfa-682e-4896-87f2-5ccd7c586507","recommended_action":"Review the changes and update your integration accordingly.","service":"{param}-service","severity":"critical","title":"Product API Response Fields Modified in {param}-service","type":"api_change","updated_at":"2026-10-13T20:00:32.10072Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-12T08:29:22.315118Z","environment":null,"id":"bfc0e77a-92cd-49eb-b03e-2d4df0ef09d2","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 2 Fields Modified in Recommendations Endpoint","type":"api_change","updated_at":"2026-10-13T19:53:50.735714Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-12T08:29:22.315118Z","environment":null,"id":"6cf70afc-5945-405f-83e7-a62b66993352","recommended_action":"Review the changes and update your integration accordingly.","service":"{param}-service","severity":"critical","title":"2 Fields Modified in {param}-service GET /{param} API Response","type":"api_change","updated_at":"2026-10-12T09:00:47.509768Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-12T08:23:35.123886Z","environment":null,"id
count_queryString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Query 'level == "ERROR" OR status_code == "ERROR"' matches 151 entries
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (9 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 104.0
frontend-web | 1.0
frontend-proxy | 1.0
fraud-detection | 5.0
flagd | 15.0
ad | 5.0
SYSTEM | 10.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp)"
→ Results (55 rows):
1.79199156e9 | 1.0
1.79199126e9 | 2.0
1.79199117e9 | 2.0
1.7919909e9 | 2.0
1.79199084e9 | 2.0
1.79199066e9 | 1.0
1.79199057e9 | 2.0
1.79199054e9 | 1.0
1.7919903e9 | 2.0
1.79199024e9 | 2.0
1.79199018e9 | 1.0
1.79199003e9 | 1.0
1.79198994e9 | 2.0
1.79198982e9 | 1.0
1.79198976e9 | 2.0
1.7919897e9 | 2.0
1.79198964e9 | 2.0
1.79198946e9 | 1.0
1.79198943e9 | 2.0
1.79198934e9 | 3.0
... +35 more
Active faults remain: 6 active incidents, 173 open issues, and 45 ERROR-level or ERROR-status telemetry events across 6 services. The newest active incident emitted an event at 2026-10-15 15:31 UTC. Newly updated critical issues include an api-service GET /api/data response change, a checkout POST /get-quote error pattern, and a frontend-web invalid-JSON runtime exception. Other unresolved critical faults include recommendation errors, payment requests exceeding 600 seconds, shipping-quote EOF failures, ProductCatalogService NOT_FOUND/INTERNAL errors, and frontend 503 failures. Error telemetry is led by flagd (14), SYSTEM (11), and recommendation, payment, fraud-detection, and ad (5 each).
Errors by service6
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-15T15:31:08.14651Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-15T07:08:46.150328Z","environment":null,"id":"74ae637e-d65a-4b8b-8d91-7a2fb53a5429","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Response Changes: 3 Fields Modified in api-service GET /api/data","type":"api_change","updated_at":"2026-10-15T08:00:38.502887Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:48:28.307931Z","environment":null,"id":"4eb25c60-f3cd-4a34-a1d1-a9cc98f61dbd","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Error Pattern Detected on POST /get-quote in Checkout Service","type":"log_pattern","updated_at":"2026-10-14T19:00:45.242141Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:47:55.475434Z","environment":null,"id":"bf0881c3-cbe2-45f8-8975-66314a0fe1aa","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"SyntaxError: Invalid JSON from Internal Server Error in frontend-web","type":"runtime_exception","updated_at":"2026-10-14T19:00:39.201769Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T05:32:12.079355Z","environment":null,"id":"113584fd-6bd6-45db-abbe-230925760efb","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-14T06:00:39.702865Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-13T19:53:44.655766Z",
count_queryString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Query 'level == "ERROR" OR status_code == "ERROR"' matches 45 entries
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 5.0
payment | 5.0
fraud-detection | 5.0
flagd | 14.0
ad | 5.0
SYSTEM | 11.0
Active faults are present. There are 6 active incidents, 173 open issues, and 44 current ERROR-level or ERROR-status telemetry events across 7 services. The newest incident emitted an event at 2026-10-16 15:29 UTC. Notable open critical issues include an api-service response change updated today, checkout POST /get-quote errors, frontend-web invalid JSON errors, recommendation-service errors, payment requests exceeding 600 seconds, and product-catalog/checkout product lookup failures. A persistent quote-service error pattern has 1,027,689 occurrences and was observed at 2026-10-16 15:31 UTC. Five active incidents appear stale, with latest events on 2026-09-26.
Errors by service6
Current error telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-16T15:29:06.706477Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-15T07:08:46.150328Z","environment":null,"id":"74ae637e-d65a-4b8b-8d91-7a2fb53a5429","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Response Changes: 3 Fields Modified in api-service GET /api/data","type":"api_change","updated_at":"2026-10-15T08:00:38.502887Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:48:28.307931Z","environment":null,"id":"4eb25c60-f3cd-4a34-a1d1-a9cc98f61dbd","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Error Pattern Detected on POST /get-quote in Checkout Service","type":"log_pattern","updated_at":"2026-10-14T19:00:45.242141Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:47:55.475434Z","environment":null,"id":"bf0881c3-cbe2-45f8-8975-66314a0fe1aa","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"SyntaxError: Invalid JSON from Internal Server Error in frontend-web","type":"runtime_exception","updated_at":"2026-10-14T19:00:39.201769Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T05:32:12.079355Z","environment":null,"id":"113584fd-6bd6-45db-abbe-230925760efb","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"New ERROR Log Pattern in Recommendation Service get_product_list Span","type":"log_pattern","updated_at":"2026-10-14T06:00:39.702865Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-13T19:53:44.655766Z",
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 5.0
payment | 5.0
fraud-detection | 5.0
flagd | 13.0
ad | 5.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-10-09T23:04:40.236548Z","id":14125965032,"is_error":false,"last_seen_at":"2026-10-16T15:31:44.270953Z","level":null,"occurrence_count":6810,"pattern":"request_type;neutral⇒incoming status_code;badge-2xx⇒<*> method;badge-POST⇒POST route;neutral⇒/ship-order protocol;right-badge-neutral⇒http duration;right-badge-neutral⇒<*> <*> user <*>","sample_message":"request_type;neutral⇒incoming status_code;badge-2xx⇒200 method;badge-POST⇒POST route;neutral⇒/ship-order attributes;text-textWeak⇒{\"client\":{\"address\":\"10.42.2.11\"},\"http\":{\"request\":{\"body\":{\"size\":261},\"method\":\"POST\"},\"response\":{\"status_code\":200},\"route\":\"/ship-order\"},\"monoscope\":{\"endpoint\":{\"route\":\"/ship-order\",\"route_inferred\":true}},\"network\":{\"protocol\":{\"version\":\"1.1\"}},\"server\":{\"address\":\"shipping\",\"port\":8080},\"url\":{\"path\":\"/ship-order\",\"scheme\":\"http\"},\"user\":{\"id\":\"84888e6d-4f72-4200-932b-f0e00369b617\"},\"user_agent\":{\"original\":\"Go-http-client/1.1\"}} protocol;right-badge-neutral⇒http duration;right-badge-neutral⇒65.4 µs user id;right-badge-neutral⇒84888e6d-4f72-4200-932b-f0e00369b617","service":"shipping","state":"acknowledged"},{"first_seen_at":"2026-03-09T10:25:00.909541Z","id":692414,"is_error":false,"last_seen_at":"2026-10-16T15:31:44.270953Z","level":"INFO","occurrence_count":1040600,"pattern":"severity_text;badge-info⇒INFO Sending Quote","sample_message":"severity_text;badge-info⇒INFO Sending Quote attributes;text-textWeak⇒{\"cents\":99,\"dollars\":8,\"log\":{\"original\":\"Sending Quote\"}}","service":"shipping","state":"acknowledged"},{"first_seen_at":"2026-05-07T09:42:26.391233Z","id":2238773692,"is_error":false,"last_seen_at":"2026-10-16T15:31:44.270953Z","level":null,"occurrence_count":42624200,"pattern":"request_type;neutral⇒incoming status_code;badge-2xx⇒<*> method;badge-POST⇒POST <*>","sample_message":"request_type;neutral⇒incoming status_code;badge-2xx⇒200 method;badge-POST⇒POST route;ne
Active faults remain: 6 active incidents, 172 open issues, and 356 ERROR-level or ERROR-status telemetry events across 10 services. The newest incident emitted an event at 2026-10-17 15:27 UTC. Newly updated critical issues include checkout POST /get-quote errors and frontend HTTP 500 errors on POST /api/checkout, both updated today. Error telemetry is dominated by otelcol-contrib (305), followed by flagd (13), SYSTEM (11), and recommendation, payment, fraud-detection, and ad (5 each). Five active incidents appear stale, with their latest events on 2026-09-26.
ERROR telemetry by service10
3 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-17T15:27:05.062415Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-17T14:43:59.232599Z","environment":null,"id":"5785149c-060f-4537-a09b-e697a59fcad0","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"New ERROR Log Pattern for Checkout POST /get-quote Requests","type":"log_pattern","updated_at":"2026-10-17T15:00:44.145952Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-17T14:43:10.208598Z","environment":null,"id":"26f6d29d-94f5-42a9-8fef-e5d5cd480722","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"HTTP 500 Errors on POST /api/checkout in Frontend Service","type":"log_pattern","updated_at":"2026-10-17T15:00:39.217588Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-15T07:08:46.150328Z","environment":null,"id":"74ae637e-d65a-4b8b-8d91-7a2fb53a5429","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Response Changes: 3 Fields Modified in api-service GET /api/data","type":"api_change","updated_at":"2026-10-15T08:00:38.502887Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:48:28.307931Z","environment":null,"id":"4eb25c60-f3cd-4a34-a1d1-a9cc98f61dbd","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Error Pattern Detected on POST /get-quote in Checkout Service","type":"log_pattern","updated_at":"2026-10-14T19:00:45.242141Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:47:55.475434Z","environment":null,"id":"bf0881c3-cbe
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (10 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 305.0
frontend-proxy | 2.0
frontend | 3.0
fraud-detection | 5.0
flagd | 13.0
checkout | 2.0
ad | 5.0
SYSTEM | 11.0
Active faults remain: 6 active incidents, 171 open issues, and 153 current ERROR-level or ERROR-status telemetry events across 7 services. The newest incident emitted an event at 2026-10-18 15:29 UTC. Error telemetry is dominated by otelcol-contrib (117), followed by flagd (15), SYSTEM (11), and recommendation, payment, fraud-detection, and ad (5 each). Recent critical issues include checkout POST /get-quote errors, frontend HTTP 500 errors on POST /api/checkout, API response changes, payment deadline exceedances after 600 seconds, shipping-quote EOF failures, recommendation errors, and ProductCatalogService NOT_FOUND/INTERNAL errors. Five active incidents appear stale, with their latest events on 2026-09-26.
Errors by service7
Error trend86
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-18T15:29:09.0323Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-17T14:43:59.232599Z","environment":null,"id":"5785149c-060f-4537-a09b-e697a59fcad0","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"New ERROR Log Pattern for Checkout POST /get-quote Requests","type":"log_pattern","updated_at":"2026-10-17T15:00:44.145952Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-17T14:43:10.208598Z","environment":null,"id":"26f6d29d-94f5-42a9-8fef-e5d5cd480722","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"HTTP 500 Errors on POST /api/checkout in Frontend Service","type":"log_pattern","updated_at":"2026-10-17T15:00:39.217588Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-15T07:08:46.150328Z","environment":null,"id":"74ae637e-d65a-4b8b-8d91-7a2fb53a5429","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Response Changes: 3 Fields Modified in api-service GET /api/data","type":"api_change","updated_at":"2026-10-15T08:00:38.502887Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:48:28.307931Z","environment":null,"id":"4eb25c60-f3cd-4a34-a1d1-a9cc98f61dbd","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Error Pattern Detected on POST /get-quote in Checkout Service","type":"log_pattern","updated_at":"2026-10-14T19:00:45.242141Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:47:55.475434Z","environment":null,"id":"bf0881c3-cbe
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 117.0
fraud-detection | 5.0
flagd | 15.0
ad | 5.0
SYSTEM | 11.0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp), resource.service.name"
→ Results (86 rows):
1.79233749e9 | otelcol-contrib | 1.0
1.79233746e9 | otelcol-contrib | 3.0
1.79233743e9 | otelcol-contrib | 3.0
1.7923374e9 | otelcol-contrib | 3.0
1.79233737e9 | otelcol-contrib | 3.0
1.79233734e9 | otelcol-contrib | 3.0
1.79233734e9 | SYSTEM | 1.0
1.79233731e9 | otelcol-contrib | 3.0
1.79233728e9 | otelcol-contrib | 3.0
1.79233725e9 | otelcol-contrib | 3.0
1.79233722e9 | otelcol-contrib | 3.0
1.79233719e9 | otelcol-contrib | 3.0
1.79233716e9 | otelcol-contrib | 3.0
1.79233713e9 | otelcol-contrib | 3.0
1.7923371e9 | otelcol-contrib | 3.0
1.79233707e9 | otelcol-contrib | 3.0
1.79233704e9 | otelcol-contrib | 3.0
1.79233701e9 | otelcol-contrib | 3.0
1.79233698e9 | SYSTEM | 1.0
1.79233698e9 | otelcol-contrib | 3.0
... +66 more
Active faults are present. There are 6 active incidents, including one with events as recently as 2026-10-19 15:28 UTC, and 167 open issues. Current telemetry shows 320 ERROR-level or ERROR-status events across 11 services, dominated by otelcol-contrib (266), flagd (15), SYSTEM (11), and frontend (3). Critical open issues include checkout POST /get-quote errors, frontend HTTP 500 errors on POST /api/checkout, and api-service GET /api/data response changes. Persistent error patterns include product-catalog database errors (3,034,737 occurrences), accounting database errors (221,496 occurrences), and frontend-proxy errors (3,800,563 occurrences). Five active incidents appear stale, with their latest events on 2026-09-26.
Current error telemetry by service11
Error telemetry over time138
8 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-19T15:28:06.137962Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-17T14:43:59.232599Z","environment":null,"id":"5785149c-060f-4537-a09b-e697a59fcad0","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"New ERROR Log Pattern for Checkout POST /get-quote Requests","type":"log_pattern","updated_at":"2026-10-17T15:00:44.145952Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-17T14:43:10.208598Z","environment":null,"id":"26f6d29d-94f5-42a9-8fef-e5d5cd480722","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"HTTP 500 Errors on POST /api/checkout in Frontend Service","type":"log_pattern","updated_at":"2026-10-17T15:00:39.217588Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-15T07:08:46.150328Z","environment":null,"id":"74ae637e-d65a-4b8b-8d91-7a2fb53a5429","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Response Changes: 3 Fields Modified in api-service GET /api/data","type":"api_change","updated_at":"2026-10-18T21:10:26.172431Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:48:28.307931Z","environment":null,"id":"4eb25c60-f3cd-4a34-a1d1-a9cc98f61dbd","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Error Pattern Detected on POST /get-quote in Checkout Service","type":"log_pattern","updated_at":"2026-10-14T19:00:45.242141Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-14T18:47:55.475434Z","environment":null,"id":"bf0881c3-cbe
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (11 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 266.0
frontend-web | 1.0
frontend-proxy | 2.0
frontend | 3.0
fraud-detection | 5.0
flagd | 15.0
checkout | 2.0
ad | 5.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-03-10T10:02:30.535168Z","id":881955,"is_error":false,"last_seen_at":"2026-10-19T15:31:44.052301Z","level":null,"occurrence_count":470329,"pattern":"span_name;neutral⇒orders receive duration;right-badge-neutral⇒<*> <*>","sample_message":"span_name;neutral⇒orders receive attributes;text-textWeak⇒{\"messaging\":{\"client_id\":\"rdkafka#consumer-1\",\"destination\":{\"name\":\"orders\"},\"kafka\":{\"consumer\":{\"group\":\"accounting\"},\"destination\":{\"partition\":0},\"message\":{\"offset\":362744}},\"operation\":\"receive\",\"system\":\"kafka\"},\"server\":{\"address\":\"accounting\"}} duration;right-badge-neutral⇒45.4 ms","service":"accounting","state":"acknowledged"},{"first_seen_at":"2026-03-10T10:12:15.245821Z","id":884347,"is_error":false,"last_seen_at":"2026-10-19T15:31:44.052301Z","level":null,"occurrence_count":1109759,"pattern":"kind;neutral⇒internal span_name;neutral⇒order-consumed duration;right-badge-neutral⇒<*> <*>","sample_message":"kind;neutral⇒internal span_name;neutral⇒order-consumed attributes;text-textWeak⇒{\"server\":{\"address\":\"accounting\"}} duration;right-badge-neutral⇒9.2 s","service":"accounting","state":"acknowledged"},{"first_seen_at":"2026-09-11T00:12:53.310266Z","id":14064926151,"is_error":true,"last_seen_at":"2026-10-19T15:31:44.052301Z","level":null,"occurrence_count":121348,"pattern":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒INSERT INTO accounting.\"order\" (order_id) VALUES (@p{integer}); INSERT INTO accounting.orderitem (order_id, product_id, item_cost_currency_code, item_cost_nanos, item_cost_units, quantity) VALUES (@p{integer}, @p{integer}, @ <*> status;badge-error⇒ERROR","sample_message":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒INSERT INTO accounting.\"order\" (order_id) VALUES (@p{integer}); INSERT INTO accounting.orderitem (order_id, product_id, item_cost_currency_code, item_cost_nanos, item_cost_units, quantity) VALUES
get_issue
→ {"activity":[{"created_at":"2026-10-19T02:07:16.81067Z","created_by":"11136227-459c-4e6b-bed2-e63540c96d91","event":"Lifecycle IEViewed"}],"issue":{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-17T14:43:59.232599Z","environment":null,"id":"5785149c-060f-4537-a09b-e697a59fcad0","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"New ERROR Log Pattern for Checkout POST /get-quote Requests","type":"log_pattern","updated_at":"2026-10-17T15:00:44.145952Z"}}
get_issue
→ {"activity":[{"created_at":"2026-10-19T02:56:56.795244Z","created_by":"11136227-459c-4e6b-bed2-e63540c96d91","event":"Lifecycle IEViewed"}],"issue":{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-17T14:43:10.208598Z","environment":null,"id":"26f6d29d-94f5-42a9-8fef-e5d5cd480722","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"HTTP 500 Errors on POST /api/checkout in Frontend Service","type":"log_pattern","updated_at":"2026-10-17T15:00:39.217588Z"}}
get_issue
→ {"activity":[{"created_at":"2026-10-19T03:51:05.806302Z","created_by":"11136227-459c-4e6b-bed2-e63540c96d91","event":"Lifecycle IEViewed"},{"created_at":"2026-10-15T16:37:14.313583Z","created_by":"11136227-459c-4e6b-bed2-e63540c96d91","event":"Lifecycle IEViewed"}],"issue":{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-15T07:08:46.150328Z","environment":null,"id":"74ae637e-d65a-4b8b-8d91-7a2fb53a5429","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Response Changes: 3 Fields Modified in api-service GET /api/data","type":"api_change","updated_at":"2026-10-18T21:10:26.172431Z"}}
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by bin_auto(timestamp), resource.service.name"
→ Results (138 rows):
1.79242389e9 | otelcol-contrib | 1.0
1.79242386e9 | otelcol-contrib | 2.0
1.79242383e9 | otelcol-contrib | 1.0
1.7924238e9 | otelcol-contrib | 3.0
1.79242377e9 | otelcol-contrib | 3.0
1.79242374e9 | otelcol-contrib | 3.0
1.79242371e9 | otelcol-contrib | 3.0
1.79242368e9 | SYSTEM | 1.0
1.79242368e9 | otelcol-contrib | 3.0
1.79242365e9 | otelcol-contrib | 3.0
1.79242362e9 | otelcol-contrib | 3.0
1.79242359e9 | otelcol-contrib | 3.0
1.79242356e9 | otelcol-contrib | 3.0
1.79242353e9 | otelcol-contrib | 3.0
1.7924235e9 | otelcol-contrib | 3.0
1.79242347e9 | otelcol-contrib | 3.0
1.79242344e9 | otelcol-contrib | 3.0
1.79242341e9 | otelcol-contrib | 3.0
1.79242338e9 | otelcol-contrib | 3.0
1.79242335e9 | otelcol-contrib | 3.0
... +118 more
Active faults are worth reporting. There are 6 active incidents and 173 open issues. One incident is still receiving events as of 2026-10-20 15:30 UTC; the other five appear stale, with latest events on 2026-09-26. Current telemetry shows 94 ERROR-level or ERROR-status events across 7 services, led by otelcol-contrib (46), flagd (17), and SYSTEM (11). Newly updated critical issues include API field changes in api-service and images-service, plus recurring checkout POST /get-quote errors and frontend HTTP 500 checkout failures. Persistent error patterns include 3,937,720 frontend /api/data events and 224,815 accounting database insert errors, both observed today.
Current error telemetry by service
Current error telemetry over time
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-20T15:30:22.019771Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-20T09:35:23.432727Z","environment":null,"id":"0c2529b4-ce04-4820-80dd-df2aa42c61da","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 2 Fields Modified in api-service Cart Endpoint","type":"api_change","updated_at":"2026-10-20T10:00:58.004552Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-20T09:35:18.38726Z","environment":null,"id":"05a0e49e-14c4-4114-a23a-bd13525407d4","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Service Updates 2 Fields in GET /api/data Response","type":"api_change","updated_at":"2026-10-20T10:01:08.611568Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-20T09:35:18.38726Z","environment":null,"id":"3d94c3b1-2a6c-475a-94a1-cc38c6e81010","recommended_action":"Review the changes and update your integration accordingly.","service":"images-service","severity":"critical","title":"API Change: Product Image Field Modified in Images Service","type":"api_change","updated_at":"2026-10-20T10:01:03.340228Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-20T09:34:17.955639Z","environment":null,"id":"07f15338-43ad-4464-96d4-0878bb7e1a8f","recommended_action":"Review the changes and update your integration accordingly.","service":"images-service","severity":"critical","title":"API Change: Product Image Field Modified in Images Service","type":"api_change","updated_at":"2026-10-20T10:00:39.460773Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-20T09:34:17.955639Z","environment":null,"id":"9174818d-fdec-435b-
list_log_patterns
→ [{"first_seen_at":"2026-05-07T09:42:26.391233Z","id":2238773692,"is_error":false,"last_seen_at":"2026-10-20T15:31:46.415088Z","level":null,"occurrence_count":43650919,"pattern":"request_type;neutral⇒incoming status_code;badge-2xx⇒<*> method;badge-POST⇒POST <*>","sample_message":"request_type;neutral⇒incoming status_code;badge-2xx⇒200 method;badge-POST⇒POST route;neutral⇒/ship-order attributes;text-textWeak⇒{\"client\":{\"address\":\"10.42.2.11\"},\"http\":{\"request\":{\"body\":{\"size\":215},\"method\":\"POST\"},\"response\":{\"status_code\":200},\"route\":\"/ship-order\"},\"monoscope\":{\"endpoint\":{\"route\":\"/ship-order\",\"route_inferred\":true}},\"network\":{\"protocol\":{\"version\":\"1.1\"}},\"server\":{\"address\":\"shipping\",\"port\":8080},\"session\":{\"id\":\"2b9aea3e-1887-44a8-8246-002f553b158a\"},\"url\":{\"path\":\"/ship-order\",\"scheme\":\"http\"},\"user\":{\"id\":\"e61cbfd1-8b8a-466f-b7c8-f26c6ee49a53\"},\"user_agent\":{\"original\":\"Go-http-client/1... protocol;right-badge-neutral⇒http duration;right-badge-neutral⇒148.0 µs session;right-badge-neutral⇒2b9aea3e-1887-44a8-8246-002f553b158a user id;right-badge-neutral⇒e61cbfd1-8b8a-466f-b7c8-f26c6ee49a53","service":"shipping","state":"acknowledged"},{"first_seen_at":"2026-10-07T07:10:34.144553Z","id":14119595683,"is_error":false,"last_seen_at":"2026-10-20T15:31:46.415088Z","level":null,"occurrence_count":8781,"pattern":"request_type;neutral⇒incoming status_code;badge-2xx⇒<*> method;badge-POST⇒POST route;neutral⇒/get-quote protocol;right-badge-neutral⇒http duration;right-badge-neutral⇒<*> ms","sample_message":"request_type;neutral⇒incoming status_code;badge-2xx⇒200 method;badge-POST⇒POST route;neutral⇒/get-quote attributes;text-textWeak⇒{\"client\":{\"address\":\"10.42.0.189\"},\"demo\":{\"shipping\":{\"cost\":{\"total\":\"8.99\"}}},\"http\":{\"request\":{\"body\":{\"size\":190},\"method\":\"POST\"},\"response\":{\"status_code\":200},\"route\":\"/get-quote\"},\"monoscope\":{\"endpoint\":{\"route\":
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 46.0
fraud-detection | 5.0
flagd | 17.0
ad | 5.0
SYSTEM | 11.0
Active faults remain. There are 6 active incidents, with the newest event at 2026-10-21 15:26 UTC, and 168 open issues. Current telemetry contains 207 ERROR-level or ERROR-status events across 9 services, dominated by otelcol-contrib (163), followed by flagd (12), SYSTEM (10), and recommendation, payment, fraud-detection, and ad (5 each). Newly updated critical issues include api-service and cart-service API response field changes. Ongoing critical faults include checkout POST /get-quote errors, frontend checkout HTTP 500s, recommendation errors, payment deadline exceedances after 600 seconds, shipping-quote EOF failures, and ProductCatalogService NOT_FOUND/INTERNAL errors. One active incident is current; five appear stale, with latest events on 2026-09-26. No external notification was sent because no notification channel or explicit send action was provided.
Errors by service9
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-21T15:26:09.948903Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-21T07:18:26.695322Z","environment":null,"id":"9e852572-0377-49ca-a6cd-14a8af8ca0ea","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 1 Field Modified in api-service","type":"api_change","updated_at":"2026-10-21T08:00:32.502199Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-21T05:56:44.672344Z","environment":null,"id":"e3e48b46-18e7-476e-bd73-585101e80ddc","recommended_action":"Review the changes and update your integration accordingly.","service":"cart-service","severity":"critical","title":"Cart Service GET /cart Response Changes 2 Fields","type":"api_change","updated_at":"2026-10-21T06:00:25.63094Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-20T21:25:39.700341Z","environment":null,"id":"9bff8834-6634-40eb-82a4-de636ed1361e","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Service API Definition Updated with 1 Modified Field","type":"api_change","updated_at":"2026-10-21T06:37:51.81796Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":3,"archived_at":null,"created_at":"2026-10-20T09:35:23.432727Z","environment":null,"id":"0c2529b4-ce04-4820-80dd-df2aa42c61da","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 2 Fields Modified in api-service Cart Endpoint","type":"api_change","updated_at":"2026-10-20T21:24:42.648834Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-20T09:35:18.38726Z","environment":null,"id":"05a0e49e-14c4-4114-a23a-bd13525407d4","recommend
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (9 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 163.0
frontend-web | 1.0
frontend-proxy | 1.0
fraud-detection | 5.0
flagd | 12.0
ad | 5.0
SYSTEM | 10.0
list_log_patterns
→ [{"first_seen_at":"2026-08-27T02:00:52.664371Z","id":12967108033,"is_error":false,"last_seen_at":"2026-10-21T15:31:41.630944Z","level":null,"occurrence_count":81596,"pattern":"request_type;neutral⇒incoming status_code;badge-2xx⇒<*> method;badge-GET⇒GET <*> protocol;right-badge-neutral⇒http duration;right-badge-neutral⇒<*> ms user id;right-badge-neutral⇒<*>","sample_message":"request_type;neutral⇒incoming status_code;badge-2xx⇒200 method;badge-GET⇒GET route;neutral⇒/api/cart attributes;text-textWeak⇒{\"http\":{\"request\":{\"method\":\"GET\"},\"response\":{\"status_code\":200},\"route\":\"/api/cart\"},\"monoscope\":{\"endpoint\":{\"route\":\"/api/cart\",\"route_inferred\":true}},\"next\":{\"route\":\"/api/cart\",\"rsc\":false,\"span_category\":\"nextjs\",\"span_name\":\"GET /api/cart\",\"span_type\":\"BaseServer.handleRequest\"},\"server\":{\"address\":\"frontend\"},\"url\":{\"path\":\"/api/cart\",\"query\":\"sessionId=33ea2ade-a103-4712-8e26-1d657e837795¤cyCode=USD\"},\"user\":{\"id\":\"33ea2ade-a103-4712-8e26-1d657e837795\"}} protocol;right-badge-neutral⇒http duration;right-badge-neutral⇒13.0 ms user id;right-badge-neutral⇒33ea2ade-a103-4712-8e26-1d657e837795","service":"frontend","state":"acknowledged"},{"first_seen_at":"2026-09-05T21:34:04.61159Z","id":13988582801,"is_error":false,"last_seen_at":"2026-10-21T15:31:41.630944Z","level":null,"occurrence_count":227358,"pattern":"request_type;neutral⇒outgoing rpc.method;neutral⇒ListRecommendations rpc.service;neutral⇒oteldemo.RecommendationService span_name;neutral⇒grpc.oteldemo.RecommendationService/ListRecommendations protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> ms <*>","sample_message":"request_type;neutral⇒outgoing rpc.method;neutral⇒ListRecommendations rpc.service;neutral⇒oteldemo.RecommendationService span_name;neutral⇒grpc.oteldemo.RecommendationService/ListRecommendations attributes;text-textWeak⇒{\"rpc\":{\"grpc\":{\"status_code\":0},\"method\":\"ListRecommendations\",\"service\":\"o
Active faults are present. There are 6 active incidents, including one with activity at 2026-10-22 15:31 UTC, and 166 open issues. Current telemetry contains 74 ERROR-level or ERROR-status events, led by otelcol-contrib (51), SYSTEM (12), flagd (5), payment (2), fraud-detection (3), and ad (1). Recent critical issues include ship-order-service and api-service API contract changes, cart and checkout API modifications, and ongoing payment, checkout, product-catalog, and recommendation faults. Five active incidents appear stale, with their latest events on 2026-09-26.
Errors by service6
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-22T15:31:07.428412Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":3,"archived_at":null,"created_at":"2026-10-22T09:44:11.443832Z","environment":null,"id":"590fb194-c311-4c9a-ac25-abae4e035307","recommended_action":"Review the changes and update your integration accordingly.","service":"ship-order-service","severity":"critical","title":"API Change: 1 Field Modified in ship-order-service /ship-order Endpoint","type":"api_change","updated_at":"2026-10-22T15:05:47.011814Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":7,"archived_at":null,"created_at":"2026-10-22T09:13:47.974567Z","environment":null,"id":"42ba62bb-8739-4f81-8226-69f486737197","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 3 Fields Modified in Currency Endpoint Response","type":"api_change","updated_at":"2026-10-22T12:56:06.746847Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":5,"archived_at":null,"created_at":"2026-10-22T09:07:43.659866Z","environment":null,"id":"a11f2042-d03b-4a1d-9eb3-282e67d153c3","recommended_action":"Review the changes and update your integration accordingly.","service":"{param}-service","severity":"critical","title":"API Change: GetCart Endpoint Modified in {param}-service","type":"api_change","updated_at":"2026-10-22T15:04:45.324017Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":6,"archived_at":null,"created_at":"2026-10-22T09:05:42.368922Z","environment":null,"id":"4492c8a0-aa78-4b94-a0be-b0aba36a1e94","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"Checkout API Updates One Field in api-service","type":"api_change","updated_at":"2026-10-22T15:07:49.300766Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-22T09:03:41.098135Z","environment":null,"id":"f6ff3e6d-
count_queryString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Query 'level == "ERROR" OR status_code == "ERROR"' matches 74 entries
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
payment | 2.0
otelcol-contrib | 51.0
fraud-detection | 3.0
flagd | 5.0
ad | 1.0
SYSTEM | 12.0
list_log_patterns
→ [{"first_seen_at":"2026-05-07T19:51:00.378372Z","id":2313644913,"is_error":false,"last_seen_at":"2026-10-22T15:31:48.828111Z","level":null,"occurrence_count":6630295,"pattern":"severity_text;badge-info⇒INFO Loading incluster config...","sample_message":"severity_text;badge-info⇒INFO Loading incluster config... attributes;text-textWeak⇒{\"log\":{\"file\":{\"path\":\"/var/log/pods/default_grafana-76b5848f46-jzpjj_ab7da2a0-c0cd-4938-bff3-d166b38cd92c/grafana-sc-dashboard/0.log\"},\"iostream\":\"stderr\",\"original\":{\"level\":\"INFO\",\"msg\":\"Loading incluster config...\",\"time\":\"2026-10-22T15:31:42.686666+00:00\"}},\"logtag\":\"F\",\"time\":\"2026-10-22T15:31:42.686666+00:00\"}","service":"otel-demo","state":"acknowledged"},{"first_seen_at":"2026-08-27T00:39:53.271311Z","id":12960881217,"is_error":false,"last_seen_at":"2026-10-22T15:31:48.828111Z","level":null,"occurrence_count":6984297,"pattern":"<*> info Logs {\"resource\": {\"host.name\": \"vps-ca{integer}f{integer}\", \"k{integer}s.namespace.name\": \"default\", \"k{integer}s.node.ip\": <*> \"k{integer}s.node.name\": \"vps-ca{integer}f{integer}\", \"k{integer}s.pod.ip\": <*> \"k{integer}s.pod.name\": \"otel-collector-{integer}cd{integer}-qb{integer}l\", \"service.instance.id\": \"{uuid}\", \"service.name\": \"otelcol-contrib\", \"service.version\": \"{float}.{integer}\"}, \"otelcol.component.id\": \"debug\", \"otelcol.component.kind\": \"exporter\", \"otelcol.signal\": \"logs\", \"resource logs\": {integer}, \"log records\": {integer}}","sample_message":"2026-10-22T15:31:38.508Z\tinfo\tLogs\t{\"resource\": {\"host.name\": \"vps-ca6245f8\", \"k8s.namespace.name\": \"default\", \"k8s.node.ip\": \"51.210.6.164\", \"k8s.node.name\": \"vps-ca6245f8\", \"k8s.pod.ip\": \"10.42.2.5\", \"k8s.pod.name\": \"otel-collector-77cd857677-qb24l\", \"service.instance.id\": \"05e33c85-0070-4a17-afbc-cba30378ff1b\", \"service.name\": \"otelcol-contrib\", \"service.version\": \"0.156.0\"}, \"otelcol.component.id\": \"debug\", \"
Active faults are worth reporting. There are 6 active incidents and 165 open issues. One incident is receiving events as recently as 2026-10-23 15:30 UTC; five appear stale, with their latest events on 2026-09-26. Current telemetry contains 21 ERROR-level or ERROR-status events across six services, led by SYSTEM (11), flagd (3), recommendation (2), payment (2), fraud-detection (2), and ad (1). Recent critical issues include API contract changes in api-service, ship-order-service, get-quote-service, cart-service, and images-service, plus recurring checkout, payment, product-catalog, and recommendation failures. No external notification was sent because no notification channel or explicit send action was provided.
Current error telemetry by service
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-23T15:30:11.796652Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-22T18:01:49.976371Z","environment":null,"id":"dd87a53a-b8bb-4236-9ae2-0360834e73d7","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: Cart Endpoint Response Field Modified in API Service","type":"api_change","updated_at":"2026-10-23T00:02:22.358929Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":7,"archived_at":null,"created_at":"2026-10-22T09:44:11.443832Z","environment":null,"id":"590fb194-c311-4c9a-ac25-abae4e035307","recommended_action":"Review the changes and update your integration accordingly.","service":"ship-order-service","severity":"critical","title":"API Change: 1 Field Modified in ship-order-service /ship-order Endpoint","type":"api_change","updated_at":"2026-10-23T06:07:16.217134Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":9,"archived_at":null,"created_at":"2026-10-22T09:13:47.974567Z","environment":null,"id":"42ba62bb-8739-4f81-8226-69f486737197","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 3 Fields Modified in Currency Endpoint Response","type":"api_change","updated_at":"2026-10-23T04:07:08.701351Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":7,"archived_at":null,"created_at":"2026-10-22T09:07:43.659866Z","environment":null,"id":"a11f2042-d03b-4a1d-9eb3-282e67d153c3","recommended_action":"Review the changes and update your integration accordingly.","service":"{param}-service","severity":"critical","title":"API Change: GetCart Endpoint Modified in {param}-service","type":"api_change","updated_at":"2026-10-23T00:02:22.316212Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":9,"archived_at":null,"created_at":"2026-10-22T09:05:42.368922Z","environment":n
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 2.0
payment | 2.0
fraud-detection | 2.0
flagd | 3.0
ad | 1.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-03-09T10:25:00.909541Z","id":692486,"is_error":false,"last_seen_at":"2026-10-23T15:31:42.735412Z","level":"INFO","occurrence_count":1107587,"pattern":"severity_text;badge-info⇒INFO Requesting quote","sample_message":"severity_text;badge-info⇒INFO Requesting quote attributes;text-textWeak⇒{\"log\":{\"original\":\"Requesting quote\"},\"quote_service_addr\":\"http://quote:8080/getquote\"}","service":"shipping","state":"acknowledged"},{"first_seen_at":"2026-03-09T10:25:00.909541Z","id":692430,"is_error":false,"last_seen_at":"2026-10-23T15:31:42.735412Z","level":"INFO","occurrence_count":746367,"pattern":"severity_text;badge-info⇒INFO Tracking ID Created","sample_message":"severity_text;badge-info⇒INFO Tracking ID Created attributes;text-textWeak⇒{\"log\":{\"original\":\"Tracking ID Created\"},\"tracking_id\":\"f5e35261-71a2-4476-91c1-28e1743170cf\"}","service":"shipping","state":"acknowledged"},{"first_seen_at":"2026-03-09T10:25:00.909541Z","id":692414,"is_error":false,"last_seen_at":"2026-10-23T15:31:42.735412Z","level":"INFO","occurrence_count":1107180,"pattern":"severity_text;badge-info⇒INFO Sending Quote","sample_message":"severity_text;badge-info⇒INFO Sending Quote attributes;text-textWeak⇒{\"cents\":95,\"dollars\":44,\"log\":{\"original\":\"Sending Quote\"}}","service":"shipping","state":"acknowledged"},{"first_seen_at":"2026-06-28T08:36:33.809733Z","id":6848243653,"is_error":false,"last_seen_at":"2026-10-23T15:31:42.387524Z","level":null,"occurrence_count":841502,"pattern":"severity_text;badge-info⇒INFO Order confirmation email sent","sample_message":"severity_text;badge-info⇒INFO Order confirmation email sent attributes;text-textWeak⇒{\"demo\":{\"order\":{\"id\":\"b7291bd0-cef6-11f1-a6ed-4ab050d02f17\"}},\"log\":{\"original\":\"Order confirmation email sent\"}}","service":"email","state":"acknowledged"},{"first_seen_at":"2026-09-05T22:24:58.809575Z","id":13991173528,"is_error":false,"last_seen_at":"2026-10-23T15:31:42.192191Z","level":nul
Active faults are present as of 2026-10-24 15:31 UTC: 6 active incidents, 168 open issues, and 40 ERROR-level or ERROR-status telemetry events. One incident is actively receiving events, with its latest event at 15:31 UTC; five appear stale, with latest events on 2026-09-26. Recent critical findings include API schema changes in api-service, ship-order-service, cart-service, get-quote-service, images-service, and _next-service, plus checkout GET /get-quote errors, frontend HTTP 500 checkout failures, payment deadline exceedances after 600 seconds, product lookup failures, and product-catalog query EOF errors. No external notification was sent because no notification channel was specified.
Current error events
3 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-24T15:31:09.683103Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-24T04:02:18.471561Z","environment":null,"id":"6ed96e1c-1c47-448a-ad57-5bb7fd109e37","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Service Endpoint Schema Modified","type":"api_change","updated_at":"2026-10-24T05:00:29.782554Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":3,"archived_at":null,"created_at":"2026-10-22T18:01:49.976371Z","environment":null,"id":"dd87a53a-b8bb-4236-9ae2-0360834e73d7","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: Cart Endpoint Response Field Modified in API Service","type":"api_change","updated_at":"2026-10-23T16:02:32.259988Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":9,"archived_at":null,"created_at":"2026-10-22T09:44:11.443832Z","environment":null,"id":"590fb194-c311-4c9a-ac25-abae4e035307","recommended_action":"Review the changes and update your integration accordingly.","service":"ship-order-service","severity":"critical","title":"API Change: 1 Field Modified in ship-order-service /ship-order Endpoint","type":"api_change","updated_at":"2026-10-24T12:22:17.128317Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":11,"archived_at":null,"created_at":"2026-10-22T09:13:47.974567Z","environment":null,"id":"42ba62bb-8739-4f81-8226-69f486737197","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 3 Fields Modified in Currency Endpoint Response","type":"api_change","updated_at":"2026-10-24T13:02:13.22269Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":7,"archived_at":null,"created_at":"2026-10-22T09:07:43.659866Z","environment":null,"id":"a11f2042-d03b-
count_queryString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Query 'level == "ERROR" OR status_code == "ERROR"' matches 40 entries
Active faults are present. There are 6 active incidents, with the newest event at 2026-10-25 15:27 UTC; five incidents appear stale, with their latest events on 2026-09-26. There are 169 open issues, including newly updated critical API schema changes in api-service, plus ongoing checkout, payment, product-catalog, recommendation, and frontend faults. Current telemetry includes recurring ERROR or ERROR-status events through 15:27 UTC, including monitor alerts from SYSTEM and repeated approximately 600-second flagd EventStream failures affecting recommendation, ad, fraud-detection, and payment.
Recent error telemetry
3 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-25T15:27:10.155751Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-25T02:09:24.987453Z","environment":null,"id":"59cf177d-33d5-4cd2-b73f-24f3300ad660","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 1 Field Modified in api-service","type":"api_change","updated_at":"2026-10-25T03:00:25.919462Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":2,"archived_at":null,"created_at":"2026-10-24T04:02:18.471561Z","environment":null,"id":"6ed96e1c-1c47-448a-ad57-5bb7fd109e37","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Service Endpoint Schema Modified","type":"api_change","updated_at":"2026-10-25T02:09:25.250769Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":4,"archived_at":null,"created_at":"2026-10-22T18:01:49.976371Z","environment":null,"id":"dd87a53a-b8bb-4236-9ae2-0360834e73d7","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: Cart Endpoint Response Field Modified in API Service","type":"api_change","updated_at":"2026-10-25T02:09:14.489486Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":9,"archived_at":null,"created_at":"2026-10-22T09:44:11.443832Z","environment":null,"id":"590fb194-c311-4c9a-ac25-abae4e035307","recommended_action":"Review the changes and update your integration accordingly.","service":"ship-order-service","severity":"critical","title":"API Change: 1 Field Modified in ship-order-service /ship-order Endpoint","type":"api_change","updated_at":"2026-10-24T12:22:17.128317Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":11,"archived_at":null,"created_at":"2026-10-22T09:13:47.974567Z","environment":null,"id":"42ba62bb-8739-4f81-8226-69f48
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Results (40 rows):
13d9e3c2-58e6-5ae4-b4be-07375dfb3426 | 2026-10-25T15:27:10.155752Z | null | monitor.alert.triggered | 5.030823e7 | SYSTEM | c848ea7d-a61b-4d10-a601-b7e2d45c36f3 | 1.792942030155752e18 | false | [3] | null | alert
b28e2375-6a49-569a-81d9-67d889ab5717 | 2026-10-25T15:21:10.230192Z | null | monitor.alert.triggered | 4.8855245e7 | SYSTEM | c848ea7d-a61b-4d10-a601-b7e2d45c36f3 | 1.792941670230192e18 | false | [3] | null | alert
8fe3bc89-fda1-53b7-bd22-1044e083fe65 | 2026-10-25T15:18:08.606571Z | cd0459a7142283377a4ccdbe56ae9b86 | flagd.evaluation.v2.Service/EventStream | 5.99982232843e11 | flagd | 419fdae404eefa37 | 1.792941488606571e18 | false | [9] | 2c17c60dadddd5eb | server
c82c8787-a2fb-508b-ad95-d77812eeaaac | 2026-10-25T15:18:08.588017Z | cd0459a7142283377a4ccdbe56ae9b86 | flagd.evaluation.v2.Service/EventStream | 6.00003601146e11 | ad | null | 1.792941488588017e18 | false | [9] | 419fdae404eefa37 | client
e9e056b2-809d-5c6f-8b99-c17cd8c52472 | 2026-10-25T15:15:27.416482Z | 966d8476ad7b7260550cb764be051caf | /flagd.evaluation.v2.Service/EventStream | 6.00005949947e11 | recommendation | null | 1.792941327416482e18 | false | [9] | c7a08453b8449a2b | client
e9306f06-a83f-5844-b9b3-a743dffa428e | 2026-10-25T15:15:27.410624Z | 966d8476ad7b7260550cb764be051caf | flagd.evaluation.v2.Service/EventStream | 6.00001192979e11 | flagd | c7a08453b8449a2b | 1.792941327410624e18 | false | [9] | 8fef1e875429a72c | server
9a766215-c849-5b25-a24d-e4d00be15476 | 2026-10-25T15:15:12.772541Z | null | monitor.alert.triggered | 6.4685073e7 | SYSTEM | c848ea7d-a61b-4d10-a601-b7e2d45c36f3 | 1.792941312772541e18 | false | [3] | null | alert
a5f4e030-7b83-5262-b2b5-238f1497a171 | 2026-10-25T15:12:13.326930Z | 565ea10a83c76ed8442e02e2ef3b98f6 | flagd.evaluation.v2.Service/EventStream | 6.00006702519e11 | fraud-detection | null | 1.79294113332693e18 | false | [9] | 0e2c4bad774f0f17 | client
0fb7382b-6f80-532e-9fbd-a704d2dbba35 | 2026-10-25T15:12:13.325094Z
Active faults are present as of 2026-10-26 15:31 UTC. There are 6 active incidents, with one receiving events through 15:28 UTC; five appear stale, with latest events on 2026-09-26. There are 169 open issues, including critical API contract changes updated today in api-service, icons-service, and other services. Current telemetry contains 131 ERROR-level or ERROR-status events across 7 services, dominated by otelcol-contrib (86) and flagd (15). A recurring otelcol-contrib export failure pattern has 194,430 occurrences and was observed at 15:31 UTC, indicating dropped telemetry data. Other persistent critical faults include accounting database errors with 240,868 occurrences and recurring quote-service POST /getquote errors with 1,219,717 occurrences.
Errors by service7
Faulty telemetry by service
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-26T15:28:07.359487Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":3,"archived_at":null,"created_at":"2026-10-25T02:09:24.987453Z","environment":null,"id":"59cf177d-33d5-4cd2-b73f-24f3300ad660","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: 1 Field Modified in api-service","type":"api_change","updated_at":"2026-10-26T11:56:55.554223Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":4,"archived_at":null,"created_at":"2026-10-24T04:02:18.471561Z","environment":null,"id":"6ed96e1c-1c47-448a-ad57-5bb7fd109e37","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Service Endpoint Schema Modified","type":"api_change","updated_at":"2026-10-26T11:56:55.545387Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":4,"archived_at":null,"created_at":"2026-10-22T18:01:49.976371Z","environment":null,"id":"dd87a53a-b8bb-4236-9ae2-0360834e73d7","recommended_action":"Review the changes and update your integration accordingly.","service":"api-service","severity":"critical","title":"API Change: Cart Endpoint Response Field Modified in API Service","type":"api_change","updated_at":"2026-10-25T02:09:14.489486Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":9,"archived_at":null,"created_at":"2026-10-22T09:44:11.443832Z","environment":null,"id":"590fb194-c311-4c9a-ac25-abae4e035307","recommended_action":"Review the changes and update your integration accordingly.","service":"ship-order-service","severity":"critical","title":"API Change: 1 Field Modified in ship-order-service /ship-order Endpoint","type":"api_change","updated_at":"2026-10-24T12:22:17.128317Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":13,"archived_at":null,"created_at":"2026-10-22T09:13:47.974567Z","environment":null,"id":"42ba62bb-8739-4f81-8226-69f48
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 86.0
fraud-detection | 5.0
flagd | 15.0
ad | 5.0
SYSTEM | 10.0
list_log_patterns
→ [{"first_seen_at":"2026-08-27T00:35:46.285353Z","id":12960587875,"is_error":false,"last_seen_at":"2026-10-26T15:31:45.203477Z","level":null,"occurrence_count":5297340,"pattern":"severity_text;badge-info⇒INFO Metrics","sample_message":"severity_text;badge-info⇒INFO Metrics attributes;text-textWeak⇒{\"data points\":240,\"log\":{\"original\":\"Metrics\"},\"metrics\":48,\"resource metrics\":1}","service":"otelcol-contrib","state":"acknowledged"},{"first_seen_at":"2026-08-27T00:35:46.281966Z","id":12960587870,"is_error":false,"last_seen_at":"2026-10-26T15:31:45.203477Z","level":null,"occurrence_count":5460889,"pattern":"severity_text;badge-info⇒INFO Logs","sample_message":"severity_text;badge-info⇒INFO Logs attributes;text-textWeak⇒{\"log\":{\"original\":\"Logs\"},\"log records\":2,\"resource logs\":1}","service":"otelcol-contrib","state":"acknowledged"},{"first_seen_at":"2026-08-27T00:35:46.280422Z","id":12960587868,"is_error":false,"last_seen_at":"2026-10-26T15:31:45.203477Z","level":null,"occurrence_count":5553344,"pattern":"severity_text;badge-info⇒INFO Traces","sample_message":"severity_text;badge-info⇒INFO Traces attributes;text-textWeak⇒{\"log\":{\"original\":\"Traces\"},\"resource spans\":1,\"spans\":4}","service":"otelcol-contrib","state":"acknowledged"},{"first_seen_at":"2026-03-10T10:02:30.535168Z","id":881955,"is_error":false,"last_seen_at":"2026-10-26T15:31:44.809589Z","level":null,"occurrence_count":508312,"pattern":"span_name;neutral⇒orders receive duration;right-badge-neutral⇒<*> <*>","sample_message":"span_name;neutral⇒orders receive attributes;text-textWeak⇒{\"messaging\":{\"client_id\":\"rdkafka#consumer-1\",\"destination\":{\"name\":\"orders\"},\"kafka\":{\"consumer\":{\"group\":\"accounting\"},\"destination\":{\"partition\":0},\"message\":{\"offset\":410124}},\"operation\":\"receive\",\"system\":\"kafka\"},\"server\":{\"address\":\"accounting\"}} duration;right-badge-neutral⇒44.6 ms","service":"accounting","state":"acknowledged"},{"first_seen_at":"20
Active faults found. There are 6 active incidents and 177 open issues, including newly updated critical checkout shipping-quote failures, frontend GET /api/data 500 errors, frontend-proxy 503 errors, and accounting PostgreSQL connection timeouts. Current telemetry contains 148 ERROR-level or ERROR-status events across 7 services, led by otelcol-contrib (103), flagd (14), SYSTEM (11), and recommendation, payment, fraud-detection, and ad (5 each). otelcol-contrib is repeatedly exporting failed data and dropping telemetry as recently as 2026-10-27 15:31 UTC. One incident is current through 15:30 UTC; five appear stale, with their latest events on 2026-09-26.
Current errors by service
Recent error telemetry
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-27T15:30:09.213129Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T11:54:58.097683Z","environment":null,"id":"e76bd12d-0c56-4779-a348-79e758105791","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Shipping Quote Failure in Checkout Service During Shipping Service POST","type":"runtime_exception","updated_at":"2026-10-27T12:00:32.882356Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T11:20:55.453266Z","environment":null,"id":"ec77bf13-09df-46e5-853d-c3ba52a1bfac","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on GET /api/data in Frontend Service","type":"log_pattern","updated_at":"2026-10-27T12:01:07.151155Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T04:06:21.024816Z","environment":null,"id":"15012515-a7d4-40f6-81bd-a8e87fb89ab0","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"503 GET / Responses Detected on Frontend Proxy","type":"log_pattern","updated_at":"2026-10-27T05:00:53.769381Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T01:14:54.998444Z","environment":null,"id":"5840745e-73fb-4980-a54e-5c1726ccbf9b","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on GET /api/data in Frontend Service","type":"log_pattern","updated_at":"2026-10-27T02:00:56.283968Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-26T21:32:57.649525Z","environment":null,"id":"0c198b8c-c00e-4afe-9915-c6b0
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 103.0
fraud-detection | 5.0
flagd | 14.0
ad | 5.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-08-31T15:11:24.331052Z","id":13485440045,"is_error":false,"last_seen_at":"2026-10-27T15:31:47.613638Z","level":null,"occurrence_count":94301,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/ListProducts span_name;neutral⇒oteldemo.ProductCatalogService/ListProducts protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> ms session;right-badge-neutral⇒<*> user <*>","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/ListProducts span_name;neutral⇒oteldemo.ProductCatalogService/ListProducts attributes;text-textWeak⇒{\"demo\":{\"product\":{\"count\":10}},\"rpc\":{\"method\":\"oteldemo.ProductCatalogService/ListProducts\",\"response\":{\"status_code\":\"OK\"},\"system\":{\"name\":\"grpc\"}},\"server\":{\"address\":\"10.42.0.175\",\"port\":8080},\"session\":{\"id\":\"f9212542-c999-4b07-b0a8-1c0647c149d3\"},\"user\":{\"id\":\"dac714ad-2d84-4093-a9ac-e91dd69556c8\"}} protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒1.7 ms session;right-badge-neutral⇒f9212542-c999-4b07-b0a8-1c0647c149d3 user id;right-badge-neutral⇒dac714ad-2d84-4093-a9ac-e91dd69556c8","service":"product-catalog","state":"acknowledged"},{"first_seen_at":"2026-08-28T03:06:30.031018Z","id":13092688259,"is_error":false,"last_seen_at":"2026-10-27T15:31:47.613638Z","level":null,"occurrence_count":1163157,"pattern":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒ SELECT p.id, p.name, p.description, p.picture, p.price_currency_code, p.price_units, p.price_nanos, p.categories FROM catalog.products p ORDER BY p.id span_name;neutral⇒sql.conn.query db.system;right-badge-postgres⇒postgres duration;right-badge-neutral⇒<*> <*> <*> user name;right-badge-neutral⇒Demo shopper <*> user <*>","sample_message":"kind;neutral⇒database db.system;neutral⇒postgresql db.query.text;text-textStrong⇒\n\t\tSELECT p.id, p.name, p.description, p.picture, \n\t\t p.price_
sample_logsString "level == \"ERROR\" OR status_code == \"ERROR\""
→ Sample logs:
- [ERROR] null (otelcol-contrib): Exporting failed. Dropping data. f95bc1d7-dfd6-5619-893e-417dd0fe1375 2026-10-27T15:31:45.006933Z n
- [ERROR] null (otelcol-contrib): Exporting failed. Dropping data. ec6434ed-01a0-530a-86d6-ed78f6da0781 2026-10-27T15:31:34.963697Z n
- [ERROR] null (otelcol-contrib): Exporting failed. Dropping data. b45e8bbe-71ae-5448-b8d8-e17d1f4442cb 2026-10-27T15:31:24.977131Z n
- [ERROR] null (otelcol-contrib): Exporting failed. Dropping data. 2150007d-6543-5f01-951c-59fd996744e3 2026-10-27T15:31:14.990103Z n
- [ERROR] null (otelcol-contrib): Exporting failed. Dropping data. 56ab3290-6239-5a70-a6ae-77937026a06a 2026-10-27T15:31:04.997102Z n
Active faults remain. There are 6 active incidents and 176 open issues. One incident is current, with activity through 2026-10-28 15:27 UTC; the other five appear stale, with their latest events on 2026-09-26. Current telemetry contains 148 ERROR-level or ERROR-status events across 11 services, led by otelcol-contrib (94), flagd (13), SYSTEM (11), and recommendation, payment, fraud-detection, and ad (5 each). Newly updated critical issues include a checkout shipping-quote failure, frontend GET /api/data 500 errors, frontend-proxy 503 GET / errors, and accounting PostgreSQL connection timeouts. A current warning pattern also reports insufficient node image filesystem capacity at 93% used, with failed cleanup attempts. No external notification was sent because no notification channel or explicit send action was provided.
Current error telemetry by service
Current warning telemetry
4 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-28T15:27:07.989753Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T11:54:58.097683Z","environment":null,"id":"e76bd12d-0c56-4779-a348-79e758105791","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Shipping Quote Failure in Checkout Service During Shipping Service POST","type":"runtime_exception","updated_at":"2026-10-27T12:00:32.882356Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T11:20:55.453266Z","environment":null,"id":"ec77bf13-09df-46e5-853d-c3ba52a1bfac","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on GET /api/data in Frontend Service","type":"log_pattern","updated_at":"2026-10-27T12:01:07.151155Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T04:06:21.024816Z","environment":null,"id":"15012515-a7d4-40f6-81bd-a8e87fb89ab0","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"503 GET / Responses Detected on Frontend Proxy","type":"log_pattern","updated_at":"2026-10-27T05:00:53.769381Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T01:14:54.998444Z","environment":null,"id":"5840745e-73fb-4980-a54e-5c1726ccbf9b","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on GET /api/data in Frontend Service","type":"log_pattern","updated_at":"2026-10-27T02:00:56.283968Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-26T21:32:57.649525Z","environment":null,"id":"0c198b8c-c00e-4afe-9915-c6b0
run_queryString "(level == \"ERROR\" OR status_code == \"ERROR\") | summarize count() by resource.service.name"
→ Results (11 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 94.0
frontend-web | 1.0
frontend-proxy | 2.0
frontend | 3.0
fraud-detection | 5.0
flagd | 13.0
checkout | 2.0
ad | 5.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-06-28T08:36:33.809733Z","id":6848243653,"is_error":false,"last_seen_at":"2026-10-28T15:31:43.181797Z","level":null,"occurrence_count":869855,"pattern":"severity_text;badge-info⇒INFO Order confirmation email sent","sample_message":"severity_text;badge-info⇒INFO Order confirmation email sent attributes;text-textWeak⇒{\"demo\":{\"order\":{\"id\":\"8fd59b7d-d2e4-11f1-a6ed-4ab050d02f17\"}},\"log\":{\"original\":\"Order confirmation email sent\"}}","service":"email","state":"acknowledged"},{"first_seen_at":"2026-03-09T10:25:00.909541Z","id":692504,"is_error":false,"last_seen_at":"2026-10-28T15:31:42.948071Z","level":"INFO","occurrence_count":1441181,"pattern":"severity_text;badge-info⇒INFO Transaction complete.","sample_message":"severity_text;badge-info⇒INFO Transaction complete. attributes;text-textWeak⇒{\"amount\":{\"currencyCode\":\"USD\",\"nanos\":629999994,\"units\":{\"high\":0,\"low\":15678,\"unsigned\":false}},\"cardType\":\"visa\",\"lastFourDigits\":\"3547\",\"log\":{\"original\":\"Transaction complete.\"},\"loyalty_level\":\"bronze\",\"transactionId\":\"caa89fda-a989-46b3-9414-70bb85c22c57\"}","service":"payment","state":"acknowledged"},{"first_seen_at":"2026-03-09T10:25:00.909541Z","id":692432,"is_error":false,"last_seen_at":"2026-10-28T15:31:42.948071Z","level":"INFO","occurrence_count":1292307,"pattern":"severity_text;badge-info⇒INFO Charge request received.","sample_message":"severity_text;badge-info⇒INFO Charge request received. attributes;text-textWeak⇒{\"log\":{\"original\":\"Charge request received.\"}}","service":"payment","state":"acknowledged"},{"first_seen_at":"2026-05-07T09:41:43.066388Z","id":2238628176,"is_error":false,"last_seen_at":"2026-10-28T15:31:41.381701Z","level":null,"occurrence_count":96405,"pattern":"<*> <*> UTC [{integer}] LOG: checkpoint starting: time","sample_message":"2026-10-28 15:30:36.640 UTC [74] LOG: checkpoint starting: time attributes;text-textWeak⇒{\"log\":{\"file\":{\"path\":\"/var/log/pods/default
Active faults remain: 6 active incidents and 177 open issues. One incident emitted an event at 2026-10-29 15:27 UTC; five appear stale, with latest events on 2026-09-26. Current ERROR or ERROR-status telemetry totals 75 events, led by otelcol-contrib (24), flagd (17), and SYSTEM (11). Persistent faults include otelcol-contrib dropping telemetry (202,091 occurrences), product-catalog database query errors (2,833,986 occurrences), and node image filesystem pressure at 94% of 71.6 GiB used, with cleanup freeing 0 bytes. Newly updated critical issues include frontend checkout 500 errors, checkout shipping-quote failures, frontend GET /api/data 500 errors, frontend-proxy 503 responses, and accounting PostgreSQL connection timeouts. No external notification was sent because no channel or explicit send action was specified.
Errors by service8
Current error events
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-29T15:27:10.067673Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-28T14:56:26.016495Z","environment":null,"id":"5c0a8ace-e67c-469a-bf0f-99538beaadb9","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on POST /api/checkout in Frontend Service","type":"log_pattern","updated_at":"2026-10-29T09:06:00.230761Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T11:54:58.097683Z","environment":null,"id":"e76bd12d-0c56-4779-a348-79e758105791","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Shipping Quote Failure in Checkout Service During Shipping Service POST","type":"runtime_exception","updated_at":"2026-10-27T12:00:32.882356Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T11:20:55.453266Z","environment":null,"id":"ec77bf13-09df-46e5-853d-c3ba52a1bfac","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on GET /api/data in Frontend Service","type":"log_pattern","updated_at":"2026-10-27T12:01:07.151155Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T04:06:21.024816Z","environment":null,"id":"15012515-a7d4-40f6-81bd-a8e87fb89ab0","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"503 GET / Responses Detected on Frontend Proxy","type":"log_pattern","updated_at":"2026-10-27T05:00:53.769381Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T01:14:54.998444Z","environment":null,"id":"5840745e-73fb-4980-a54e
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (8 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 24.0
frontend-proxy | 1.0
fraud-detection | 5.0
flagd | 17.0
ad | 5.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-09-05T21:06:15.13459Z","id":13987278359,"is_error":false,"last_seen_at":"2026-10-29T15:31:45.205044Z","level":null,"occurrence_count":306064,"pattern":"kind;neutral⇒internal span_name;neutral⇒get_product_list duration;right-badge-neutral⇒<*> ms <*>","sample_message":"kind;neutral⇒internal span_name;neutral⇒get_product_list attributes;text-textWeak⇒{\"demo\":{\"feature_flag\":{\"recommendation_cache\":false},\"product\":{\"count\":10,\"filtered\":{\"count\":9,\"list\":[\"1YMWWN1N4O\",\"0PUK6V6EV0\",\"L9ECAV7KIM\",\"LS4PSXUNUM\",\"9SIQT8TOJO\"]}}},\"server\":{\"address\":\"recommendation\"},\"session\":{\"id\":\"d0a6e167-c460-488d-9707-b0bdf92e1a5c\"}} duration;right-badge-neutral⇒4.5 ms session;right-badge-neutral⇒d0a6e167-c460-488d-9707-b0bdf92e1a5c","service":"recommendation","state":"acknowledged"},{"first_seen_at":"2026-09-05T21:06:15.161114Z","id":13987278373,"is_error":false,"last_seen_at":"2026-10-29T15:31:45.205044Z","level":null,"occurrence_count":306025,"pattern":"request_type;neutral⇒outgoing rpc.method;neutral⇒ListProducts rpc.service;neutral⇒oteldemo.ProductCatalogService span_name;neutral⇒/oteldemo.ProductCatalogService/ListProducts protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> ms <*>","sample_message":"request_type;neutral⇒outgoing rpc.method;neutral⇒ListProducts rpc.service;neutral⇒oteldemo.ProductCatalogService span_name;neutral⇒/oteldemo.ProductCatalogService/ListProducts attributes;text-textWeak⇒{\"rpc\":{\"grpc\":{\"status_code\":0},\"method\":\"ListProducts\",\"service\":\"oteldemo.ProductCatalogService\",\"system\":{\"name\":\"grpc\"}},\"server\":{\"address\":\"recommendation\"},\"session\":{\"id\":\"d0a6e167-c460-488d-9707-b0bdf92e1a5c\"}} protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒3.9 ms session;right-badge-neutral⇒d0a6e167-c460-488d-9707-b0bdf92e1a5c","service":"recommendation","state":"acknowledged"},{"first_seen_at":"2026-08-28T03:05:50.159619Z","id":13092640876,"is_error":false
save_artifact
→ Saved document '2026-10-29' (Recurring telemetry faults — 2026-10-29, 963 characters).
Active faults remain: 6 active incidents and 177 open issues. One incident is current, with its latest event at 2026-10-30 15:28 UTC; five appear stale, with latest events on 2026-09-26. Current ERROR or ERROR-status telemetry totals 178 events across 7 services, led by otelcol-contrib (135), flagd (13), SYSTEM (11), and recommendation, payment, fraud-detection, and ad (5, 4, 5, and 5 respectively). Critical open issues include frontend POST /api/checkout 500s, checkout shipping-quote failures, frontend GET /api/data 500s, frontend-proxy 503s, accounting PostgreSQL timeouts, and API contract changes updated today. No external notification was sent because no notification channel or explicit send action was provided.
Errors by service7
Current error events
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-30T15:28:04.896218Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-28T14:56:26.016495Z","environment":null,"id":"5c0a8ace-e67c-469a-bf0f-99538beaadb9","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on POST /api/checkout in Frontend Service","type":"log_pattern","updated_at":"2026-10-29T09:06:00.230761Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T11:54:58.097683Z","environment":null,"id":"e76bd12d-0c56-4779-a348-79e758105791","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Shipping Quote Failure in Checkout Service During Shipping Service POST","type":"runtime_exception","updated_at":"2026-10-27T12:00:32.882356Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T11:20:55.453266Z","environment":null,"id":"ec77bf13-09df-46e5-853d-c3ba52a1bfac","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"New 500 Errors on GET /api/data in Frontend Service","type":"log_pattern","updated_at":"2026-10-27T12:01:07.151155Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T04:06:21.024816Z","environment":null,"id":"15012515-a7d4-40f6-81bd-a8e87fb89ab0","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-proxy","severity":"critical","title":"503 GET / Responses Detected on Frontend Proxy","type":"log_pattern","updated_at":"2026-10-27T05:00:53.769381Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-27T01:14:54.998444Z","environment":null,"id":"5840745e-73fb-4980-a54e
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 4.0
otelcol-contrib | 135.0
fraud-detection | 5.0
flagd | 13.0
ad | 5.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-09-05T21:34:06.007289Z","id":13988583315,"is_error":false,"last_seen_at":"2026-10-30T15:31:46.646855Z","level":null,"occurrence_count":217676,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/ListProducts span_name;neutral⇒oteldemo.ProductCatalogService/ListProducts protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> <*> <*>","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/ListProducts span_name;neutral⇒oteldemo.ProductCatalogService/ListProducts protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> <*> session;right-badge-neutral⇒<*>","service":"product-catalog","state":"acknowledged"},{"first_seen_at":"2026-08-27T00:35:45.410864Z","id":12960587685,"is_error":false,"last_seen_at":"2026-10-30T15:31:46.646855Z","level":null,"occurrence_count":4271640,"pattern":"request_type;neutral⇒incoming <*> <*> protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> ms session;right-badge-neutral⇒<*>","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒oteldemo.ProductCatalogService/GetProduct span_name;neutral⇒oteldemo.ProductCatalogService/GetProduct attributes;text-textWeak⇒{\"demo\":{\"product\":{\"id\":\"L9ECAV7KIM\",\"name\":\"Lens Cleaning Kit\"}},\"rpc\":{\"method\":\"oteldemo.ProductCatalogService/GetProduct\",\"response\":{\"status_code\":\"OK\"},\"system\":{\"name\":\"grpc\"}},\"server\":{\"address\":\"10.42.0.175\",\"port\":8080},\"session\":{\"id\":\"11ef49df-a318-4caa-85ea-4c45fa2668e1\"}} protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒3.8 ms session;right-badge-neutral⇒11ef49df-a318-4caa-85ea-4c45fa2668e1","service":"product-catalog","state":"acknowledged"},{"first_seen_at":"2026-03-11T02:42:21.452482Z","id":997992,"is_error":false,"last_seen_at":"2026-10-30T15:31:46.646855Z","level":"INFO","occurrence_count":74893398,"pattern":"severity_text;badge-info⇒INFO Product Found","sample_messag
save_artifact
→ Saved document '2026-10-30' (Recurring Fault Monitoring — 2026-10-30, 861 characters).
Active faults remain: 7 active incidents and 201 open issues. Two incidents are current, with events at 15:30 UTC; five appear stale since 2026-09-26. Newly updated critical issues include frontend connection timeouts to 10.43.3.105:8080, frontend-web GET /api/data 500s, recommendation get_product_list errors, checkout product-not-found failures, and frontend-proxy GET /api/products/{param} 500s. Current telemetry contains 142 ERROR-level or ERROR-status events across 7 services, led by otelcol-contrib (99), flagd (13), SYSTEM (11), recommendation (5), payment (5), ad (5), and fraud-detection (4). No external notification was sent because no notification channel was specified.
Current error telemetry by service7
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"2f0fd306-93d5-42ac-a0c8-c05dbb845bee","issueId":"bf78a952-86d7-42b8-aea0-25e8c6fca9b0","lastEventAt":"2026-10-31T15:30:14.331406Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-31T15:30:14.331406Z"},{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-10-31T15:30:16.003029Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T10:50:58.248664Z","environment":null,"id":"ab591bb7-3257-454c-9320-526592ee710a","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"Frontend Connection Times Out Connecting to 10.43.3.105:8080","type":"runtime_exception","updated_at":"2026-10-31T11:00:32.305696Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T10:48:49.588716Z","environment":null,"id":"a3500f5d-da6e-4091-81a7-f048799ed884","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"500 Errors Increase on frontend-web /api/data GET Requests","type":"log_pattern","updated_at":"2026-10-31T11:00:54.596938Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T06:24:37.376253Z","environment":null,"id":"9ddb5fd0-c62c-47d4-97c9-48668d54104a","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"Recommendation Service Logs Errors in get_product_list Span","type":"log_pattern","updated_at":"2026-10-31T07:01:01.21294Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T06:24:32.837213Z","environment":null,"id":"339c7c59-fee2-488c-96f8-194c08360fa8","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Product Not Found Error for OLJCESPC7Z in Checkout Service","type":"runtime_exception","updated_at":"2026-10-31T07:00:39.545577Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T06:24:32.821Z","environment":null,"id":"5397b2
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (7 rows):
recommendation | 5.0
payment | 5.0
otelcol-contrib | 99.0
fraud-detection | 4.0
flagd | 13.0
ad | 5.0
SYSTEM | 11.0
list_log_patterns
→ [{"first_seen_at":"2026-09-05T17:06:45.991866Z","id":13978665705,"is_error":false,"last_seen_at":"2026-10-31T15:31:46.814668Z","level":null,"occurrence_count":24613,"pattern":"kind;badge-info⇒document url;text-textStrong⇒/ size;right-badge-neutral⇒<*> KiB duration;right-badge-neutral⇒<*> <*> session;right-badge-neutral⇒<*> user <*>","sample_message":"kind;badge-info⇒document url;text-textStrong⇒/ attributes;text-textWeak⇒{\"enduser\":{\"id\":\"cadd35e3-5ad4-4cb8-ac1a-77d07940000d\"},\"http\":{\"response\":{\"body\":{\"size\":3543}},\"response_content_length_uncompressed\":11880},\"server\":{\"address\":\"frontend-proxy\"},\"session\":{\"id\":\"21579206-c6d4-4c23-8847-edd9832ae4f7\"},\"tenant\":{\"id\":\"otel-demo\",\"name\":\"OpenTelemetry Demo\"},\"url\":{\"full\":\"http://frontend-proxy:8080/\",\"path\":\"/\"},\"user\":{\"id\":\"cadd35e3-5ad4-4cb8-ac1a-77d07940000d\"}} size;right-badge-neutral⇒3.5 KiB duration;right-badge-neutral⇒11.9 ms session;right-badge-neutral⇒21579206-c6d4-4c23-8847-edd9832ae4f7 user id;right-badge-neutral⇒cadd35e3-5ad4-4cb8-ac1a-77d07940000d","service":"frontend-web","state":"acknowledged"},{"first_seen_at":"2026-08-27T01:01:34.811857Z","id":12962583725,"is_error":false,"last_seen_at":"2026-10-31T15:31:46.814668Z","level":null,"occurrence_count":598177,"pattern":"kind;badge-action⇒click target;text-textStrong⇒BUTTON","sample_message":"kind;badge-action⇒click target;text-textStrong⇒BUTTON attributes;text-textWeak⇒{\"event_type\":\"click\",\"monoscope\":{\"display\":{\"label\":\"click · Place Order\"},\"kind\":\"interaction\"},\"page\":{\"title\":\"Otel Demo - Cart\",\"url\":\"http://frontend-proxy:8080/cart\"},\"pageview\":{\"id\":\"c5a57d45-283a-4e38-a1d1-babc40529bc2\"},\"server\":{\"address\":\"frontend-proxy\"},\"session\":{\"id\":\"21579206-c6d4-4c23-8847-edd9832ae4f7\"},\"tab\":{\"id\":\"fd285a11-3fb4-4ede-9c57-90f5c851d2fc\"},\"target\":{\"tag_name\":\"button\"},\"target_element\":\"BUTTON\",\"target_xpath\":\"//html/body/div/main/div
save_artifact
→ Saved document '2026-10-31' (Recurring telemetry fault review — 2026-10-31, 605 characters).
Active faults remain: 6 active incidents and 199 open issues. One incident is current through 2026-11-01 15:28 UTC; five appear stale since 2026-09-26. Current telemetry shows 35 ERROR-level or ERROR-status events across six services, led by flagd (15), SYSTEM (10), and recommendation, payment, fraud-detection, and ad (5 each). Recent critical issues include frontend connection timeouts, frontend-web GET /api/data 500s, recommendation get_product_list errors, checkout product-not-found failures, frontend-proxy product 500s, and shipping POST /getquote timeouts/500s. A critical get-quote-service API change was updated at 15:20 UTC. No external notification was sent because no channel or explicit send action was provided.
Current errors by service6
Current error telemetry
5 tool calls
list_incidents
→ {"incidents":[{"closedAt":null,"id":"fd7de639-a6c0-4658-9cfa-6bffd119adf3","issueId":"704419f0-5ca0-41f4-8642-ae5e3541da9e","lastEventAt":"2026-11-01T15:28:08.175829Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-10-07T16:21:04.457325Z"},{"closedAt":null,"id":"fa32515e-40f1-4b46-af85-480ba5d666db","issueId":"3a14088e-059b-49b3-a0f5-ce94c3a28ccf","lastEventAt":"2026-09-26T13:09:05.701585Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-22T16:28:03.707967Z"},{"closedAt":null,"id":"ccd7653a-b2ec-4104-8492-ca2bcf9d933c","issueId":"0ba7a6e0-496f-4308-9008-1623503fb7d4","lastEventAt":"2026-09-26T13:09:07.140372Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.557011Z"},{"closedAt":null,"id":"9924583f-59e5-4fd5-8d49-90cf17cb1833","issueId":"6edd3acf-987f-45b1-bfbe-cd14b5443d53","lastEventAt":"2026-09-26T13:09:06.13768Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:38.299138Z"},{"closedAt":null,"id":"2cc102ff-0883-42f6-9af6-687d2399e8aa","issueId":"8a66df9a-fb14-470e-87e4-6b9b17db8cb1","lastEventAt":"2026-09-26T13:08:04.924965Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.55928Z"},{"closedAt":null,"id":"8bf7b37d-4186-4642-a7cb-3dc0f3b64f7e","issueId":"2101a293-44d6-4ca2-af86-d7062e6afb56","lastEventAt":"2026-09-26T13:09:06.551536Z","phase":"active","projectId":"00000000-0000-0000-0000-000000000000","startedAt":"2026-09-10T13:00:37.124672Z"}],"phase":"active"}
list_issues
→ {"issues":[{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T10:50:58.248664Z","environment":null,"id":"ab591bb7-3257-454c-9320-526592ee710a","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend","severity":"critical","title":"Frontend Connection Times Out Connecting to 10.43.3.105:8080","type":"runtime_exception","updated_at":"2026-10-31T11:00:32.305696Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T10:48:49.588716Z","environment":null,"id":"a3500f5d-da6e-4091-81a7-f048799ed884","recommended_action":"Review the changes and update your integration accordingly.","service":"frontend-web","severity":"critical","title":"500 Errors Increase on frontend-web /api/data GET Requests","type":"log_pattern","updated_at":"2026-10-31T11:00:54.596938Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T06:24:37.376253Z","environment":null,"id":"9ddb5fd0-c62c-47d4-97c9-48668d54104a","recommended_action":"Review the changes and update your integration accordingly.","service":"recommendation","severity":"critical","title":"Recommendation Service Logs Errors in get_product_list Span","type":"log_pattern","updated_at":"2026-10-31T07:01:01.21294Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T06:24:32.837213Z","environment":null,"id":"339c7c59-fee2-488c-96f8-194c08360fa8","recommended_action":"Review the changes and update your integration accordingly.","service":"checkout","severity":"critical","title":"Product Not Found Error for OLJCESPC7Z in Checkout Service","type":"runtime_exception","updated_at":"2026-10-31T07:00:39.545577Z"},{"acknowledged_at":null,"affected_clients":1,"affected_requests":1,"archived_at":null,"created_at":"2026-10-31T06:24:32.821Z","environment":null,"id":"5397b2
run_queryString "level == \"ERROR\" OR status_code == \"ERROR\" | summarize count() by resource.service.name"
→ Results (6 rows):
recommendation | 5.0
payment | 5.0
fraud-detection | 5.0
flagd | 15.0
ad | 5.0
SYSTEM | 10.0
list_log_patterns
→ [{"first_seen_at":"2026-09-05T21:20:20.065684Z","id":13987953498,"is_error":false,"last_seen_at":"2026-11-01T15:31:43.735416Z","level":null,"occurrence_count":341474,"pattern":"request_type;neutral⇒incoming rpc.method;neutral⇒GetAds rpc.service;neutral⇒oteldemo.AdService span_name;neutral⇒oteldemo.AdService/GetAds protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒<*> <*> <*>","sample_message":"request_type;neutral⇒incoming rpc.method;neutral⇒GetAds rpc.service;neutral⇒oteldemo.AdService span_name;neutral⇒oteldemo.AdService/GetAds attributes;text-textWeak⇒{\"demo\":{\"ad\":{\"context_keys\":\"[travel]\",\"count\":1,\"request_type\":\"TARGETED\",\"response_type\":\"TARGETED\"}},\"network\":{\"peer\":{\"address\":\"10.42.0.189\",\"port\":55518},\"type\":\"ipv4\"},\"rpc\":{\"grpc\":{\"status_code\":0},\"method\":\"GetAds\",\"service\":\"oteldemo.AdService\",\"system\":{\"name\":\"grpc\"}},\"server\":{\"address\":\"ad\",\"port\":8080},\"session\":{\"id\":\"0990d15c-3698-4f2f-a6ed-6786aa885e6f\"},\"thread\":{\"id\":78,\"name\":\"grpc-default-executor-12\"}} protocol;right-badge-neutral⇒rpc duration;right-badge-neutral⇒3.4 ms session;right-badge-neutral⇒0990d15c-3698-4f2f-a6ed-6786aa885e6f","service":"ad","state":"acknowledged"},{"first_seen_at":"2026-08-27T01:04:50.667216Z","id":12962806698,"is_error":false,"last_seen_at":"2026-11-01T15:31:43.735416Z","level":null,"occurrence_count":628569,"pattern":"kind;neutral⇒internal <*> duration;right-badge-neutral⇒<*> <*> session;right-badge-neutral⇒<*> user name;right-badge-neutral⇒Demo shopper <*> user id;right-badge-neutral⇒<*>","sample_message":"kind;neutral⇒internal span_name;neutral⇒getAdsByCategory attributes;text-textWeak⇒{\"code\":{\"function\":{\"name\":\"getAdsByCategory\"},\"namespace\":\"oteldemo.AdService\"},\"demo\":{\"ad\":{\"category\":\"flashlights\",\"count\":0}},\"exemplar\":\"true\",\"server\":{\"address\":\"ad\"},\"session\":{\"id\":\"d0b59cd9-4a59-432b-8be9-bf3dafc2231c\"},\"thread\":{\"id\":78,\"
save_artifact
→ Saved document '2026-11-01' (Recurring Fault Metrics Review — 2026-11-01, 962 characters).