Anomalous Error Spike and Service Timeouts

Routine schedule

Active

Every day · Next run 20 Sep, 15:35 UTC

Recent results

check the metrics for anything that looks wierd

Several anomalies stand out. There are 501 ERROR entries, 501 ERROR-status entries, and 501 spans with duration over 1s; these counts matching exactly suggests a concentrated incident or synthetic alert stream rather than ordinary background noise. The latest samples are monitor.alert.triggered, occurring repeatedly within seconds around 15:33–15:34 UTC. Facets also show 87 server-side timeout messages and 44 gRPC DEADLINE_EXCEEDED exceptions. Feature-flag data indicates active failure scenarios including productCatalogFailure, cartFailure, emailMemoryLeak, kafkaQueueProblems, adHighCpu, and intlShippingSlowdown. The initial broad time-series query timed out, so this assessment is based on targeted counts and facets.

Open in Explorer

6 tool calls

run_queryString "| summarize count() by bin_auto(timestamp), level"

→ Error: Query execution failed - ! AnnotatedException ! Underlying exception type: HasqlException Hasql statement error (statement 1/1, prepared=True): SELECT jsonb_build_array(extract(epoch from time_bucket('2 hours', timestamp))::integer, count(*)::float, count(*) OVER()) FROM otel_logs_and_spans WHERE project_id='00000000-0000-0000-0000-000000000000' and (TRUE) GROUP BY time_bucket('2 hours', timestamp) ORDER BY time_bucket('2 hours', timestamp) DESC limit 100 Server error code: 57014 message: canceling statement due to statement timeout Annotations: * Annotation @([Char],Text) ("selectLogTable","SELECT jsonb_build_array(extract(epoch from time_bucket('2 hours', timestamp))::integer, count(*)::float, count(*) OVER())\n FROM otel_logs_and_spans\n WHERE project_id='00000000-0000-0000-0000-000000000000' and (TRUE)\n GROUP BY time_bucket('2 hours', timestamp) \n ORDER BY time_bucket('2 hours', timestamp) DESC\n limit 100 ") CallStack (from HasCallStack): checkpoint, called at src/Models/Apis/LogQueries.hs:312:9 in monoscope-0.1.0.0-inplace:Models.Apis.LogQueries

get_facets

→ Facet data: attributes.rpc.service: "flagd.evaluation.v2.Service" (1611), "oteldemo.ProductCatalogService" (947), "oteldemo.CurrencyService" (864), "flagd.evaluation.v1.Service" (625), "oteldemo.CartService" (480), "oteldemo.RecommendationService" (468), "oteldemo.AdService" (455), "oteldemo.PaymentService" (210), "oteldemo.CheckoutService" (207) attributes.user_id: "9308ede3-a5c3-460b-b08a-1c244a88ec94" (1), "3b09fbe7-7884-4d86-965f-be9fea001b14" (1), "07b407bf-8877-4057-80f7-03d8c706d00f" (1), "9be9c61c-58bc-43eb-8e9e-019e20c0e74e" (1), "748ebb9e-df14-43a0-8b57-fe2564d24bf1" (1), "bcb94f65-2bc1-41db-b418-f364db8ee02d" (1), "a702553a-aef2-4622-84d1-ea37e18b6ff3" (1), "c2bda147-18f7-4d52-a139-3f983aa5ee1e" (1), "d4076839-99f9-4629-8c97-91c6fbd13073" (1), "ba8beeea-29ea-47b3-b045-5fbc596cdbeb" (1) resource.k{integer}s.statefulset.uid: "bdfcc9c3-f1a2-402a-ae84-825767b8774b" (2) body.response.shippingAddress.city: "Mountain View" (64), "Ottawa" (24), "Los Gatos" (21), "Menlo Park" (20), "Redmond" (19), "Santa Clara" (17), "Seattle" (15), "Cupertino" (14), "San Francisco" (13) body.response.items[*].quantity: "1.0" (138), "2.0" (61), "5.0" (47), "10.0" (41), "3.0" (36), "4.0" (32), "12.0" (2), "21.0" (2), "6.0" (1), "9.0" (1) attributes.url.query.productIds: "" (33), "2ZYFJ3GM2N" (28), "9SIQT8TOJO" (17), "0PUK6V6EV0" (17), "L9ECAV7KIM" (16), "1YMWWN1N4O" (16), "66VCHSJNUP" (16), "6E92ZMYYFZ" (14), "LS4PSXUNUM" (12), "OLJCESPC7Z" (10) attributes.messaging.message.body.size: "189.0" (17), "195.0" (15), "194.0" (14), "178.0" (10), "187.0" (10), "192.0" (6), "183.0" (6), "184.0" (6), "167.0" (5), "226.0" (5) attributes.resource spans: "1.0" (191) [email protected]: "94043" (61), "95014" (24), "94025" (20), "K2P1L4" (19), "95032" (19), "95054" (19), "98052" (18), "98109" (15), "94103" (13) attributes.demo.shipping.amount: "8.99" (76), "17.98" (69), "89.9" (39), "26.96" (34), "44.95" (33), "35.96" (30), "62.92" (15), "107.87" (15), "125.85" (11),

count_queryString "level == \"ERROR\""

→ Query 'level == "ERROR"' matches 501 entries

count_queryString "status_code == \"ERROR\""

→ Query 'status_code == "ERROR"' matches 501 entries

count_queryString "duration > 1s"

→ Query 'duration > 1s' matches 501 entries

sample_logsString "level == \"ERROR\""

→ Sample logs: - [ERROR] monitor.alert.triggered (SYSTEM): {..} 84d2aae4-54c8-5e99-a8de-960559c32ae6 2026-09-18T15:34:13.423169Z null monitor.alert.triggered 2 - [ERROR] monitor.alert.triggered (SYSTEM): {..} 5188b48b-dc20-5bd0-8b75-01ff2441be62 2026-09-18T15:34:10.892507Z null monitor.alert.triggered 2 - [ERROR] monitor.alert.triggered (SYSTEM): {..} 1f54b723-6440-5848-9d1c-a32ecc22c61b 2026-09-18T15:34:08.291779Z null monitor.alert.triggered 2 - [ERROR] monitor.alert.triggered (SYSTEM): {..} ed422d69-3b63-5db3-925a-28f1921c3b02 2026-09-18T15:34:04.614580Z null monitor.alert.triggered 2 - [ERROR] monitor.alert.triggered (SYSTEM): {..} ec423aeb-fe75-50b4-a3f3-45866b2c2daf 2026-09-18T15:33:03.708562Z null monitor.alert.triggered 1