A. Observations The E2E’s monitoring team received lots of servers unreachable alerts.
B. Immediate Actions Taken ( * ) • What action we have taken to identify the reported issue? Monitoring team immediately checked the alerts detail and identify that the issue with the selected pools. Therefore they immediately reported the issue to the network team. Network team verify all switches status and logs and did not find any internal issue in the network and reported the same issue to ISP team.
ANALYSIS & ROOT CAUSE: ISP Team confirmed they are observing drop in the traffic and checking it further. They identify the issue with one of the ISP and opened a ticket with them. The identified ISP team confirmed the internal outage from their side.
Actions Taken to Resolve the Issue: ISP have suspended / blocked the network in the network therefore all the started to respond normally once ISP took the action from their end.