Get webhook notifications whenever Network & Infrastructure creates an incident, updates an incident, resolves an incident or changes a component status.
We have BPG/links that are flapping on some of our g5-a9. It is also affecting the vRack.
Update(s):
Date: 2016-11-22 13:50:25 UTC Hello,
ACLs allow for blocking/accepting traffic to and from switches/routers.
When reconfiguring the ACLs on the switches which manage the vRacks, we had a debug of
deployment of the ACLs on some switches in the scripts that manage this update.
The lines are not correctly configured on 50% of the switches, which has provoked a
blockage of access to these switches.
So the switches stopped communicating with the rest of the infrastructure.
We had to manually intervene on these switches, with a serial cable, to put
back the lines of the ACL configurations.
This explains the (very) long time that was needed to correct the issue.
We will modify the scripts to verify the status of the configuration before
applying the modification in order to make sure that the configuration in place
corresponds to the expected state prior to the update.
Only then will we apply the modifications.
This verification should have already been coded.
Overconfidence is deadly when it comes to the network.
We apologize for the disruption of service.
Best Regards,
Octave
Date: 2016-11-21 15:50:31 UTC All switches are back now.
Date: 2016-11-21 15:42:49 UTC Some additional details:
This failure was caused by a bug in the ACL update system of some switches.
We have identified the specific causes of the problem that are found in the configuration of these switches and a problem in the code checking.
Date: 2016-11-21 15:26:54 UTC There are 6 switch that remains affected by this issue
Date: 2016-11-21 15:17:53 UTC We found the source, it seems to be isolating to the following switches: