OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#14154 — 52C03
Incident Report for Network & Infrastructure
Resolved
The FEX in this rack is not responding. It separated fromt eh 2 x N5.
We will investigate.

Update(s):

Date: 2015-07-28 09:37:42 UTC
all up

# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 FEX0100 Online N2K-C2248TP-1GE SSI142520WD
101 FEX0101 Online N2K-C2248TP-1GE SSI14440285
102 FEX0102 Online N2K-C2248TP-1GE SSI142810TW
103 FEX0103 Online N2K-C2248TP-1GE SSI1444023E
104 FEX0104 Online N2K-C2248TP-1GE SSI144506J8
105 FEX0105 Online N2K-C2248TP-1GE SSI14450H7E
106 FEX0106 Online N2K-C2248TP-1GE SSI144401WG
107 FEX0107 Online N2K-C2248TP-1GE SSI1444023D
108 FEX0108 Online N2K-C2248TP-1GE SSI144401X3
109 FEX0109 Online N2K-C2248TP-1GE SSI14280VQ4
111 FEX0111 Online N2K-C2248TP-1GE SSI144401XE

No more issues on this couple.

Date: 2015-07-28 09:37:24 UTC
The FEXs are remounting
# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
100 FEX0100 Connected N2K-C2248TP-1GE SSI142520WD
101 FEX0101 Connected N2K-C2248TP-1GE SSI14440285
102 FEX0102 Connected N2K-C2248TP-1GE SSI142810TW
103 FEX0103 Connected N2K-C2248TP-1GE SSI1444023E
104 FEX0104 Connected N2K-C2248TP-1GE SSI144506J8
105 FEX0105 Connected N2K-C2248TP-1GE SSI14450H7E
106 FEX0106 Connected N2K-C2248TP-1GE SSI144401WG
107 FEX0107 Connected N2K-C2248TP-1GE SSI1444023D
108 FEX0108 Connected N2K-C2248TP-1GE SSI144401X3
109 FEX0109 Connected N2K-C2248TP-1GE SSI14280VQ4
111 FEX0111 Connected N2K-C2248TP-1GE SSI144401XE

Date: 2015-07-28 09:37:08 UTC
Reloading b in progress

n5# reload power-cycle
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y

Date: 2015-07-28 09:36:56 UTC
The FEXs are up but the system is still a bit overloaded. We need to use MTS buffer n5# sh internal system mts buffers
MTS buffers in use = 234


Date: 2015-07-28 09:34:02 UTC
It definitely seems to be a side effect of the upgrade to 7.1.1. The FEXs are randomly disconnecting.

After a power-cycle reload, the FEXs are back.
I'll wait until all the FEXs are up before reloading the second N5.

Date: 2015-07-28 09:32:39 UTC
The FEX is not recognised.

We checked the optical and fibre links and tested a spare FEX.

I rebooted one of the 2 Nexus. It may be a side effect of the upgrade to 7.1.1. this evening (http://status.ovh.net/?do=details&id=10277).
Posted Jul 28, 2015 - 09:31 UTC