OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#12269 — 176.31.232.0/24 and 176.31.233.0/24
Incident Report for Network & Infrastructure
Resolved
One of the 2 nexus 5k rebooted twice, once at 2pm and now.
There is no impact because the packets are forwarded by the 2nd nexus of the pair.

The cause is the bug on port-security,
Reason: Reset triggered due to HA policy of Reset
System version: 6.0(2)N2(2)
Service: eth_port_sec hap reset

We are monitoring the switch stability before updating it.

Update(s):

Date: 2014-12-23 06:38:41 UTC
The second fex was updated.

all servers are up.

Date: 2014-12-23 06:38:11 UTC
We are updating the second n5.

Date: 2014-12-23 06:37:51 UTC
All fex was updated hot swap :

Module 100: Non-disruptive upgrading.
SUCCESS

Module 101: Non-disruptive upgrading.
SUCCESS

Module 102: Non-disruptive upgrading.
SUCCESS

Module 103: Non-disruptive upgrading.
SUCCESS

Module 104: Non-disruptive upgrading.
SUCCESS

Module 105: Non-disruptive upgrading.
SUCCESS

Module 106: Non-disruptive upgrading.
SUCCESS

Module 107: Non-disruptive upgrading.
SUCCESS

Module 108: Non-disruptive upgrading.
SUCCESS

Module 109: Non-disruptive upgrading.
SUCCESS

Module 111: Non-disruptive upgrading.
SUCCESS

Install has been successful.

Date: 2014-12-23 06:37:24 UTC
All fex have downloaded their new images:

2014 Dec 23 05:17:19 sw %$ VDC-1 %$ %SATCTRL-FEX101-2-SATCTRL_IMAGE: FEX101 Image update complete. Install pending
2014 Dec 23 05:17:21 sw %$ VDC-1 %$ %SATCTRL-FEX100-2-SATCTRL_IMAGE: FEX100 Image update complete. Install pending
2014 Dec 23 05:17:31 sw %$ VDC-1 %$ %SATCTRL-FEX102-2-SATCTRL_IMAGE: FEX102 Image update complete. Install pending
2014 Dec 23 05:17:50 sw %$ VDC-1 %$ %SATCTRL-FEX103-2-SATCTRL_IMAGE: FEX103 Image update complete. Install pending
2014 Dec 23 05:17:55 sw %$ VDC-1 %$ %SATCTRL-FEX104-2-SATCTRL_IMAGE: FEX104 Image update complete. Install pending
2014 Dec 23 05:18:19 sw %$ VDC-1 %$ %SATCTRL-FEX105-2-SATCTRL_IMAGE: FEX105 Image update complete. Install pending
2014 Dec 23 05:18:21 sw %$ VDC-1 %$ %SATCTRL-FEX106-2-SATCTRL_IMAGE: FEX106 Image update complete. Install pending
2014 Dec 23 05:18:28 sw %$ VDC-1 %$ %SATCTRL-FEX107-2-SATCTRL_IMAGE: FEX107 Image update complete. Install pending
2014 Dec 23 05:18:36 sw %$ VDC-1 %$ %SATCTRL-FEX108-2-SATCTRL_IMAGE: FEX108 Image update complete. Install pending
2014 Dec 23 05:18:42 sw %$ VDC-1 %$ %SATCTRL-FEX109-2-SATCTRL_IMAGE: FEX109 Image update complete. Install pending
2014 Dec 23 05:18:57 sw %$ VDC-1 %$ %SATCTRL-FEX111-2-SATCTRL_IMAGE: FEX111 Image update complete. Install pending

Date: 2014-12-23 06:36:54 UTC
The issue is in progress. The image is being downloaded on the fex:

Pre-loading modules.
[This step might take upto 20 minutes to complete - please wait.]
[*Warning -- Please do not abort installation/reload or powercycle fexes*]
[# ]

Date: 2014-12-23 06:36:15 UTC
No more space :

# install all system bootflash:n5000-uk9.7.0.5.N1.1a.bin kickstart bootflash:n5000-uk9-kickstart.7.0.5.N1.1a.bin
Pre-upgrade check failed. Return code 0x40930062 (free space in the filesystem is below threshold).

We will start the plugin to make some space:

sw.176.31.232.249# load bootflash:plug
Loading plugin version 6.0(2)N2(3)
###############################################################
Warning: debug-plugin is for engineering internal use only!
For security reason, plugin image has been deleted.
###############################################################
Successfully loaded debug-plugin!!!
Linux(debug)# cd /var/sysmgr/logs
Linux(debug)# cat /dev/null > lcsm.out
Linux(debug)# df -h | grep sysmgr
none 1000M 32M 969M 4% /var/sysmgr
none 400M 84K 400M 1% /var/sysmgr/ftp
none 100M 0 100M 0% /var/sysmgr/ftp/cores
none 400M 15M 386M 4% /var/sysmgr/startup-cfg
Linux(debug)# exit

Date: 2014-12-23 06:35:30 UTC
The n5 A is not stable (ssh connection refused, etc.). We rebooted it to start within a stable basis before updating the pair to 7.0.5.N1.1a.
Posted Dec 23, 2014 - 06:34 UTC