rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#15229 — RBX

Attached to Project— Network
Incident
Roubaix
In progress
50%
There was an optical flap at RBX DC.
We are investigating
Comment by OVH - Wednesday, 06 December 2017, 22:14PM

https://travaux.ovh.net/?do=details&id=28864


Comment by OVH - Wednesday, 06 December 2017, 22:14PM

We are going back 100G links to RBX.
For now, we find a saturation between RBX and AMS.


Comment by OVH - Wednesday, 06 December 2017, 22:15PM

We are shutting AMSIX / Worldstream temporarily.


Comment by OVH - Wednesday, 06 December 2017, 22:16PM

Good evening,
On November 9, we encountered a big problem
on our optical network at RBX. The problem was related
to a software bug on the equipment we use
which caused the deletion of the configuration.

Since then we have updated the equipment on everything
our network. Also to prevent this type of bug from
never again causes a worry about our DCs, we have
decided to divide equipment clusters into 3
on the RBX website. So, if we ever have again
this bug, the configuration would only impact 30%
traffic.

During the preparation of the maintenance that was to
start at 23:00, the configuration disappeared again
at 8:20 pm and all the links were down again. !!!!!
The database has been deleted while we are using
the latest software version. So there is another bug!
http://travaux.ovh.net/?do=details&id=28835

We quickly returned the configuration. Some
links have returned, but not yet all. Currently
traffic passes Paris> GRA> AMS> RBX instead of
directly Paris> RBX. Hence the important latencies.

We look with Cisco to understand why all
the links are not UP while the configuration was
delivery to RBX.

We are not going to do the intervention tonight.
I want to understand why the configuration fades
and how can we do this maintenance without
again have the least concern about production!

Sincerely
Octave


Comment by OVH - Wednesday, 06 December 2017, 22:17PM

The configuration is in place but it does not work.
We erase each configuration and rebuild them to
by hand.

RBX <> LDN: UP


Comment by OVH - Wednesday, 06 December 2017, 22:17PM

Traffic isolation between Amsterdam and Warsaw to reduce saturation


Comment by OVH - Wednesday, 06 December 2017, 22:25PM

Isolation of VAC Roubaix to reduce saturation


Comment by OVH - Wednesday, 06 December 2017, 22:37PM

We rebuild the circuits by hand Paris <> RBX.
In a few minutes we should have 400G of UP capability which will reduce the latency in the network.


Comment by OVH - Wednesday, 06 December 2017, 22:48PM

4 GSW <> RBX circuits are UP. The others are reactivated.


Comment by OVH - Wednesday, 06 December 2017, 22:56PM

4 other circuits are UP.
We rebuild the other circuits that are still down.


Comment by OVH - Wednesday, 06 December 2017, 23:42PM

RBX<>FRA 400G UP


Comment by OVH - Thursday, 07 December 2017, 00:08AM

RBX<>LDN 600G UP


Comment by OVH - Thursday, 07 December 2017, 00:09AM

RBX<>BHS UP


Comment by OVH - Thursday, 07 December 2017, 00:22AM

We are working to bring up the links between RBX and SBG. 200G are actually UP


Comment by OVH - Thursday, 07 December 2017, 00:55AM

we brought a 100G link between RBX and AMS


Comment by OVH - Thursday, 07 December 2017, 01:05AM

600G between RBX and AMS


Comment by OVH - Thursday, 07 December 2017, 01:05AM

600G between RBX and AMS


Comment by OVH - Thursday, 07 December 2017, 01:06AM

Links to BRU POP are now UP


Comment by OVH - Thursday, 07 December 2017, 01:27AM

Everything is working now.

We are so sorry for this breakdown, we will working on this to avoid any other issue like this one


Comment by OVH - Thursday, 07 December 2017, 02:37AM

600G between RBX et GSW (Paris) are up