Due to a fibre cut between Telehouse2 and Equinix in Paris, some Gandi services are disrupted.  We are attempting to mitigate as much as possible, but some services are dependent on this inter-site connectivity.  Our fibre suppliers are working to resolve the issue as soon as possible.

14:00 CET (13:00 GMT): Connectivity has been restored.

One of our transit suppliers has scheduled emergency maintenance during the night of 24 to 25 November, starting at 23:00 GMT and ending at 02:00 GMT.  During this maintnenance, the Gandi network may experience some instability for up to an hour.

We apologise for any inconvenience that may be caused.

Following a network incident with one of our IP transits, Gandi's network is currently unstable. Our team is currently fixing the problem.

12:55 GMT:  Our transit operator has resolved the incident.  All of our connections are again operational and stable.

As a result of a large denial of service attack followed by transit BGP routing updates, we encountered network instability (several interruptions lasting a few minutes) at 16:30 GMT and then again between 17:00 GMT and 18:00 GMT.  The situation is now stable and we are analysing the root cause of the issue.

We have experienced a partial power failure at Equinix PA2 at 10:07 CEST this morning.  The power failure lasted a few seconds, but had a knock-on effect on some equipment, notably some older generation equipment with single power supply causing them to reboot, and lose network connectivity with some backend services.

Our teams are working to restore all affected services.

Due to large DDoS attacks against hosts within the Gandi Network, network connectivity is suboptimal.  We are seeing peak traffic in excess of 30Gbps as part of this attack.  We are attempting to mitigate the effects of this attack, but some of these actions will result in sub-optimal routing between the Gandi network and some of our customers as long as the attacks persist.

We apologise for any inconvenience caused.

Due to an equipment failure some services are unavailable (blog admin, mail)

Our teams are working on it

We will give you more information as soon as possible.

EDIT  02:21 CEST:  situation is back to normal. We have had a loadbalancer failure and we have also encountered some difficulties switching to the backup loadbalancer.

Please accep our sincerest apologies for the inconvenience.

We are currently looking into reports of Comcast users experiencing difficulties accessing Gandi's services.

Update: The problem was due to a bug in bind bind9.8.0 which we have just corrected. Please accept our apologies for any inconvienence this may have caused.

We are planning a couple of scheduled maintenance windows to perform software upgrades on core and distribution network equipment.

The planned maintenance windows are as follows:

Monday 16 May 2011, 05:00 - 06:00 GMT.  For operations in Baltimore - Expected impact is minimal, though potentially up to 10 minutes of connectivity disruption in case of the need to roll-back the upgrade.

Update 07:00 UTC:  The upgrade on the Baltimore Core was successful.  We have delayed the upgrade to the Distribution, however as there will be an impact on downstream equipment.  We will schedule a new maintenance window for this equipment.

Thursday 26 May 2011, 00:00 - 02:00 GMT - to perform software upgrades to distribution hardware in Paris.  As with Baltimore, minimal impact is expected, but may incur up to 10 minutes of connectivity disruption in case a rollback is required.

Update 01:42 GMT (03:42 CET) - we have upgraded one of the distribution switch systems to the new software.  We will reschedule the other two switching systems for a later date.  Currently hosting operations (stop/start/restart/create, etc.) are suspended and will resume once the systems have stabilised and our teams have verified the integrity of the architecture.  We will keep you updated.

Update: 02:00 GMT (04:00 CET):  Hosting Operations have been resumed.  No further maintenance work is scheduled for tonight.

Friday 27 May 2011, 03:30 - 05:30 GMT - software upgrades on the core routers.  Impact will be localised to external transit and/or peering connectivity for each the associated routers.  Traffic will be re-routed to alternative paths, but minor route disruption is to be expected as routing reconverges.  There will be up to six minor disruptions of this kind during this maintenance window as each router will need to be reloaded during the upgrade.

Update 05:04 GMT:  The core router software upgrades have been completed and this maintenance is now terminated.  The remaining distribution systems from yesterday's maintenance window are postponed for a future date and will be announced separately.

We apologise for any inconvenience.

Our transatlantic connections are currently degraded due to multiple fibre cuts with separate providers.

One connection (Paris Equinx to Baltimore via Yellow) is out of service due to a cut on the Yellow submarine cable system and our supplier estimates 14 days to resolution. 

A second connection (Paris Telehouse2 to Baltimore via AC2 and TAT14) is currently out of service due to a fibre cut along the I-295 corridor between Washington and Baltimore. The supplier of this circuit has engineers in the field currently working to resplice the broken fibre cables, but we have not yet received an estimated time to fix.  This fibre cut is also impacting one of our IP transit connections in Baltimore.

A third connection (Paris Equinix to Baltimore via Apollo South) is working, but is experiencing BER (Bit-Error-Rate) on the line trace.  We have raised this issue with the supplier to investigate further. 

During this time, network performance between Baltimore and Paris is degraded and may show signs of packet loss and/or retransmissions.  We will keep you updated as we receive further updates from our suppliers.

We apologise for any inconvenience.

Update: 05:54 GMT (07:54 CEST):  The supplier for the connection Paris Telehouse2 to Baltimore via AC2/TAT14 has reported that the engineers have exposed the fibre and have begun prepping for splicing.  The splicing operation will commence within the next hour.

Update: 06:10 GMT (08:10 CEST):  Our supplier of the Paris Equinix to Baltimore connection via Apollo South has indicated a fibre cut in Washington affecting protected and unprotected services which explains the performance degradation on this circuit.  They have not yet provided a time to fix.

Update 06:55 GMT (08:55 CEST):  The supplier for the circuit via Apollo South has reported that splicing work is commencing and have provided an estimated time to repair of approximately six hours.

Update 07:35 GMT (09:36 CEST):  The supplier for the circuit over AC2/TAT14 has indicated that resplicing has commenced.  They have not yet indicated an estimated time to repair.

Update 08:34 GMT (10:34 CEST):  Our connection from Paris Telehouse2 to Baltimore via AC2/TAT14, and the affected IP Transit connection at Baltimore have both been restored.  Our supplier indicates that splicing work is still ongoing and we are monitoring the situation closely.  The connection from Paris Equinix to Baltimore via Apollo South is on the backup fibre path between Washington and Baltimore currently and resplicing of the fibre cut in Washington is ongoing.

Update 15:18 GMT (17:19 CEST) : The circuit via Apollo South is now  restored since 09:21 GMT (11:21 CEST).