25/09/2022 12:03 until 25/09/2022 20:45
At 12h03 our monitoring detected an issue with some virtual machines on the hardware node HV10.
During a first investigation there is an indication that a hardware failure is causing the issue. In the datacenter there was an attempt to do a reboot of the hardware machine, but that attempt failed. The hardware vendor - HPE - will also start an intervention soon.
All services are operational again
We will send a a full post incident report to customers with cloud servers that were impacted by this incident.
History
Update 13h50 : we are still investigating.
Update 14h30 : HPE has sent technicians to the datacenter to resolve the hardware problem.
Update 15h25 : De software of the virtualisation software is developing a work-around, so we can move the virtual machines.
Update 15h45 : We are migraring the virtual machines one-by-one to other hardware. The first machines are back online.
Update 16h20 : Most servers are migrated to other hardware nodes and are running.
Update 16h45 : All virtual servers have been migrated to other hardware nodes.
Update 20h00 : The hardware node HV10 is not repaired by an engineer of HPE. All tests are succesfull and HV10 is re-integrated in the cloud cluster. We will be moving some non-critical servers to HV10. We will be monitoring this very closely.