VPN Tunnel fail when removing SLAVE from HA
I have experienced and issue that I was not prepared to encounter. Here is the scenario:
1000D in HA running 5.2.4
Multiple IPSEC tunnels with OSPF running inside
In order to upgrade to 6.0.5, wanted to take the slave out, upgrade it to 6.0.5 using the upgrade path as recomended by Fortinet, and put back in service at a later maintenance window. This is to shorten the outage from 6 outages due to the upgrade path to only 1 (swap of the old master to the new 6.0.5 master). Upgrading an HA set that has VPN tunnels and OSPF riding overtop will cause an outage everytime the ha fails over for each upgrade step. IPSEC HA sessions are not carriered over.
instead of doing the exec shutdown on the slave, i went and disconnected the 2 HA ports, and 1/2 sec later disconnected the 2 10Gb ports that all the traffic passes through. This caused all the tunnels to go down and ultimately the OSPF routing also. For some reason, the tunnels would not come up. Having management breathing down my neck, I re-instated the Slave and reconnected the 4 cables (2xHA and 2x10Gb). after a few minutes seeing the tunels where not comming back up, i forced resettes on of the tunnels and all came back on-line.
Now, of course, need to do a post mortum and root cause analysis, but I have no data to go on apart from the fact that the firewalls are running old unsupported release.
I know that there is a MTU mismatch issue on OSPF interfaces introduced in 5.2 but not sure if this is the cause.
Anyone have other ideas or have encountered this issue before?