We were configuring a pair of sophisticated layer 7 Firewalls for high availability (stateful fail over). Stateful failover allows the firewalls to maintain actual session states, providing an almost seamless transition from one unit to the other. Layer 7 refers to firewalls that provide application-layer filtering. In test mode everything worked fine and when first brought them on line, they worked perfectly.

However… when some significant changes were necessary the two units got out of sync. When complex technology fails it really makes a BIG mess. In this case the high availability feature went out of control and the two firewalls failed to work at all. Even worse the unit with the older configuration tried to take control and erase the configuration of the primary unit!

So while we installed two units to protect against a single failure, the end result was total failure. The story gets worse from here. Once the two units were re-synchronized some, but not all, outside traffic failed to pass through the pair of firewalls. This was very difficult to diagnose due to the inconsistent nature of this failure. Ultimate we rebooted an upstream router and the problem was resolved.

There is a high cost to high availability, often a simple redundant solution is safer and more cost effective than using complex technology.

Tags: fail over, Firewall, High Availability

This entry was posted on Thursday, December 4th, 2008 at 3:59 am and is filed under Firewall, Hardware. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.