Monday, September 21, 2009

Question on Multi-subnet clusters and using static routes with Exchange 2007 CCR on Windows Server 2008

Q. We are deploying a multisite Exchange 2007 SP1 cluster using Cluster Continuous Replication (CCR). The two cluster nodes will be located in separate datacenters. Exchange runs on Windows Server 2008 SP2 and we plan to have the public and private interfaces located in different subnets in each datacenter. As you know, this means we must use routing between the cluster nodes.
We have no problem configuring the public interface according to the instructions in "". But when we configure the default gateway on the private interface, we receive the warning message shown in Figure below:

Based on this warning message, we suspect things will not work properly if we specify multiple default gateways on each node in our multisite CCR cluster. This leads us to our question: How should we configure the private network interface in this type of scenario?

A. Because specifying multiple default gateways on a multisite CCR cluster will cause major issues. The proper configuration requires persistent, static routes for each private interface.

To get started, make sure the public interface is listed first on the connection order list under Advanced Settings in the Network Connections control panel. Next, make sure you have specified a default gateway on the public network interface for each cluster node.

Finally, configure routes on the private interfaces so that all traffic that doesn't match the route created will use the default gateway of the public interface

The –P parameter specifies that the created routes are persistent and won't be cleared after a reboot. This configuration will ensure proper networking for each interface in the cluster nodes. Its recommended to configure the private network as a mixed network so that the Enable-ContinuousReplicationHostName cmdlet can be used to direct replication activity over the redundant network.

With the enhancements in Windows 2008 to allow for multi-subnet clustering it is becoming more common to see this utilized with Exchange 2007 SP1 installations.

When implementing a clustered solution, it is a requirement that there be a minimum of two interfaces on each node, and that each node can maintain communications across those interfaces. Two different fashions to implement this requirement with multi-subnet clusters:
  • The “public” interface of each node resides in different subnets with the “private” interfaces residing in a stretched subnet.
  • The “public” interface of each node resides in different subnets with the “private” interfaces also residing in different subnets.
For users that have a configuration where both network interfaces are in different subnets this will generally require routing between those two subnets. A common mis-configuration that I see in this design is the use of default gateways on both of these network interfaces.

When a user attempts to configure two network interfaces each with a default gateway, the error in above screenshot is noted from the operating system.

The text in this message is specifically important as it highlights at this time that this configuration will not produce the desired results.

The most likely cluster configuration where Exchange is used, with this type of clustering, is cluster continuous replication (CCR). When multiple default gateways are defined, users may see inconsistent results in the performance and ability to replicate logs between the nodes. The replication issues between nodes are also exacerbated when continuous replication hostnames are used utilizing the secondary networks with the default gateway assigned. These issues are secondary to any issues that the cluster service many have maintaining communications between the nodes and any communications issues clients may have connecting to the nodes.

If the default gateways are removed from the “private” adapters, reliable routed communications can only occur over the “public” interface. So…if two default gateways cannot be used, how should we ensure proper communications over both the “public” interface and “private” interface where both reside in different routed subnets.

The first part of this solution is to ensure that the binding order of the network interfaces is set correctly in the operating system. To confirm the binding order:
  • Open the network connections control panel.
  • Choose the advanced menu (if menu is disabled, enable it by selecting Organize –> Layout –> Menu Bar).
  • Select advanced settings from the advanced menu.
  • On the adapters and bindings tab, ensure that the “public” interface is first in the list, with all secondary interfaces following after.
The second part of the solution is to maintain the default gateway on the “public” interface.

The third part of the solution is to enable persistent static routes on the “private” interfaces. In terms of the routes we simple need to configure routes to other “private” networks using gateway addresses that have the ability to route between those “private” networks. All other traffic not matching this route should be handled by the default gateway of the “public” adapter.

Let’s take a look at an example.

I desire to have a two node Exchange 2007 SP1 CCR cluster on Windows 2008 with each node residing in a different subnet.

Node A:

  • IP Address
  • Subnet Mask
  • Default Gateway
  • IP Address
  • Subnet Mask
  • Gateway on network
Node B:

  • IP Address
  • Subnet Mask
  • Default Gateway
  • IP Address
  • Subnet Mask
  • Gateway on network
(Note that gateway on network is not the default gateway setting but is the gateway on the private interface network that can route packets to the private network on the other nodes.)

In this case I would want to establish the necessary persistent static routes on each node. In order to accomplish this, I can use the route add command. The structure of the route command:

NodeA: Route add mask –p

NodeB: Route add mask –p

The –p switch will ensure that the routes are persistent lasting after a reboot. Failure to use the –p will result in the routes being removed post a reboot operation.

You can verify that the routes are correct by running route print and reviewing the persistent route information.

By utilizing only a default gateway on the “public” adapter, and static routes on the “private” adapters, you can ensure safe routed paths for client communications, cluster communications, and replication service log shipping.

No comments: