Question on Multi-subnet clusters and using static routes with Exchange 2007 CCR on Windows Server 2008

Q. We are deploying a multisite Exchange 2007 SP1 cluster using Cluster Continuous Replication (CCR). The two cluster nodes will be located in separate datacenters. Exchange runs on Windows Server 2008 SP2 and we plan to have the public and private interfaces located in different subnets in each datacenter. As you know, this means we must use routing between the cluster nodes.

We have no problem configuring the public interface according to the instructions in "http://technet.microsoft.com/en-us/library/aa997910.aspx". But when we configure the default gateway on the private interface, we receive the warning message shown in Figure below:

Based on this warning message, we suspect things will not work properly if we specify multiple default gateways on each node in our multisite CCR cluster. This leads us to our question: How should we configure the private network interface in this type of scenario?

A. Because specifying multiple default gateways on a multisite CCR cluster will cause major issues. The proper configuration requires persistent, static routes for each private interface.

To get started, make sure the public interface is listed first on the connection order list under Advanced Settings in the Network Connections control panel. Next, make sure you have specified a default gateway on the public network interface for each cluster node.

Finally, configure routes on the private interfaces so that all traffic that doesn't match the route created will use the default gateway of the public interface

The –P parameter specifies that the created routes are persistent and won't be cleared after a reboot. This configuration will ensure proper networking for each interface in the cluster nodes. Its recommended to configure the private network as a mixed network so that the Enable-ContinuousReplicationHostName cmdlet can be used to direct replication activity over the redundant network.

With the enhancements in Windows 2008 to allow for multi-subnet clustering it is becoming more common to see this utilized with Exchange 2007 SP1 installations.

When implementing a clustered solution, it is a requirement that there be a minimum of two interfaces on each node, and that each node can maintain communications across those interfaces. Two different fashions to implement this requirement with multi-subnet clusters:

• The “public” interface of each node resides in different subnets with the “private” interfaces residing in a stretched subnet.
• The “public” interface of each node resides in different subnets with the “private” interfaces also residing in different subnets.

For users that have a configuration where both network interfaces are in different subnets this will generally require routing between those two subnets. A common mis-configuration that I see in this design is the use of default gateways on both of these network interfaces.

When a user attempts to configure two network interfaces each with a default gateway, the error in above screenshot is noted from the operating system.

The text in this message is specifically important as it highlights at this time that this configuration will not produce the desired results.

The most likely cluster configuration where Exchange is used, with this type of clustering, is cluster continuous replication (CCR). When multiple default gateways are defined, users may see inconsistent results in the performance and ability to replicate logs between the nodes. The replication issues between nodes are also exacerbated when continuous replication hostnames are used utilizing the secondary networks with the default gateway assigned. These issues are secondary to any issues that the cluster service many have maintaining communications between the nodes and any communications issues clients may have connecting to the nodes.

If the default gateways are removed from the “private” adapters, reliable routed communications can only occur over the “public” interface. So…if two default gateways cannot be used, how should we ensure proper communications over both the “public” interface and “private” interface where both reside in different routed subnets.

The first part of this solution is to ensure that the binding order of the network interfaces is set correctly in the operating system. To confirm the binding order:

• Open the network connections control panel.
• Choose the advanced menu (if menu is disabled, enable it by selecting Organize –> Layout –> Menu Bar).
• Select advanced settings from the advanced menu.
• On the adapters and bindings tab, ensure that the “public” interface is first in the list, with all secondary interfaces following after.

The second part of the solution is to maintain the default gateway on the “public” interface.

The third part of the solution is to enable persistent static routes on the “private” interfaces. In terms of the routes we simple need to configure routes to other “private” networks using gateway addresses that have the ability to route between those “private” networks. All other traffic not matching this route should be handled by the default gateway of the “public” adapter.

Let’s take a look at an example.

I desire to have a two node Exchange 2007 SP1 CCR cluster on Windows 2008 with each node residing in a different subnet.

Node A:

Public

• IP Address 192.168.0.100
• Subnet Mask 255.255.255.0
• Default Gateway 192.168.0.254

Private

• IP Address 10.0.0.1
• Subnet Mask 255.255.255.0
• Gateway on network 10.0.0.254

Node B:

Public

• IP Address 192.168.1.100
• Subnet Mask 255.255.255.0
• Default Gateway 192.168.1.254

Private

• IP Address 10.0.1.1
• Subnet Mask 255.255.255.0
• Gateway on network 10.0.1.254

(Note that gateway on network is not the default gateway setting but is the gateway on the private interface network that can route packets to the private network on the other nodes.)

In this case I would want to establish the necessary persistent static routes on each node. In order to accomplish this, I can use the route add command. The structure of the route command:

NodeA: Route add 10.0.1.0 mask 255.255.255.0 10.0.0.254 –p

NodeB: Route add 10.0.0.0 mask 255.255.255.0 10.0.1.254 –p

The –p switch will ensure that the routes are persistent lasting after a reboot. Failure to use the –p will result in the routes being removed post a reboot operation.

You can verify that the routes are correct by running route print and reviewing the persistent route information.

By utilizing only a default gateway on the “public” adapter, and static routes on the “private” adapters, you can ensure safe routed paths for client communications, cluster communications, and replication service log shipping.

----------------------------------------------------------------------------------------

If you have more than one public folder database in your organization, do NOT put it on a CCR cluster

Microsoft has long stated that locating a public folder database on a CCR cluster in an organization where there is more than public folder database is no supported. As with many things that Microsoft says are unsupported but that actually works, I took this to be more of a guideline than a hard and fast rule.

In one of my Exchange environments, we have a lot of regional offices with between 1,000 and 3,000 users. In the 1,000 user location, we did not want to invest additional hardware in a dedicated public folder server so we threw the public folder database on the CCR cluster. During testing that we move the clustered mailbox server (CMS) from node to the other, the public folder database moved just fine.

You guessed it, the active node failed a few weeks ago and the public folder database did not remount. We could not get it to mount at all. Period. End of story. Kaput. Dead database. I had a PSS engineer sitting right next to me and he could not rescue it either. Exchange Server 2007 SP1 is apparently hard coded not to allow the database to be recovered, even if you accept a lossy failover.

So, the moral of the story. If you have more than one public folder database in your organization, do NOT put it on a CCR cluster.

In previous versions of Exchange Server, Exchange Virtual Servers (EVSes) are not very different from standalone servers. Besides mailboxes, they can host protocol virtual servers (SMTP, IMAP4, POP3, HTTP/OWA), Public Folders, etc.

Exchange Server 2007's clustering model is simplified further to provide high availability for mailboxes. There is no protocol support - SMTP is the domain of Hub Transport servers, IMAP4, POP3 and HTTP (OWA, Outlook Anywhere or RPC over HTTP, Exchange ActiveSync) are the responsibility of Client Access Server role. Unlike standalone/non-clustered Exchange Server 2007 servers, Clustered Mailbox Servers (CMS - the Exchange 2007 term for EVS) do not co-exist with any other server role.

Clustered Mailbox Servers can host Public Folders, but there are some caveats. The Public Folder Store hosted by the CMS should be the only Public Folder Store in the Organization. If you have Public Folder Stores on other Exchange servers in the Organization, the Public Folder Store on a Clustered Mailbox Server will fail to mount in the case of an unscheduled failover, until the original server and all transaction logs for the Storage Group hosting the Public Folder Store are available.

This is documented at http://technet.microsoft.com/en-us/library/bb123996.aspx in Exchange Server 2007 documentation.

Public Folders have their own high-availability mechanism built-in, and it's been around for a long time. It's Public Folder replication. Clustered Mailbox Servers (using Cluster Continuous Replication) are not good candidates for replication.

Cluster Continuous Replication and Public Folder Databases

CCR and public folder replication are two very different forms of replication built into Exchange. Due to interoperability limitations between continuous replication and public folder replication, if more than one Mailbox server in the Exchange organization has a public folder database, public folder replication is enabled and public folder databases should not be hosted in CCR environments.

The following are the recommended configurations for using public folder databases and CCR in your Exchange organization:

• If you have a single Mailbox server in your Exchange organization and that Mailbox server is a clustered mailbox server in a CCR environment, the Mailbox server can host a public folder database. In this configuration, there is a single public folder database in the Exchange organization. Thus, public folder replication is disabled. In this scenario, public folder database redundancy is achieved using CCR; CCR maintains two copies of your public folder database.

• If you have multiple Mailbox servers you can host a public folder database in a CCR environment provided that there is only one public folder database in the entire Exchange organization. In this scenario, public folder database redundancy is also achieved by using CCR. In this configuration, there is a single public folder database in the Exchange organization. Thus, public folder replication is disabled.

• If you are migrating public folder data into a CCR environment, you can use public folder replication to move the contents of a public folder database from a stand-alone Mailbox server or a clustered mailbox server in an SCC to a clustered mailbox server in a CCR environment. After you create the public folder database in a CCR environment, the additional public folder databases should only be present until your public folder data has fully replicated to the CCR environment. When replication has completed successfully, all public folder databases outside of the CCR environment should be removed, and you should not host any other public folder databases in the Exchange organization.

• If you are migrating public folder data out of a CCR environment, you can use public folder replication to move the contents of a public folder database from a clustered mailbox server in a CCR environment to a stand-alone Mailbox server or a clustered mailbox server in an SCC. After you create the additional public folder database outside of the CCR environment, the public folder database in the CCR environment should only be present until your public folder data has fully replicated to the additional public folder databases. When replication has completed successfully, all public folder databases inside of all CCR environments should be removed and all subsequent public folder databases should not be hosted in storage groups that are enabled for continuous replication.

During any period where more than one public folder database exists in the Exchange organization and one or more public folder databases are hosted in a CCR environment (such as the migration scenarios described previously), consider the differences in behavior for scheduled (Lossless) and unscheduled (lossy) outages:

• If a successful scheduled Lossless outage occurs, the public folder database will come online and public folder replication should continue as expected.

• If an unscheduled outage occurs, the public folder database will not come online until the original server is available and all logs for the storage group hosting the public folder database are available. If any data is lost as a result of the outage, CCR will not allow the public folder database to come online when public folder replication is enabled. In this event, the original node must be brought online to ensure no data loss, or the public folder database must be re-created on the clustered mailbox server in the CCR environment and its content must be recovered using public folder replication from public folder databases that are outside the CCR environment.

So, the moral of the story. If you have more than one public folder database in your organization, do NOT put it on a CCR cluster.

----------------------------------------------------------------------------------------

Exchange Server 2007 SP2 and VSS Backups support in Windows Server 2008

Exchange Server 2007 SP2 includes a VSS plug-in for Windows Server Backup to support Exchange backups. Once SP2 is installed, you can use Windows Server Backup to back up and restore your Exchange 2007 SP2 databases.

The new plug-in is delivered in the form of a single executable called WSBExchange.exe. This plug-in is automatically installed by SP2 on all Exchange 2007 Mailbox servers. The plug-in enables Windows Server Backup to be able to make Exchange-aware VSS backups as described below:

• Backups are VSS-based only. You cannot perform streaming ESE backups using Windows Server Backup with or without the plug-in.

• Backups taken with Windows Server Backup occur at volume level. To back up a storage group and database, you must back up the entire volume containing the storage group and database. You cannot back up any data without backing up the entire volume containing the data.

• The backup must be run locally on the server being backed up, and you cannot use the plug-in to take remote VSS backups. There is no remote administration of Windows Server Backup or the plug-in. You can, however, use Remote Desktop or Terminal Services to remotely manage Windows Server Backup and your backup jobs.

• The backup can be created on a local drive, or on a remote network share.

• Only Full backups can be taken. Log truncation will occur only after a successful completion of a full backup of a volume containing an Exchange storage group and database.

• The plug-in does not support the Exchange Replication VSS Writer; as a result, you cannot perform backups of passive copies of databases in a continuous replication environment.

• When restoring data, it is possible to restore only Exchange data. This data can be restored to its original location, or to an alternate location. If you restore the data to its original location, Windows Server Backup and the plug-in will automatically handle the recovery process, including dismounting any existing databases and replaying logs into the recovered database.

• The restore process does not directly support the Recovery Storage Group (RSG). However, if you restore the data to an alternate location, then you can manually move the restored data from the alternate location into an RSG, if needed.

• When restoring Exchange data, all backed up storage groups must be restored together. You cannot restore a single storage group or database.

----------------------------------------------------------------------------------------

Service 'MSExchangeTransport' failed to reach status 'Running' on this server.

Issue:

While you install Exchange Server 2007 SP1 or Exchange Server 2010 Beta on Windows 2008, you might get error while installing the Hub Transport role:

Hub Transport Role Failed
Error:The execution of: "$error.Clear(); if ($RoleStartTransportService) { start-SetupService -ServiceName MSExchangeTransport }", generated the following error: "Service 'MSExchangeTransport' failed to reach status 'Running' on this server.".
Service 'MSExchangeTransport' failed to reach status 'Running' on this server.

or

Hub Transport Role Failed
Error:The execution of: "$error.Clear(); install-ExsetdataAtom -AtomName SMTP -DomainController $RoleDomainController", generated the following error: "An error occurred with error code '2147504141' and message 'The property cannot be found in the cache.'.".
An error occurred with error code '2147504141' and message 'The property cannot be found in the cache.'.

Resolution:

Aparently, this normally happens when you disable IPv6 in Local Area Connection.
This is due to the behavior of IPv6 in Windows Server 2008, when you disable IPv6 in Local Area Connection on Windows 2008 server.

Either enable IPv6. Or else, if you wish to disable IPv6, you have to make note that IPv6 is not disabled on tunnel interfaces if you disable IPv6 in Local Area Connection. By default, IPv6 is enabled in Windows Server 2008. If you want to disable IPv6 for any reasons, you have to use Registry Editor to completely disable IPv6.
To completely disable IPv6 on a Windows Server 2008-based computer yourself, follow these steps:
Open Registry Editor.
Locate the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters
In the details pane, click New, and then click DWORD (32-bit) Value.
Type DisabledComponents, and then press ENTER.
Double-click DisabledComponents, and then type ffffffff in Hexadecimal or 4294967295 in Decimal.

Note: If the setup still fails, remove IPv6 entry from the hosts file available at %systemroot%\system32\drivers\etc

----------------------------------------------------------------------------------------

Exchange Server 2007 Hub Transport and Client Access Server on the Same NLB Cluster

In order to keep the number of servers down in a high availability environment, administrators have been looking at using Network Load Balancing (NLB) for CAS and then co-locating the HT role on each node of the NLB cluster to also provide high availability for the HT role.

This configuration can work, and it really is not too difficult to configure. It is extremely important to note that using NLB to load balance the default SMTP receive connectors (using port 25) is not supported and is completely unnecessary since they are load balanced for all intra-Exchange communications like HT to HT communications. However, using NLB to provide redundancy and load balancing for connections to HTs that are hosting Client SMTP receive connectors (using port 587) is fully supported and may be desireable if you have a large number of external SMTP/POP and SMTP/IMAP clients that need to connect to this receive connector.

The steps that you need are to:

1. Setup two servers running Windows Server 2003 with two NICs in each server
2. Install Exchange Server2007 Hub Transport and Client Access Service (CAS) on each server
3. Configure one NIC for the Network Load Balance cluster and setup the other NIC in a separate network so it can be managed through that IP address
4. Configure NLB with Unicast and even load balancing
5. Setup the port rules:
• Port 25 to 25 for both TCP and UDP and select the radio button to disable this port range (this will exclude port 25 from being listed to using the virtual IP address of the NLB cluster, but still allow the individual server IPs to still listen to port 25)
• Port 465 to 465 for both TCP and UDP and selected the radio button to disable this port range
• Port 80 to 80 for both TCP and UDP and set affinity to none (I recommend "none" so you can easily test and verify that it works)
• Port 587 to 587 for both TCP and UDP, affinity none (this is for the client SMTP receive connector)
• Port 443 to 443 for both TCP and UDP, affinity none
• Port 110 to 110 for both TCP and UDP, affinity none
• Port 993 to 993 for both TCP and UDP, affinity none
• Port 143 to 143 for both TCP and UDP, affinity none
• Port 995 to 995 for both TCP and UDP, affinity none
6. With affinity set to none, you can more readily test the CAS (after updating the web pages to show which server is actually responding) and verify that the load is being shared. You can also test to make sure the NLB cluster does not respond to SMTP on port 25, which it shouldn't if you set it right, and verify that each server does respond to SMTP as an individual server name.
7. You can configure protocol logging for the other protocols and telnet to the ports using the NLB IP address to see if they are loading balancing like they should. You can also use the NLB IP for the testing by sending and receiving messages and checking the message tracking logs to see that the traffic was being balanced. It all worked.

NOTE: You may want to change affinity to either single (especially if it is being used internally) or Class C (especially if it is accessible from the Internet) once your testing is done.

Good luck, and have lots of fun!

----------------------------------------------------------------------------------------

Failover cluster features supported by Exchange 2007 SP1

Failover cluster features supported by Exchange 2007 SP1 are depicted in the table below:

----------------------------------------------------------------------------------------

What is Local Continuous Replication in Exchange 2007

Local Continuous Replication

In previous version Exchange 2000/2003 single server database, let say database suddenly crashed or disk failed to read\write database. So here my database is no more in use due to some disaster and my solution would bring back my database by mode of replacement of disk and then restoring the database by mode of RSG. What will be my server availability and time consumption to restore my database it is comparatively high and SLA will effect and business ofcourse.

But in Exchange 2007 we have concept of LCR where you can mount your database as soon as possible with less time consumption. LCR is nothing but it will keep a copy of the database and log files in another location which is called as “Passive Storage Group” from the Active Storage Group”. For copying the database and log file it uses shipping technology which is nothing replicating the data from source to destination.

Background Process of LCR

Whenever you enable the LCR function, the database are first pass through seeding process, seeding is nothing but a process that creating starting point to copy the database file, then the database is copied to destination folder and then finally logs are shipped into it. Then it will write that information to the database. When you enable the LCR you will find this event id in the application log 2014, 2015 & 2016.

Prior with the release of Exchange 2007 Beta Version we need to do seeding manually for seeding we need manually enter this following command from the powershell

suspend storagegroupcopy -identity "first storage group' (temporarily stop LCR )

update-storagegroupcopy -identity "first storage group" (seeding done here)

resume-storagegroupcopy -identity "first storage group" (resume the LCR)

With the release of Beta2 by default seeding is done at the time of enabling LCR.

Monitoring the LCR

Msexchange replication under this

copygenerationnumber & replay queue length

copygenerationnumber : The number of log file which is currently replicated

Replayqueuelength : the number of log file which is still need to play and waiting in the queue

How you can recover the database through LCR Mode

Let say I have one database called as Mailbox Store stored under First Storage Group.

Open => EMC => Server Configuration => Mailboxes => First Storage Group => Enable LCR and then follow the wizard for selecting the folder.

Now the database is seeded then log files are shipped to target folder.

One fine day let say mailbox database.edb got corrupted.

Run this command Restore-Storagegroupcopy -identity "Third Storage Group" -ReplaceLocations:$true

The above command will check the path of LCR like database, logs and check point file it will redirect the path location in EMC to LCR folder.

Then you can mount the database (mount-database –identity “Third mailbox store”)

Once you mount the database then you need to re-enable the LCR because it will stop the LCR function but before that you need to clean up all the files if you are using any old LCR folder or create a new folder instead

Disadvantages of LCR

Even though it is very good pre-backup solution for database but still there is some disadvantages

1) We can't expect that the all the data can be potentially recovered because the log files will not be in synch as they are shipped.

2) If our active storage group fails then we need to manually switch to passive storage group but it is inexpensive solution.

3) It will be used for the storage group which has only one database per storage group

----------------------------------------------------------------------------------------

A few tips for Exchange Server 2007 SCC & CCR clustered Installations

Have you tried to install Exchange Server 2007 yet? Clustering is different. First of all Exchange Virtual Servers (EVS) are gone, replaced with Clustered Mailbox Server (CMS). You now have two options for clustering, Single Copy Cluster (meaning one CMS per server really) and it is not the default installation. The default is now Continuous Cluster Replication (CCR) which is something new to Exchange Server 2007,

You still install and configure clustering first, then install Exchange. But passive and active nodes are supposed to be handled differently now. And active/active clustering is simply not allowed anymore I would use setup.exe or the GUI to install both nodes as passive. This will put the Exchange bits on machine, but won't create the CMS just yet. From the command prompt you would use something like this syntax:

setup.com /mode:install /roles:mb

Which would should come back with:

Welcome to Microsoft Exchange Server 2007 Unattended Setup Preparing Exchange Setup
Copying Setup Files ......................... COMPLETED

The following server roles will be installed:

Management Tools
Mailbox Role
Performing Microsoft Exchange Server Prerequisite Check
Mailbox Role Checks ......................... COMPLETED
Configuring Microsoft Exchange Server
Copying Exchange files ......................... COMPLETED
Mailbox Server Role ......................... COMPLETED

The Microsoft Exchange Server setup operation completed successfully.
Do this for both nodes. Then on the Active node you will need to run exsetup.exe. Why? Because of this little gem in the books online:

If you already have one or more server roles installed on a computer, you cannot use the Exchange Server 2007 Setup wizard or the Setup.exe command to add or remove server roles. Instead, you must use the ExSetup.exe command.

DOH! Not knowing this means you can try Setup.exe until you are blue in the face and never get Exchange clustered.

So the ExSetup.exe to create the CMS is (replace CMSNAME with a real meanful name, replace cip with a real IP):

exsetup /mode:install /clustered /cn:CMSNAME /cip:198.168.1.100

Which would should come back with:

Welcome to Microsoft Exchange Server 2007 Unattended Setup
No server roles will be installed

Clustered Mailbox Server

Performing Microsoft Exchange Server Prerequisite Check
Configuring Microsoft Exchange Server
Clustered Mailbox Server ......................... COMPLETED
The Microsoft Exchange Server setup operation completed successfully.
This should create all the clustered resource Exchange needs.

Why not install the bits and create all the clustered resources together with the Setup.exe command? The syntax would look like this:

setup.com /mode:install /roles:mb /newcms /cn:CMSNAME /cip:198.168.1.100

If you Active Directory is large it could take a little bit for the CMS to be registered properly, in that case you might get an error this like:

Welcome to Microsoft Exchange Server 2007 Unattended Setup

No server roles will be installed

Clustered Mailbox Server Performing Microsoft Exchange Server Prerequisite Check
Clustered Mailbox Role Checks ......................... COMPLETED
Configuring Microsoft Exchange Server
Clustered Mailbox Server ......................... FAILED

The computer account 'CMSNAME' was created on the domain controller '\\dc02.clusterhelp.ad', but has not replicated to the desired domain controller (dc01.clusterhelp.ad) after waiting approxmately 60 seconds. Please wait for the account to replicate and re-run exsetup /newcms.
The Exchange Server setup operation did not complete. Visit http://support.microsoft.com and enter the Error ID to find more information.
So, you only get 60 seconds for AD to fully replicate - DOH! Interesting. See AD was using our DNS (dc02) on the Public, but the Exchange was on the private near dc01. The spelling error is Microsoft's by the way, not mine. You can add /dc:dc02.clusterhelp.ad but setup will fail if the DNS is not on the same AD site as the Exchange server. You will see this error:

Setup cannot use domain controller 'dc02.clusterhelp.ad' because it belongs to Active Directory site 'Public'. Setup must use a domain controller in the same site as this computer (Private).

Other DNS messages you might get:

Exchange setup cannot continue because DNS information for the clustered mailbox server "CMSNAME" has not finished replicating. Please run setup again after replication has completed. After replication has completed, the command "nslookup CMSNAME" should succeed.

This one again means you need to let it bake some more, let DNS replicate. Check your event logs. In some environments, you might have to wait 30 to 45 minutes for things to settle down. Just rerun the exsetup command again.

Lastly you might get this error:

Error:
Error of unknown type occured while performing exsetdata operation; the original error code was 0xc103fd2c

This means might have a duplicate DNS record that needs to be removed before CMSNAME be created. Check your event logs for the exact error and believe them! The spelling error is again Microsoft's not mine. If Exchange can't create the record look for duplicates.

If after all that you just want to give up, the command would look like this:

exsetup /mode:uninstall /removecms /cmsname:CMSNAME

So why am I only showing you the command line for ExSetup.exe? Because you can only run the GUI seutp once, after that you have to use ExSetup.exe.

Good luck, check your event logs and hopefully this will help someone else out.

----------------------------------------------------------------------------------------