Lotus Domino Cluster / High Availability

After several crashes of our production Lotus Domino Server dueto some application corruption, we finally decided to upgrade it with latest version and also decided to add Cluster server for fail over support. Clustering is an excellent topology which really works well specifically in domino environment, and is also used for high availability / load balancing high load. Following is an small howto reference guide which I created for future reference. Using this guide you can also create cluster server for your domino server. It really helped me in many situation when database corruption occurred at production server or if i have to plan any upgrade and downtime is required. Notes client switch over to cluster is almost seamless and most of the time, client even don’t notice it that he is on cluster server 🙂 ~

Some Best Practices for clustering that I have learned so far:

[Some points in below paragraph have been taken from Lotusphere 2011 – How to build a better cluster – By Gabriella Davis – Paul Mooney]

– Clustered Servers should not use the same disk storage array for example SAN/NAS, Its because if somehow there is some problem or failure with the SAN/NAS, secondary should continue its services using its local storage or alternate storage.

– Clustered Servers should not use the same power source, Its because if somehow the power source of primary server fails, secondary server services should be available via alternate power source.

– They should not be in the same building and never in the same room, Its because if there is some physical disturbance like Fire, Earthquake or others, secondary server should keep continue serving its mail service.

– Should have enough CPU power and memory. „It’s common that clustered servers need more power and more disk resources than un clustered servers.

– Few minutes or an hour spent on testing could save you countless hours in deployment

– Ignoring errors or alerts that appear during replica creation or cluster setup will come back to haunt you later

– Always use latest Version of Domino/Notes clients possible with latest available Feature Pack (FP) , Currently 8.5.3 with FP3 is good. Also failover in notes client is good, fast and almost transparent to users.

Ok lets start  . . .

I assume that you already have working domino server in place. first thing first, create server ID that will be use when secondary domino server installation is initiated at cluster server.

Current Scenario:

Primary Mail Server: D1
Secondary Mail Server: D2
Domain Name: syed

Create Server ID for Secondary Server

–  Open Domino Administrator Client,
–  Goto Configuration ,
–  On right side panel, click on Registration > Server ,
–  In Server Name, type your Secondary Server name,
–  In Domain name, type your domain name
–  In Location for storing server ID, click on in file, and select your destination folder. (Save it any folder that you can access later from secondary server to copy this ID file)
–  Click on Green Icon of tick, and then click on REGISTER

As showed in the images below . . .

Now move to your secondary cluster server,

SECONDARY CLUSTER SERVER SETUP:

At your Secondary Server, Initiate Domino Server Setup, Click NEXT to Continue . . .

Click on Setup an additional server

Click on Browse and select the D2 server id that we created earlier at production server. (You can copy the id file from the production server to this cluster server)

.

Type in your Production server name , so this secondary server will know from where to pull its replication data.

After clicking on Setup it will start configuring its initial setup.

After it finishes, double click on the DOMINO SERVER icon on your desktop to launch the server.

After initiating, it will start replicating names.nsf

Now its time to add this secondary server into cluster group at production server configuration.

Add Secondary Server into Cluster Group at Primary Server

– Open Domino Administrator Client,
– Goto Configuration ,
– Goto All Server Document, Here you will see both server names, Primary and Secondary
– Select both servers and click on Add to cluster

As showed in the images below . . .

A popup will ask you if you want to continue, Click on YES
It will ask you for cluster name with option of “Create new cluster“, click OK to continue

As showed in the images below . . .

Now enter your cluster name and click on OK OK
It will ask you if you want to create it immediately, click YES

As showed in the images below . . .

Wait few minutes so that it creates cluster.
Now at your Primary Server domino Console. Issue following command to start replication of Primary Domino Databases to Secondary Server. (not mail boxes)

replicate D2

As showed in the image below . . .

After its replication finishes, its time to create replicas of your mail files to secondary server. For this purpose you have to first assign necessary permissions at secondary server.

Assigning Permissions for Create Replica

– At your Primary server,
–  Open Domino Administrator Client,
–  Goto Configuration ,
–  Goto All Server Document, Here you will see both server names, Primary and Secondary
–  Double click on Secondary Server.

As showed in the image below . . .

Goto Security,
In Server Access – Who Can section, Add your Admin ID and Primary Server name in following Sections.
  – Create database & templates
  – Create new replicas

As showed in the image below . . .

Click on Save & Close

Create Replicas of Users Mail Files

To create mail replica copies of user mail files from primary to secondary, First you have to create mail folder in secondary server, because when domino first install, it donot create mail folder.

– At your Primary server,
– Open Domino Administrator Client,
– Goto File / Open Server , Select your secondary server, if its not appearing in the list, then click on Other and it will show you both servers, click on secondary server. You can also type in your secondary server name with its domain, for example D2/syed .Now you will be connected to your secondary server.
– Goto Files , Right click on Mail Parent folder of Domino, and create New Folder name mail (In my case I installed domino at D:\Lotus\domino\data

After the folder is created, Switch back to your primary server.
– Goto Files section,
– Select your desired mail file, (For test select single mail file, later you can select all or multiple mail files at a time to replicate), On right side panel, Click on Databases > Create Replica(s)

As showed in the image below . . .

A new window will popup with some options, First click on “Show me only cluster members” so that your cluster servers should appear in the list. Now select secondary server, and click on ADD,
Then select all three options of
– Copy Access Control List
– Create full text indexing for searching
– Exchange unread marks on replication.

As showed in the image below . . .

Now click Ok to continue.

Depends on the mail file size , It will few minutes or longer to complete the task. After it creates replica, the user mail file will appear on secondary server under mail folder.

TIP: You can force immeidate replication by using following command at console on both servers.

tell adminp p all

and monitor the console for information or errors.

–

. . .  TESTING  . . .

Before going further, some important tips and information on Clients and Clustering

[Some points in below paragraph have been taken from Lotusphere 2011 – How to build a better cluster – By Gabriella Davis – Paul Mooney]

The Notes client is “Cluster Aware”

● When it communicates with a Domino server that is in a cluster, that server updates a file on the client with its cluster name and its cluster mates
─ this makes the client aware that the server it wants is in a cluster and means even if the server is down, the client has built in information about where to go next
─ if you add a domino server to a cluster and rapidly shut down the original server, the notes clients will never get a chance to “learn” about the cluster
● If a primary server fails, the Notes client will search the other cluster mates it knows about for a replica of the database it wants

─ Searches cluster mates alphabetically,  not by availability
– this is very important in your planning
─ Opens database on the first available cluster mate it finds

Cluster.NCF

● The cluster.ncf is a file in the data directory on the client
● It contains details of all clustered servers you have connected to from that client, along with their cluster mates
– there can be several clusters in your cluster.ncf
● If you delete the cluster.ncf the Notes client will forget all about any clusters it was aware of, but will relearn the first time you reconnect to a clustered server

Notes 8.5.2 – Improved Client Clustering/Repilcas

● Client Transparency
● In the 8.5.2 or above client, the ability to failover to a cluster mate without the user being notified or stopped in their work has been improved
● During the following activities, if the primary server goes down, the user will be invisibly transferred to one of the cluster mates
─ Creating a new entry for PIM views (Mail message, Calendar entry, or To Do)
─ Opening a document in an application
─ Opening a different folder or view
─ Replying or forwarding with attachment
─ Creating a new mail message while multiple documents are selected
─ Deleting a single or multiple document
─ Sending a message
─ Switching back to Primary server when it becomes available

Ok Enough theory, its time to hit the road.

Configure your Lotus Notes Client and send an test message to any user or yourself.
You will see in logs that it immediately replicate/duplicate mail from your primary server to secondary server.

As showed in the image below . . .

You can also verify it by directly open user mail file from secondary server using admin client 😀 🙂

Testing from the client  side . . .

To test fail over switch at client end, temporarily disable Primary server LAN ,
At client end Open Notes client , when it will fail to find primary server , it will auto switch over to secondary server.
Also Try to send an test message to yourself for test purpose while the primary server is down for test purpose.

As showed in the image below . . . (client have switched over to D2 i.e our secondary clustered server)

At secondary server console , you will see notification of user failing over from primary to secondary.

As showed in the image below . . .

.

.

Now enable LAN on Primary server that we temporarily disabled for test purposes.
After few minutes, Secondary server will auto replicate data to primary server , so that primary server can update itself with the changes that occurred while it was offline.

As showed in the image below . . . (Replication in progress from D2 secondary server to D1 primary server)

& at your client end, you will be switch back to primary server where you can see your email that you sent from the secondary server while the primary was down.

.

.

.

.

Triggering Failover for CLUSTER

[Some points in below paragraph have been taken from Lotusphere 2011 – How to build a better cluster – By Gabriella Davis – Paul Mooney]

You can force all users to fail to another cluster server
─ Useful for Disaster Recovery testing
─ Useful for server maintenance work
─ Useful for migrations

● On the server console type the following command

Set Config Server_Restricted=
– 1 Prevent any new sessions opened on this server until restart

● Administrators bypass this restriction if using the Domino Administrator client

– 2 Prevent any new sessions opened on this server until setting changed

● Administrators bypass this restriction if using the Domino Administrator client

● Setting is persistent even through server reboots

– 0 Feature disabled

Also issue drop all command to force disconnection of all connected clients. („By the way, iMap and POP3 users will still have access to there Mail).

You can also set the maximum number of concurrent NRPC users allowed to connect to a server in NOTES.INI using following variable

Server_MaxUsers

„Set variable to a number determined in planning stage

Set variable using console command Or use NOTES.INI tab in server configuration document

Set config Server_MaxUsers = desired maximum number of active concurrent users, „Additional users will fail over to the other members of  the cluster

.

.

Creating Replication Document for backup purposes

Its a good idea to always have a connection document replicating between server cluster members because it replicates stubs too. Also if something got missed from routine replication, this forced replication will make sure everything is in synch.

.

Centralized Management is GOOD 🙂 by having single Configuration Document for all servers

Its a good idea to centralize manage all servers in cluster group.

Open Admin Client, Goto Configuration tab, here you will see two configuration docs for primary and secondary.
Double click on primary server configuration doc to take its properties,
As showed in the image below . . .

.

On Basic tab scree, select check on “Use these settings as the default settings for all servers:”
Then click on Save & Close.
As showed in the image below . . .

.

.

Now you will see * [All  Servers] doc
As showed in the image below . . .

Now rename the second doc with the secondary server name with any other name so secondary server will not take data from it, all configuration settings should be taken from centralized document * [All  Servers]

As showed in the image below . . .

WAN Connectivity Tips for Primary & Secondary Server

There are several scenarios that wan connectivity can be maintained. For example A live can be configured on Mail server WAN interface , but I will not recommend to put mail server on danger by directly attaching it to the internet, better to configure live ip on any of your router/firewall and then configure port forwarding of smtp/pop3 traffic to your server.
In this scenario If you have only single live IP , then you have to manually change the IP of primary server to secondary in case your primary goes down and you want to receive mails on your secondary server. Lot of discussion can be made here, but no time for it at a moment 🙂

Scenario # 1

You have public ip pool of /30. For example 1.1.1.1 and 2.2.2.2. Both ips are configured on Router and router do port forwarding for mail traffic. First configure 1.1.1.1 to forward mail traffic to primary server and 2.2.2.2 to forward mail traffic to secondary server. Now Request your web server hosting company to set secondary MX record to 2.2.2.2 (It is required so when no response is received from primary server smtp/pop3 , then it should try to send traffic to secondary MX which is 2.2.2.2

Now to test , quit primary server, wait few minutes, and then send an test email from the internet to your domain, it should arrive in your secondary server, and when the primary will be available , it will replicate its data with one another.

MORE TO COME . . .

Regard's
SYED JAHANZAIB