Recovery of DMA RM admin account / Howto View MYSQL encrypted password

Encrypted Password Recovery / Syed Jahanzaib

Last Updated: 8th August, 2015 ,

Note:
1) it is recommended to create bash script to detect invalid password login attempts , if a user tries to login more then X times in a minute, then it should be considered as HACKING Attempt and this account should be locked to further prevent any bruteforce attempt. 1) Donot use default ADMIN account.

---

All user id and passwords are stored in MYSQL database name radius . Manager id’s are stored in rm_manager table and all other normal user id’s used for user login are stored in rm_users table.

---

Method 1# How to add additional admin account in RM

A workaround is to add another manager with admin privileges . ONce its added, login with this new manager ID, and change the ADMIN account password from the Manager list.

Login to mysql, and use following commands

mysql -uroot -pYOUR_MYSQL_PASS

use radius;

INSERT INTO `radius`.`rm_managers` (`managername`, `password`, `firstname`, `lastname`, `phone`, `mobile`, `address`, `city`, `zip`, `country`, `state`, `comment`, `company`, `vatid`, `email`, `balance`, `perm_listusers`, `perm_createusers`, `perm_editusers`, `perm_edituserspriv`, `perm_deleteusers`, `perm_listmanagers`, `perm_createmanagers`, `perm_editmanagers`, `perm_deletemanagers`, `perm_listservices`, `perm_createservices`, `perm_editservices`, `perm_deleteservices`, `perm_listonlineusers`, `perm_listinvoices`, `perm_trafficreport`, `perm_addcredits`, `perm_negbalance`, `perm_listallinvoices`, `perm_showinvtotals`, `perm_logout`, `perm_cardsys`, `perm_editinvoice`, `perm_allusers`, `perm_allowdiscount`, `perm_enwriteoff`, `perm_accessap`, `perm_cts`, `enablemanager`, `lang`) VALUES ('adminx', 'adminx', 'adminx', 'adminx', '', '', '', '', '', '', '', '', '', '', 'aacable@hotmail.com', '1000.00', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', 'English');

UPDATE rm_managers SET password = MD5('12345') WHERE managername = 'adminx'

Done. Now login with following |ID and goto Managers and change your old ADMIN account password

id = adminx
pass = adminx

---

Method 2# Other methods to view old password (only if its simple form of password)

Passwords are stored in encrypted format using SHA1 algorithm.
I used the following method to retrieve the old password (without changing it)

Login to your Linux box using root account and execute following commands

mysql -h localhost -u root -s -pYOURPASSWORD
use radius;
SELECT * FROM `rm_managers`;

It will show you some scattered information of all the admin accounts with there details and Encrypted passwords.

TIP: You can also use PHPMYADMIN to get info via nice GUI 🙂 , but as I am a creature living in the dark, therefore I like to use black screen to perform my functions 😉

As showed in the image below . .

As you can see in above image, First column in Yellow marking are Manager Id’s stored in the DB radius. and second column marked in RED are passwords stored in encrypted format. Select & copy the encrypted password. Now goto http://crackstation.net/ (or there are other websites too that can encode hash encrypted passwords) and paste your password here and click crack hashes. and you will see your password in plain text in result window.    :)~

As showed in the image below . . .

---

How-to view Radius Manager USER’s account password

Login to your Linux box using root account and execute following commands.

mysql -h localhost -u root -s -pyour_password
use radius;
select * from radcheck order by UserName;

It will show you all users Ids’s along with passwords in clear text format.

If you want to view only specific data, use the following script.

---

OR use the SCRIPT to view all users password in clear text format

First create script and assign it execute rights.

touch /etc/rmuserlist.sh
chmod+x /etc/rmuserlist.sh

Now edit rmuserlist.sh
nano /etc/rmuserlist.sh

and paste the following data

#!/bin/bash
# Script Source: http://wiki.mikrotik.com/wiki/Bash_scripts_for_Linux/Mysql/Freeradius/PPPoE
# Syed Jahanzaib / aacable@hotmail.com
# https://aacable.wordpress.com

LUSERNAME="$1"

if [ -z "$LUSERNAME" ]; then
NAME=unspecified
fi

case $NAME in
unspecified)
MYCMD="mysql -h localhost -u root -s -pYOURPASSWORD -t -e "
$MYCMD "use radius; select * from radcheck order by UserName;"
$MYCMD "use radius; select * from radreply order by UserName;"
;;
*)
MYCMD="mysql -h localhost -u root -s -pYOURPASSWORD -e "
$MYCMD "use radius; select * from radcheck order by UserName;" |grep $LUSERNAME
$MYCMD "use radius; select * from radreply order by UserName;" |grep $LUSERNAME
;;
esac

Source: http://wiki.mikrotik.com/wiki/Bash_scripts_for_Linux/Mysql/Freeradius/PPPoENote: Make sure to change the password in above script.Save & EXIT.Now to view user list, simply type

/etc/rmuserlist.sh

it will show you all user list.To view particular user password, simply type its name like

/etc/rmuserlist.sh testing

 

---

Some Useful commands to reset admin / manager password.

Change OLD Admin Password (may not work)

UPDATE rm_managers SET password = MD5('12345') WHERE managername = 'admin';

 

---

View Specific Manager Users list with passwords.

To get User Details for specific Manager & store in a file called manager_users.txt

mysql -sN -u root '-pView*pak' -e 'use radius; select username from rm_users where owner = "MANAGER_NAME_HERE" order by UserName;' > /tmp/manager_users.txt

Now create a bash script

#!/bin/bash
# Syed Jahanzaib / aacable@hotmail.com
# https://aacable.wordpress.com
#set -x
SQLPASS="YOUR MYSQL SQL PASS HERE"
TMP="/tmp/manager_users.txt"
num=0
cat $TMP | while read users
do
num=$[$num+1]
USR=`echo $users |awk '{print $1}'`
PAS=`mysql -sN -u root -s -p$SQLPASS -e "use radius; select * from radcheck where username = '$USR';" | grep Cleartext-Password | awk '{print $5}'`
#echo "$PAS" '
echo "$USR / $PAS"
#fi
done

Done. now execute the script and it will show you the password for specific managers only.

---

Change Radius user password in mysql

SET PASSWORD FOR 'conntrack'@'localhost' = PASSWORD('NEWPASSWORD');

---

~zaiB

---

Regard’s
SYED JAHANZAIB

Lotus Domino not recreating log.nsf automatically after deletion

AFTER DELETION OF LOG.NSF , DOMINO DOES NOT RE-CREATED IT AFTER RESTART

My Domino server LOG.NSF got grow in size crossing 16GB which is quite a huge size for any log file. Upon examining I found out that there were some flooding of dictionary base mail sending retry from some internet spammer. Anyhow that’s another story that how i coped with it.

For the LOG.NSF, I simply QUIT the domino server, deleted LOG.NSF from the DATA folder and then I restarted Domino Server. By default, Domino recreates LOG.NSF automatically if it does not found log.nsf in DATA directory, but strangely it didn’t. At console it complained that it was unable to found the LOG.NSF file and then exit. I found out that I accidentally deleted LOG.NTF file too which is actually a template that is used to create LOG.NSF So I copied log.ntf from the backup tape to DATA folder and started domino server again, and Alhamdolillah, IT STARTED OK  🙂

I then used space-saver option to delete documents older then 7 days.

Also I read somewhere that You can copy log.ntf file and past this copy in the same folder. Now rename this file to log.nsf. Lotus server will work .

 

Regard’s
SYED JAHANZAIB

Veeam Backup & Replication Deployment for Virtual Servers/Guests for High Availability + Tips

At my office, I have multiple ESXi VM Servers with several guests hosted. I am using VEEAM B&R software for backup & replication. I have created Replicas of all my primary servers to DR site on secondary replica server using Veeam. I have hen scheduled replication to run Daily in night. Whenever any disaster occurs either at server hardware level or guest OS level, I simply fail over to replica using Veeam, and when primary server/guest becomes available, I simply Fail Back to Production using Veeam console and all changes replicate back to production from replica. 🙂 Replication can be scheduled to run continously but not recommened, Try to set 6/12/24 hours schedule instead to update replicas.

I will distribute this article in following sections.

1- One time simple backup of VM guest using Veeam
2- Adding daily backup job
3- Restoring VM from backup to production
4- Adding replication job between primary and D.R site
5- Test Failover to REPLICA & FAILBACK to PRIMARY (PRODUCITON)
6- VEEAM Email Notification Configuration

.

1- ONE TIME SIMPLE BACKUP OF VM GUEST USING VEEAM

If you want to take simple backup of your guest OS, simply follow this.

After fresh installation of VEEAM, first define its Backup Folder where it will save the images.

Adding Backup Repositories to store images

# Open VEEAM console, Goto Infrastructure
# Right click on Backup Repositories and select Add Backup Repository
# New Window will appear asking for Backup Name, rename it to some meaningful name , for example if you have External USB Harddrive attached, name it External USB Storage and click NEXT.
# Now it will ask storage type, Select Microsoft Windows server and click NEXT
# In Server, Select This Server and click NEXT
# Now it will ask for Folder Location where it will save the veeam images, Simply browse it to the folder where you want to save the images, for example H:\images and click NEXT
# on vPowerNFS window click NEXT
# On Review window, click NEXT to continue select FINISH

ADDING your Virtual Server in VEEAM

# Open VEEAM console, Goto Virtual Machines
# Right Click and select Add Server
# A new window will popup, asking for virtual server type, Select appropriate server, for example VMware vSphere, and select NEXT,
# Now it will ask virtual server IP or DNS name, I always prefer to connect via IP instead of DNS. Enter IP address of your VM server and click NEXT
# Now enter your user id password of virtual server and click NEXT.
# It will show you the summary of server, click FINISH

As showed in the images below . . .

Now that you have created Backup Repository and added your first virtual server into veeam, its time to take our first backup.

Open VEEAM console, Goto Virtual Machines, and select your VM Server, on right window, you will see your virtual guest machines.
Either select one machine or all, right click and select VEEAMZIP
Now Select the storage where you want to place veeam backup image, select and click on OK, it will start backup the guest VM to the backup repository folder.

.

.

2-  ADDING  DAILY  BACKUP  JOB

To do automatic daily backup of your important vm guests , for example daily in night at 1:00 AM, You have to add JOB ,
# Open VEEAM console,
# Goto Virtual Machines , Select your desired guests, right click and select Add to Backup Job / New Job and select NEXT
# Enter any meaningful name for this job, for example Daily Backup in Night @ 1:00 AM and select NEXT
# Select your desire Virtual Machine(s) from the list and select NEXT
# Select your backup repository for example External USB HDD and select NEXT
# On Guest Processing, simply select NEXT
# On schedule window, Click on Run the job automatically, Now select Daily at this time and select your desired time , e.g: 1:00 AM you can also select it run on weekly basis, periodically or whatever is best for your environment and select CREATE.
# Click on FINISH.  (Also click on Run the job when I click Finish if you want to run the job immediately)

As showed in the images below . . .

.

.

3-  RESTORING VM from Backup to Production using VEEAM

To restore any VM from VEEAM backup,

# Open VEEAM console, Goto

To be continued . . .

.

.

4–  ADDING REPLICAITON JOB between Primary & DR Site

Replication is another great feature of VEEAM. With replication you can always have an ready to user REPLICA of your selected or all Production Virtual Guests. Replication can be scheduled to run on continuous or scheduled timings. For example I have replication schedule to replicate all the servers on daily at night to secondary DR site. To create Replicas of your production VM guests, You should have

1- Primary VM Server with ESXi or Hyper-V
2- Secondary VM Server with ESXi or Hyper-V (REPLICA SERVER with same ESXI or HYPER-V installation and enough storage space same or above as production server)
3- VEEAM B&R application either Physically installed on any OS like Windows 7 or virtually at DR /PRI Site. I have installed it at DR site.

Let’s start Replicating 😉

# Open VEEAM console, ADD both servers in VEEAM > Virutal Machines section.
# On Primary Server, Right click on your guest machine(s) you want to replicate to REPLICA server and select Add to replication job / New and select NEXT
# Enter any meaningful name for this job, for example Daily Replication in Night @ 1:00 AM and select NEXT
# Select your desire Virtual Machine(s) from the list and select NEXT
# In destination, select your Secondary server (Replica Server) and select NEXT
# In Job Settings window, select your backup repository for example External USB HDD and select NEXT
# On Guest Processing, simply select NEXT
# On schedule window, Click on Run the job automatically, Now select Daily at this time and select your desired time , e.g: 1:00 AM you can also select it run on weekly basis, periodically or whatever is best for your environment and select CREATE.
# Click on FINISH.  (Also click on Run the job when I click Finish if you want to run the job immediately)

As showed in the images below . . .

To view status of replication, Open Veeam console , goto Backup & Replication / Jobs , select Replication. On right window, you will see the job you created in earlier stage. Right click on it and select Statistics and you will see something like below . ..

.

5- Test FAILOVER to REPLICA & FAILBACK to PRIMARY (PRODUCITON)

FAILOVER TEST

# Open VEEAM console,
# Goto Backup & Replication
# Goto Replicas, On Right Side window, you will see your REPLICATED VM Guests,
# Right click on your desired guest you want to failover to, and select FAILOVER NOW
# It will again ask for which machines you want to fail over, select your required guest machine(s) and select Next
# In Restore Reason, type reason why you want to failover (for record purpose , if you want to track later why you or other admin used FAILOVER), Click Next and FINISH. Veeam will auto power ON the guest on DR server,

As showed in the images below . . .

.

For test purpose, Create few folders or files on Replica Guest Server.

As showed in the images below . . .

.

FAILBACK TO PRODUCTION TEST

Now we will switch back to Production Server (Failback to production).

.

# Open VEEAM console,
# Goto Backup & Replication
# Goto Replicas, On Right Side window, you will see your REPLICATED VM Guests with (ACTIVE) caption and Green Play Icon.
# Right click on your desired guest you want to failback to production, and select FAILBACK TO PRODUCTION NOW
# It will again ask for which machines you want to fail BACK, select your required guest machine(s) and select Next
# In Destination , select Failback to the original VM
# In summary, it will show you the Details of the guest, Select Power on VM after restoring and click Finish

As showed in the images below . . .

.

.

After FAILBACK complete, You will see that Guest OS on DR REPLICA server get shutdown and automatically powered ON at Primary Server
As showed in the images below. . .

Now open Your GUEST OS at production server, and you will see the changes here that you made at REPLICA server 🙂

.

.

.

6- VEEAM Email Notification Configuration

If you want to be notified when the veeam complete its backup or failed, Configurie Email Notification in VEEAM and add them in JOB.
See the images below . . .

Now Add it in your JOB.

.

REPORT SAMPLE received on EMAIL

To be continued . . .

.

Regard’s
Syed Jahanzaib

//

Lotus Domino ICM (Internet Cluster Manager for HTTP failover/load balancing)

Recently we configured Lotus Domino clustere in our company.Everything worked fine, Failover to secondary and move back to primary server when available working good, but the HTTP does not works this way. Browsers themselves are not cluster aware, they ask for specific host-names. For example if one of my primary server goes down, then the web mail users wont be able to access there mail from the primary server mail address.

For this purpose I googled a  little bit and found out that Domino ICM (Internet cluster manager) can fulfill our requirement. This makes our web mail servers highly available to clients. You can run the ICM on a Lotus Domino server configured in clusters (not necessarily clustered). The ICM supports the HTTP protocols acting as an High Availability Bridge between HTTP clients and the Domino Web servers in a cluster. ICM also maintain information about the availability of servers and databases in the cluster.  ICM Sends periodic probes to the Web servers in the cluster to determine their availability.

ICM installation and configuration was a piece of cake, in fact there was nothing to configure much 🙂 You should include the ICM configuration information on every Web server in the cluster, not just the server on which you run the ICM.
This is done because each Web server uses its own Server document to determine how to generate URLs that refer to the ICM
Following is a small howto for my future reference.

I assume you already have Primary and secondary server (clustered) configured. Following is example of network

D1 = primary domino server
D2 = secondary domino server
D3 = domino server for ICM for HTTP

From your primary server ,

1- Register a new server ID with the name of D3
2- Install Domino on 3rd server , run its setup, and configure it as Addition server and provide it D3.id that you created in step 1.
3- Add D3 in existing cluster
3- Goto D3 console, and quit http server (use command tell http quit), otherwise it will gonna conflict with the ICM service which also uses the same tcp 80 port. OR you can change ports as per your requirement IF you want to run ICM and HTTP on same server.
4- Load ICM by issuing following command at D3 console.

load icm

Now from your client end, browse to http://D3 and you will be redirected to the first available server i.e http://D1
For test purpose, quit D1 server, wait few seconds,

Now browse again to http://D3 and this time you will be redirected to http://D2 instead of D1 because ICM maintain the list of available server in the list and will redirect request to first available server.

When an HTTP client is connected to a server that fails, the client receives a typical browser message stating that the server is not responding. To fail over to a different replica, the user must contact the ICM again by entering the D3 address in URL again.

Some information provided in this article have been taken from multiple external resources. For example
Understanding IBM Lotus Domino server clustering By Reetu Sharma /Ranjit Rai

Regard’s
Syed Jahanzaib

Lotus Domino Cluster / High Availability

After several crashes of our production Lotus Domino Server dueto some application corruption, we finally decided to upgrade it with latest version and also decided to add Cluster server for fail over support. Clustering is an excellent topology which really works well specifically in domino environment, and is also used for high availability / load balancing high load. Following is an small howto reference guide which I created for future reference. Using this guide you can also create cluster server for your domino server. It really helped me in many situation when database corruption occurred at production server or if i have to plan any upgrade and downtime is required. Notes client switch over to cluster is almost seamless and most of the time, client even don’t notice it that he is on cluster server 🙂 ~

Some Best Practices for clustering that I have learned so far:

[Some points in below paragraph have been taken from Lotusphere 2011 – How to build a better cluster – By Gabriella Davis – Paul Mooney]

– Clustered Servers should not use the same disk storage array for example SAN/NAS, Its because if somehow there is some problem or failure with the SAN/NAS, secondary should continue its services using its local storage or alternate storage.

– Clustered Servers should not use the same power source, Its because if somehow the power source of primary server fails, secondary server services should be available via alternate power source.

– They should not be in the same building and never in the same room, Its because if there is some physical disturbance like Fire, Earthquake or others, secondary server should keep continue serving its mail service.

– Should have enough CPU power and memory. „It’s common that clustered servers need more power and more disk resources than un clustered servers.

– Few minutes or an hour spent on testing could save you countless hours in deployment

– Ignoring errors or alerts that appear during replica creation or cluster setup will come back to haunt you later

– Always use latest Version of Domino/Notes clients possible with latest available Feature Pack (FP) , Currently 8.5.3 with FP3 is good. Also failover in notes client is good, fast and almost transparent to users.

Ok lets start  . . .

I assume that you already have working domino server in place. first thing first, create server ID that will be use when secondary domino server installation is initiated at cluster server.

Current Scenario:

Primary Mail Server: D1
Secondary Mail Server: D2
Domain Name: syed

Create Server ID for Secondary Server

–  Open Domino Administrator Client,
–  Goto Configuration ,
–  On right side panel, click on Registration > Server ,
–  In Server Name, type your Secondary Server name,
–  In Domain name, type your domain name
–  In Location for storing server ID, click on in file, and select your destination folder. (Save it any folder that you can access later from secondary server to copy this ID file)
–  Click on Green Icon of tick, and then click on REGISTER

As showed in the images below . . .

Now move to your secondary cluster server,

SECONDARY CLUSTER SERVER SETUP:

At your Secondary Server, Initiate Domino Server Setup, Click NEXT to Continue . . .

Click on Setup an additional server

Click on Browse and select the D2 server id that we created earlier at production server. (You can copy the id file from the production server to this cluster server)

.

Type in your Production server name , so this secondary server will know from where to pull its replication data.

After clicking on Setup it will start configuring its initial setup.

After it finishes, double click on the DOMINO SERVER icon on your desktop to launch the server.

After initiating, it will start replicating names.nsf

Now its time to add this secondary server into cluster group at production server configuration.

Add Secondary Server into Cluster Group at Primary Server

– Open Domino Administrator Client,
– Goto Configuration ,
– Goto All Server Document, Here you will see both server names, Primary and Secondary
– Select both servers and click on Add to cluster

As showed in the images below . . .

A popup will ask you if you want to continue, Click on YES
It will ask you for cluster name with option of “Create new cluster“, click OK to continue

As showed in the images below . . .

Now enter your cluster name and click on OK OK
It will ask you if you want to create it immediately, click YES

As showed in the images below . . .

Wait few minutes so that it creates cluster.
Now at your Primary Server domino Console. Issue following command to start replication of Primary Domino Databases to Secondary Server. (not mail boxes)

replicate D2

As showed in the image below . . .

After its replication finishes, its time to create replicas of your mail files to secondary server. For this purpose you have to first assign necessary permissions at secondary server.

Assigning Permissions for Create Replica

– At your Primary server,
–  Open Domino Administrator Client,
–  Goto Configuration ,
–  Goto All Server Document, Here you will see both server names, Primary and Secondary
–  Double click on Secondary Server.

As showed in the image below . . .

Goto Security,
In Server Access – Who Can section, Add your Admin ID and Primary Server name in following Sections.
  – Create database & templates
  – Create new replicas

As showed in the image below . . .

Click on Save & Close

Create Replicas of Users Mail Files

To create mail replica copies of user mail files from primary to secondary, First you have to create mail folder in secondary server, because when domino first install, it donot create mail folder.

– At your Primary server,
– Open Domino Administrator Client,
– Goto File / Open Server , Select your secondary server, if its not appearing in the list, then click on Other and it will show you both servers, click on secondary server. You can also type in your secondary server name with its domain, for example D2/syed .Now you will be connected to your secondary server.
– Goto Files , Right click on Mail Parent folder of Domino, and create New Folder name mail (In my case I installed domino at D:\Lotus\domino\data

After the folder is created, Switch back to your primary server.
– Goto Files section,
– Select your desired mail file, (For test select single mail file, later you can select all or multiple mail files at a time to replicate), On right side panel, Click on Databases > Create Replica(s)

As showed in the image below . . .

A new window will popup with some options, First click on “Show me only cluster members” so that your cluster servers should appear in the list. Now select secondary server, and click on ADD,
Then select all three options of
– Copy Access Control List
– Create full text indexing for searching
– Exchange unread marks on replication.

As showed in the image below . . .

Now click Ok to continue.

Depends on the mail file size , It will few minutes or longer to complete the task. After it creates replica, the user mail file will appear on secondary server under mail folder.

TIP: You can force immeidate replication by using following command at console on both servers.

tell adminp p all

and monitor the console for information or errors.

–

. . .  TESTING  . . .

Before going further, some important tips and information on Clients and Clustering

[Some points in below paragraph have been taken from Lotusphere 2011 – How to build a better cluster – By Gabriella Davis – Paul Mooney]

The Notes client is “Cluster Aware”

● When it communicates with a Domino server that is in a cluster, that server updates a file on the client with its cluster name and its cluster mates
─ this makes the client aware that the server it wants is in a cluster and means even if the server is down, the client has built in information about where to go next
─ if you add a domino server to a cluster and rapidly shut down the original server, the notes clients will never get a chance to “learn” about the cluster
● If a primary server fails, the Notes client will search the other cluster mates it knows about for a replica of the database it wants

─ Searches cluster mates alphabetically,  not by availability
– this is very important in your planning
─ Opens database on the first available cluster mate it finds

Cluster.NCF

● The cluster.ncf is a file in the data directory on the client
● It contains details of all clustered servers you have connected to from that client, along with their cluster mates
– there can be several clusters in your cluster.ncf
● If you delete the cluster.ncf the Notes client will forget all about any clusters it was aware of, but will relearn the first time you reconnect to a clustered server

Notes 8.5.2 – Improved Client Clustering/Repilcas

● Client Transparency
● In the 8.5.2 or above client, the ability to failover to a cluster mate without the user being notified or stopped in their work has been improved
● During the following activities, if the primary server goes down, the user will be invisibly transferred to one of the cluster mates
─ Creating a new entry for PIM views (Mail message, Calendar entry, or To Do)
─ Opening a document in an application
─ Opening a different folder or view
─ Replying or forwarding with attachment
─ Creating a new mail message while multiple documents are selected
─ Deleting a single or multiple document
─ Sending a message
─ Switching back to Primary server when it becomes available

Ok Enough theory, its time to hit the road.

Configure your Lotus Notes Client and send an test message to any user or yourself.
You will see in logs that it immediately replicate/duplicate mail from your primary server to secondary server.

As showed in the image below . . .

You can also verify it by directly open user mail file from secondary server using admin client 😀 🙂

Testing from the client  side . . .

To test fail over switch at client end, temporarily disable Primary server LAN ,
At client end Open Notes client , when it will fail to find primary server , it will auto switch over to secondary server.
Also Try to send an test message to yourself for test purpose while the primary server is down for test purpose.

As showed in the image below . . . (client have switched over to D2 i.e our secondary clustered server)

At secondary server console , you will see notification of user failing over from primary to secondary.

As showed in the image below . . .

.

.

Now enable LAN on Primary server that we temporarily disabled for test purposes.
After few minutes, Secondary server will auto replicate data to primary server , so that primary server can update itself with the changes that occurred while it was offline.

As showed in the image below . . . (Replication in progress from D2 secondary server to D1 primary server)

& at your client end, you will be switch back to primary server where you can see your email that you sent from the secondary server while the primary was down.

.

.

.

.

Triggering Failover for CLUSTER

[Some points in below paragraph have been taken from Lotusphere 2011 – How to build a better cluster – By Gabriella Davis – Paul Mooney]

You can force all users to fail to another cluster server
─ Useful for Disaster Recovery testing
─ Useful for server maintenance work
─ Useful for migrations

● On the server console type the following command

Set Config Server_Restricted=
– 1 Prevent any new sessions opened on this server until restart

● Administrators bypass this restriction if using the Domino Administrator client

– 2 Prevent any new sessions opened on this server until setting changed

● Administrators bypass this restriction if using the Domino Administrator client

● Setting is persistent even through server reboots

– 0 Feature disabled

Also issue drop all command to force disconnection of all connected clients. („By the way, iMap and POP3 users will still have access to there Mail).

You can also set the maximum number of concurrent NRPC users allowed to connect to a server in NOTES.INI using following variable

Server_MaxUsers

„Set variable to a number determined in planning stage

Set variable using console command Or use NOTES.INI tab in server configuration document

Set config Server_MaxUsers = desired maximum number of active concurrent users, „Additional users will fail over to the other members of  the cluster

.

.

Creating Replication Document for backup purposes

Its a good idea to always have a connection document replicating between server cluster members because it replicates stubs too. Also if something got missed from routine replication, this forced replication will make sure everything is in synch.

.

Centralized Management is GOOD 🙂 by having single Configuration Document for all servers

Its a good idea to centralize manage all servers in cluster group.

Open Admin Client, Goto Configuration tab, here you will see two configuration docs for primary and secondary.
Double click on primary server configuration doc to take its properties,
As showed in the image below . . .

.

On Basic tab scree, select check on “Use these settings as the default settings for all servers:”
Then click on Save & Close.
As showed in the image below . . .

.

.

Now you will see * [All  Servers] doc
As showed in the image below . . .

Now rename the second doc with the secondary server name with any other name so secondary server will not take data from it, all configuration settings should be taken from centralized document * [All  Servers]

As showed in the image below . . .

WAN Connectivity Tips for Primary & Secondary Server

There are several scenarios that wan connectivity can be maintained. For example A live can be configured on Mail server WAN interface , but I will not recommend to put mail server on danger by directly attaching it to the internet, better to configure live ip on any of your router/firewall and then configure port forwarding of smtp/pop3 traffic to your server.
In this scenario If you have only single live IP , then you have to manually change the IP of primary server to secondary in case your primary goes down and you want to receive mails on your secondary server. Lot of discussion can be made here, but no time for it at a moment 🙂

Scenario # 1

You have public ip pool of /30. For example 1.1.1.1 and 2.2.2.2. Both ips are configured on Router and router do port forwarding for mail traffic. First configure 1.1.1.1 to forward mail traffic to primary server and 2.2.2.2 to forward mail traffic to secondary server. Now Request your web server hosting company to set secondary MX record to 2.2.2.2 (It is required so when no response is received from primary server smtp/pop3 , then it should try to send traffic to secondary MX which is 2.2.2.2

Now to test , quit primary server, wait few minutes, and then send an test email from the internet to your domain, it should arrive in your secondary server, and when the primary will be available , it will replicate its data with one another.

MORE TO COME . . .

Regard's
SYED JAHANZAIB