Syed Jahanzaib Personal Blog to Share Knowledge !

February 22, 2013

IBM Lotus Domino Migration and common troubleshooting Tips


lotus

Recently, in our company, we had a power failure dueto UPS malfunction. After restoring everything back to normal, our email system IBM Lotus Domino 8.5.2 (FP2) services failed to start. We also had Symantec for Lotus Domino installed. Its kernel also got panic, and failed to fire up. Few mail boxes were also corrupted and there inbox view disappeared.
Note that we have very big email boxes for many users like 15,20 30GB and mailtracking account crying loud at 40+GB.

After many hours of stress and painful struggling , removing Symantec for mail security, its port configuration in ports section, I was finally able to start the Domino Server and emails start flowing .

When everything got back to normal (except for the Symantec mail security) and many hours downtime dueto R&D, I finally thought to create a DR server for DOMINO so all data should be moved to this DR server, and replace the Production server hardware with newer model of IBM Xseries M4 server.

Following is a complete method on how I migrated my Lotus Domino server to new machine.

  • SCENARIO #1

IBM LOTUS DOMINO PRODUCTION SERVER

OS / APPLICATION
Windows 2008 R2 SP1 installed in OS C: drive
IBM Lotus Domino 8.5.2 FP4 installed in D:\LOTUS
Symantec For Lotus Domino 8.0.9.151 installed at setup default location

HARDWARE
IBM Xseries M2
RAID1 for OS
RAID5 for Lotus Domino Installation & DATA

For migration I followed following steps.

1) Install Windows 2008 R2 SP1 on new machine in C: Drive
2) Run Windows Update
3) Install IBM Lotus Domino 8.5.2 (or whatever Version you have) in D:\LOTUS {make same folder structure as previous one}
BUT
DO NOT RUN SERVER CONFIGURATION WIZARD . . .
4) Install Domino FP4, if any (or whatever FP Version you have)
5) Install Symantec For Lotus Domino 8.0.9.151 at setup default location (If you have SAV)
6) Copy notes.ini from Production Domino Server to this new machine at D:\LOTUS\DOMINO
7) STOP DOMINO SERVER AT PRODUCTION SERVER by using quit command in console.
(You can copy data from live server to this DR but chances are high that few mail boxes will require consistency check and possibly few attachments or user inbox view can be corrupted and need to re rebuild by refresh design and fixup. In my case I had the DATA folder in my file server, so I copied it from there.

8) Copy complete DATA folder from Production Domino Server (or File Server if you have backup it to file server) to this machine at D:\LOTUS\DOMINO  (Overwrite all files).
Please note that DATA folder can be big in size and can take long time for copying as it also contains mail and possibly archive folder which contains users mail files. In my case, DATA folder size was around 400GB and it took around 5 hours to copy the whole data.
The DR server was low specs IBM Xseries 3650 refurbished server with RAID5 for DATA .
At production server, I was using IBM Xseries M2 with RAID5 for DATA , and Gigabit connectivity, Copy time can vary as per the network and hardware capacity.

9) Unplug the Production server from LAN.
10) Rename the DR server name with the Production server name and same ip address, same as production domino server. and restart

Upon rebooting, Start the Lotus Domino Server console and monitor all the messages appearing on the screen. It may take some time to start the server because it will first initialize all the LOGs file. In my case it took 15-20 minutes to initialize the server at first time. You can delete un necessary logs file like LOG.NSF to speed up the process. Also dont forget to install license for Symantec Mail Security For Domino. Usually there are two license files for SAV, one for product, second for PAS, and renewal updates.

 


 

LOTUS DOMINO CRASH NiGHTMARE    :-o)

Last week I received following error at my Domino Server.

LN-ERROR

I tried restarting server several times, install all windows updates, removed antivirus for test purpose, removed all un necessary services, Updates Server hardware Firmwares and all other patches but still i received above error showed in the image.

QUICK FIX: Delete mil.box and log.nsf file from DATA folder and see if it helps. It helped me in starting Domino server. ( I guess some corrupt mail with undetermined string crashed the server). Also open the domino cosole, sometimes the last line gives you some crucial information. If Several crashes which occur just after mail delivery to a specific user , better to check his mail database, rules set, agents etc. Also check web access logs, name picker from iNotes DB also causes some issues.

I then re installed fresh copy of Windows 2008 OS on DR VM, Updates patches, Re install Fresh copy of Domino, and then I imported the DATA folder from the backup, but still this error keep appearing at the time of startup at various timings. After lot of googling, and a good working TIP from previous admin, I was able to solve the issue.

– Remove Lotus Domino, delete all its leftover entries and folders, usually D:\lotus folder
– Restart Server,
– Re Install Domino Server, Update FP if any, Its always a good idea to install latest FP availble
– Don’t import whole DATA folder from previous server, Just copy following data
>> LOTUS\DOMINO\notes.ini
>> LOTUS\DOMINO\DATA\MAIL folder which contains users mail
>> LOTUS\DOMINO\DATA\ARCHIVE  folder which contains users archive (If any)
>> LOTUS\DOMINO\DATA\name.nsf   admin4.nsf  server.id admin.id cert.id
>> LOTUS\DOMINO\DATA\INOTES  (If mail web access is enabled)
>> LOTUS\DOMINO\DATA\dominocg.nsf , iwaredir.nsf (If mail web access is enabled)

Now restart Domino server, hopefully it will work fine 🙂


 

Server Error: File truncated – file may have been damaged

LN-File-Truncated-message

If you see the above error message when you try to open your mail box via Lotus Notes, or from the Domino Admin client, then simply run fixup on your mail box via console.

load fixup -f mail/yourmailbox.nsf
(also add -J if you have transactional logs enabled)


 

Cannot find external name: NAMESORTVIEWPO

IMAGE$582E57263ED9BF60

Open your inbox
Select ActionsFolderUpgrade folder design
Choose “Automatic” and complete the process.
Close/re-open mail.

This should fix your problem.


 

Domino Console Commands

To show task or users
show task
sh task

show users

Administration Process Force update
tell adminp p all

To restart HTTP / Router or SMTP task
tell http quit
tell http start

tell smtp quit
load smtp

# Update router configuration

tell router update config

# Tell router task to quit

tell router quit
load router

To disconnect all notes users on server

Drop all

To restrict all users from opening new session on this lotus server
SET CONFIG SERVER_RESTRICTED=4
or add entry in notes.ini as following
Server_Restricted=4
and restart lotus notes server
once all done you Un-Restrict by
set config Server_Restricted=

Mail File Repairing / Compacting / ODS conversion commands

During the day mailboxes only grow even if you delete every email the mailbox size will not change. During the night hours you should have the compactor service run to remove all free space in your mailbox. Example:

To initiate archiving the mail DB (You should run this first)
load compact -a mail\yourmailfile.nsf

To compact DB with removing whitespace and reduce file size.
load compact -B mail\yourmailfile.nsf

To repair mail files, (-J for Transactional logs, if you are not using it then remove -J)
load fixup -F -J mail\

To convert Database ODS from previous to new (Dont forget to add Create_R85_Databases=1 in notes.ini)
load compact -c mail\

To shutdown / cancel compact process
tell compact quit

UPDALL keep view indexes and full-text indexes up-to-date.
load updall -R mail\


 

LOTUS DOMINO SERVER MAIL RETRY VALUES

Messages going out to the Internet are placed into a Retry state, if there’s issue attempting to send the message . Default retry timings are

15 minutes,
then 30 minutes,
and then every 45 minutes for 24 hours (by default). This is also configurable.

To force delivery , you can use route * command in console.


LOTUS TRAVELER 8 CONSOLE COMMAND

https://www-01.ibm.com/support/docview.wss?uid=swg21668603&acss=danl_722_web


More experiences for second time lotus restoration – 19-Nov-2018

WE have 2 email servers running, mail1 is primary & mail2 is secondary replica. Last week we had multiple disk failure on our mail1 server. after re-creating RAID, I reinstalled lotus domino, & was able to restore the primary MAIL1 server using previous mail server data files, & at the moment all functions seems to be normal. Following are few issues I encountered,

1) When I started the server 1st time, It gave following error,
Configuration Error: Server record not found. The Server Name in ID file [server/org], does not match any Server found in the Domino Directory in the ($Locations) view. Check that the Server name field in the Server record is a hierarchical name

To sort this i issued following

Load convert -d names.nsf * pubnames.ntf

Afterwards server started fine, but some users entries were missing in directory of MAIL1. I deleted names.nsf and create new replica from mail2 to mail1 & this part went fine.

2) admin4.nsf was there but on console I was seeing missing admin4 entry, therefore I re-copied this file from the backup,

3) mail2 was not replicating the new emails TO mail1 server, after doing restart of all servers, the problem went away and syncing started to work fine, including lotus traveler as well.

4) Today users are complaining that the email that they have read on mail2 server, are now showing UNREAD in mail1 server, I dont have solution for it so I am advising them to to manually mark them as read.

Still watching console for any error if any …

19/Nov/2018


Helo command rejected: Host not found

When I try to send one particular domain, my email always bounce back with following error

Error transferring to mail.XXXX.XXX; SMTP Protocol Returned a Permanent Error 554 5.7.1 : Helo command rejected: Host not found

After getting some help from spiceheads, I was managed to sort it.

I made one change that solved this issue.

Under Config > Server > Current Server Conf > Basics >  “Fully qualified Internet host name” > it was set to LAN DNS name, I changed it to my mail server internet FQDN likemail.mydomain.com 

More information here.

 


Regard’s
Syed Jahanzaib

%d bloggers like this: