Chad Scott

If you are unfamiliar with the strace utility for Linux, it is quite handy for diagnosing a wide variety of problems, and it is especially good at hang-type conditions. I recently set up a new server on Ubuntu 9.10 (I know, not supported) and was preparing it for some server.load testing. After creating the mail files, I ran the Idle Workload test just to baseline the number of users the system would handle before bonking. With almost no load on the machine, I fired up a test of 500 users. The output went to 470ish before Domino entered a hang state. Weird. I killed the server and tried again. Same problem. There was no obvious cause of the hang, and reviewing some NSD call stack snapshots did not show much other than waiting threads.

On the third try, I used strace to attach to the server process, as follows:

> strace -p <server PID> -f

This allowed me to view the system calls for the server process and all its children (i.e. the server threads). It took about two seconds to find the problem. Various system calls were failing with errors indicating that too many files were open. Duh. It's a new server, and I forgot to change the open file limits. Once I set this up using Technote 1377724, the test ran to completion with no problems.

I would have resolved the problem eventually through other means, but using strace saved me a boatload of time. If you have Linux systems in your environment, I highly recommend that you carve out some time to play with it.

A new Technote has been released to document the history and fix for the issue Erik Brooks blogged about last week. In short, there was a regression caused by a fix in recent releases that could cause existing code to return an error where it previously did not. With the new fix, this no longer occurs and the fix for the original problem with future-dated views is preserved.

I'm sure you've seen that the DAOS fix bundle 8.5.1 FP1 IF1 was released to Fix Central today. If you are running DAOS, you should strongly consider installing IF1 to address the issues outlined in the Technote. Most DAOS issues we're running into are resolved by one or more of these fixes. If you are experiencing any problems at all with DAOS, then you definitely need to install this fix and reassess the problem afterward.

New to 8.x is a feature that allows AdminP requests to be created directly in the admin4.nsf of the server that will process the request. From Administration Help:

The Administration Process can deposit requests directly into the Administration Requests database (ADMIN4.NSF) -- The Administration Process can now deposit administration requests directly into a target server's Administration Requests database (ADMIN4.NSF) instead of depositing the request into a local copy of ADMIN4.NSF.

The direct deposit feature also applies on the client side. In particular, an 8.x client will attempt to deposit the Change HTTP Password in Domino Directory request in the administration server's admin4.nsf following a password change when synchronization is configured. In complex environments, regional clients and servers may not have a direct path to the administration server, causing the client to fail when attempting to create the request. In this case, you can use the ADMINP_DONT_ATTEMPT_DIRECT_DEPOSIT=1 parameter to instruct the client to bypass the direct deposit route and create the synchronization request in the home server's admin4.nsf. Replication will then carry the request to the administration server, allowing the process to complete successfully.

Be on the lookout for an issue where 8.0.x clients will reapply desktop settings even though the policy specifies that they should only be set initially. This is being tracked in SPR HYAI7DR4Z5 and has been fixed for 8.0.3. The issue does not impact 8.5.

I was asked about a theoretical limitation on databases on a Domino cluster based on some (apparently) archived information suggesting a limitation of about 8,000. I found that in older releases, it was necessary to use the CLUSTER_DIRECTORY_MAX_POOLSIZE_MB notes.ini parameter to increase the default 16 MB memory pool for the cluster directory. This was changed in 7.0.3 and 8.0 to be dynamically sized up to 250 MB, theoretically allowing for about 125,000 databases. Additional memory pools like RTR_MAX_POOLSIZE_MB and DIRMAN_POOLSIZE_MB would also come into play, along with many other practical limitations.

In response to: Concerns over DAOS Estimator?

The second "Total Duplicate Attachments found" line does precede the accurate statistic that was formerly labeled "Total DAOS Eligible Attachments." I've created SPR CSCT7PKM8R to correct this. The big thing to remember is that the data is still good. Just remember that the second line is the number of DAOS-eligible attachments.

I ran into an issue this morning where new and modified program documents are not recognized by the Domino job scheduler. If I added, deleted, or modified a program document, I saw no change in the output from "show sched" on the console, and the program documents never ran. It turns out that this is behavior introduced by the set of NAMELookup fixes documented in Technote 1254752. The various NAMELookup flags are enabled using the notes.ini parameter DEBUG_ENABLE_UPDATE_FIX, which has the following default/maximum value by release:

• 6.5.5 FP2 - 7.0.2: 0/8191
• 7.0.3: 0/16383
• 8.0 and 8.0.1: 8191/16383
• 8.0.2: 8191/32767
• 8.5: 32767/32767

These values enable flags that turn off unnecessary view rebuilds by functional area. For the program document issue, that flag is decimal value two, which means that if you are running any release of 8.x without the notes.ini parameter set or 6.5.5 FP2 or later with the notes.ini parameter set to a value that includes this flag, you will experience this issue. To work around the problem, you can remove the flag, which means subtracting two from the value of DEBUG_ENABLE_UPDATE_FIX in notes.ini. If the parameter is not there, you should set it to the maximum for your release minus two.

I wrote last week that you can run DAOS Estimator from your client to collect information from the server. Unfortunately, there's a problem with this approach. Estimator will iterate through the server databases without a problem, but it will attempt to open those databases locally. Obviously a client will not have all of the server databases local and in the same relative path, so these database opens will fail with 0x103 (File does not exist). SPR DANG7NPHYJ has been created to request additional functionality that will allow Estimator to open databases on the server instead of trying to open them locally.

Did you know you can run DAOS Estimator from your client machine? To do so, place the appropriate daosest executable in your Notes program directory, and run the following from a command prompt:

> daosest Server/Org!!

This is much safer than running daosest on the server, as problems should be limited to your client and not the server. This also has the benefit of far less administrative overhead, as all servers can be scanned from a single workstation. Note that when running in this fashion, daosest will run with the credentials of the active user ID instead of the server ID, so plan accordingly.

Official documentation is available here.

Notes/Domino 8.0.2 FP1 is now available on Fix Central. This is the first fix pack ever offered for the Notes client. You can read the details here. Note that in addition to the dozens of fixes, there is one configuration change: Streaming Cluster Replication is now re-enabled by default. This was disabled in 8.0.2 due to some bugs that caused it not to work well, but those have since been fixed.

I've worked in Domino support for nearly five years, and in that time I've learned that the various monitoring tools included with the product tend to be drastically underutilized by customers. I think it's largely a problem of quantity; there are so many tools that it can be hard to know where to begin. As such, I'm going to cover a multipart introduction to Domino monitoring to help administrators get started using this powerful suite of features.

We'll begin by looking at statistics, monitoring's most basic building block. Domino captures a wide range of statistics by default, and these can be viewed in the Administration client on the Server -> Statistics tab or by running the console command "show stat". You can drill down on individual statistics through the Admin client or run a modified version of the console command to retrieve specific statistics (e.g. 'show stat Server.AvailabilityIndex'). These are great tools for getting a quick snapshot of a particular statistic, but most of the value lies in tracking statistics over time, and it certainly isn't practical to refresh the Admin client or run the console command repeatedly.

To solve this problem, you can create a Statistic Collection document that, not surprisingly, collects statistics. Once a statistic collection document is in place, snapshots of statistics from your servers will be saved to statrep.nsf, giving you an archive of prior statistics. A word of warning: statrep.nsf can grow very large if you let it. The database has a default document purge interval of 14 days, and you should think long and hard before changing it. Note also that statrep.nsf is not a replica throughout the domain. If you decide to make statrep.nsf replicate to all servers, make sure you plan accordingly. If you have a lot of servers, it's a near guarantee that this database will be huge.

With a Statistic Collection document in place, you will be able to leverage additional monitoring features that will make your life much easier. And really, isn't that the goal of every administrator? I'll cover one of those additional features--statistic charting--in Part 2.

The DAOS development team has published a great best practices guide for DAOS backup and restore. Be sure to check it out.

Heads up on a bug with 2048-bit encryption and DAOS. This problem manifests if you use 2048-bit encryption (not the default) and choose to encrypt NLOs (default). Changing either works around the problem, and a hot fix is available should be available this week if you need both.