Chapter 12.  Periodic Maintenance

So far we have concentrated on the notion that disk usage for a study can be planned for in advance. For many larger trials it may not be possible to determine the exact number of CRF pages that will be received. For example, the accrual rate may be different than expected, or the percentage of re-faxed pages that occurs is higher than normal. For such trials, it is important to continually monitor information about current disk usage patterns and base future disk demand planning decisions on that information.

The best way to plan future disk needs is to monitor your current disk usage patterns and extrapolate. This can be accomplished using DF_WFdiskusage, which lists and graphs, in chronological order, the total and average disk usage by week for study fax pages. Example 12.1, “Sample output from DF_WFdiskusage” is an example of the information that the report provides. DF_WFdiskusage accepts several options to customize its behavior. For additional details see Standard Reports Guide, DF_WFdiskusage.

DF_WFdiskusage: Disk Space Used Per Week.         DFstudy 251. Feb 26,2015 10:30
Disk usage from week 201449 to 201505 inclusive
Yr   Wk     Kb      Each * represents 9 Kb
                    +---------+---------+---------+---------+---------+-------
2014 49:         521|*********************************************************max
2014 50:         346|**************************************
2014 51:         354|***************************************
2014 52:          82|*********min
2015 03:         182|********************
2015 04:         155|*****************
2015 05:         154|*****************
                    +---------+---------+---------+---------+---------+-------
Total:  1794 Kbytes in 7 weeks
 Mean:  256 Kbytes per week
  Min:  82 Kbytes (week 201452)
  Max:  521 Kbytes (week 201449)

A periodic procedure should be established whereby DF_WFdiskusage is executed for each active DFdiscover study. If available disk space is low, it should be run at least once per week. If available disk space is not a current concern, that interval can be extended to at least once per month. In either case, execution of this application at regular intervals is a task well suited to the UNIX cron facility.

With the output from DF_WFdiskusage, consider the following points when planning future disk needs:

The quickest method of recovery from the loss of your primary server is to have a duplicate server that can be deployed as soon as the failure of your primary server is detected. Ideally, your backup server should be identical to your primary server, including at least availability of other external services (modem lines, Internet access, power supply backup). Ideally, your backup server should be at a different physical location. If your primary server is located in an area that is more susceptible to hurricanes or earthquakes, pick a location that historically has lower risk for such events.

There are many software solutions available for synchronizing two systems. At DF/Net Research, Inc., we use rsync. With rsync and cron, two identical systems can be synchronized at a scheduled interval (e.g., once nightly, once per hour). Only data that has changed is transmitted between servers, making this a very efficient method of keeping two systems up to date with each other. Once two systems have been synchronized, you may want to limit what directories get copied to make the most of whatever network bandwidth you have available. This should include the same directories you would normally backup using tape. Usage of rsync is described in the man page for the application. See man rsync for details.

Because recovery involves the use of a second server, a license key for that server must be obtained before it can be tested or used in an actual disaster recovery. Test license keys are available from DF/Net Research, Inc. for this purpose. Should your primary server be involved in an actual disaster, you must request a temporary license transfer to your backup system.

True backups require a quiet filesystem. If this is done during off-hours, it is unlikely that any user will be modifying a study database. However faxes may still be arriving, especially from international sites. This problem can be solved at two levels:

The former solution has the problem that it may frustrate sites that are trying to fax while the backup is being done. The latter solution should be careful to not backup the incoming fax directory as its contents will not be quiet as long as one or more incoming faxes are being received.

DFdiscover can be halted from a shell script (like the cron process that performs the backups) by executing the command

#  /opt/dfdiscover/bin/DFshutdown -f

and then subsequently restarted using

#  /opt/dfdiscover/bin/DFbootstrap

If only an individual study directory (or group of study directories) needs to be backed up, then DFdisable.rpc is used to temporarily disable the study servers that must not be running while the backup is executing. This allows users of other studies to continue with their DFdiscover activities while the backup proceeds. When the backup is complete, DFenable.rpc must be used to re-enable the halted study servers.

There are many freely and commercially available applications for performing system backup. Discuss the options with your corporate IT team.

# cd /opt/studies/val254
# tar cf /dev/rmt/0 bkgd data dfsas dfschema drf ecbin ecsrc lib lut pages pages_hd reports/QC work

# cd /opt/studies/val254
# tar cf /dev/rmt/0 .

# cd /opt/studies/
# tar cf /dev/rmt/0 val254

It is highly recommended that a study setup be thoroughly tested before real data is accepted from investigative sites. This testing should include completing blank case report forms with actual data, faxing the case report forms into the system, validating the ICRed data records, and creating Query Reports to test the visit map, page map, and/or conditional plate and termination maps. The DFdiscover Study Setup Worksheets are an excellent aid in ensuring that all of the required steps are completed and documented.

The result of this testing will be a study database that contains data, CRF images, and Query Reports that are not relevant to the real study data. It is important before going live with a study to remove all of this test data. It is straightforward to remove this test data before the real data arrives; it is much more tedious to remove it once it becomes combined with real subject data.

For documentation purposes, the setup should be printed from the Study > Print menu in DFsetup and the current user permissions should be printed from the Permissions dialog in DFadmin.

A DFdiscover study is stored on disk as an inverted tree structure in the filesystem. The information required at any moment during the use of a study is available as one or more files in that filesystem structure. Using the UNIX filesystem directly has the advantage that this same information is also readily available to applications outside of DFdiscover, for example, for the purposes of scripting or working with third-party applications. However, this flexibility also has the drawback that UNIX filesystem permissions and the permissions required by DFdiscover are not always in perfect agreement. This can lead to users that are unable to open files that should otherwise be permitted to. The purpose of this section is to describe the permissions that DFdiscover requires and suggest ongoing maintenance to ensure that those permissions are maintained.

% /opt/dfdiscover/utils/DFstudyPerms #

# /opt/dfdiscover/utils/DFstudyPerms -f #

12.5.2.3.  Default study filesystem permissions

Table 12.1, “Study filesystem permissions” lists the study filesystem permissions. The permissions are reported as 3 triples of 3 characters. The first triple is owner permissions, the second group, and the third other. The 3 character positions, rwx, represent read permission, write permission, and search permission respectively. If a particular permission is not granted, it appears as a dash, -, in the listing. If a file or directory is checked by DFstudyPerms it is also checked for either exact permissions or minimum permissions. If it is checked for exact permissions, it must have exactly the listed permissions - any other permission will generate a message. If it is checked for minimum permissions, then additional permissions (for example, additional write permissions for group) are acceptable and will not generate a message.

	Note
	Most of the permissions are checked by DFstudyPerms but not all of them. It is expected that a future version of DFstudyPerms will include checking of these additional files.

Table 12.1. Study filesystem permissions

Name	File or Directory	Permissions	Type of check	Notes
.	Directory	rwxr-x---	Minimum	This is the study parent directory. If users are permitted to create their own sub-directories, the permissions will need to be rwxrwx---
batch	Directory	rwxr-x---	Minimum
bkgd	Directory	rwxrwx---	Minimum
bkgd/DFbkgd???.tif	File	rw-rw----	Minimum
bkgd/plt???	File	rw-rw----	Minimum
bkgd/DFbkgd???	File	rw-rw----	Minimum
data	Directory	rwxr-x---	Minimum	Write permissions on this directory should never be granted to any account other than datafax.
data/*.dat	File	rw-------	Exact
data/*.idx	File	rw-------	Exact
data/*.jnl	File	rw-r-----	Exact	These audit trail files must not be writable by any account other than datafax. They are readable for the purposes of audit trail reports like DF_ATmods.
drf	Directory	rwxrwx---	Minimum
dde	Directory	rwxrwx---	Minimum
dde/sets	Directory	rwxrwx---	Minimum
dfsas	Directory	rwxrwx---	Minimum
ecbin	Directory	rwxr-x---	Minimum
ecsrc	Directory	rwxr-x---	Minimum
lib	Directory	rwxrwx---	Minimum
lib/DFcenters	File	rw-rw----	Minimum
lib/DFfile_map	File	rw-rw----	Minimum
lib/DFschema	File	rw-rw----	Minimum
lib/DFschema.stl	File	rw-rw----	Minimum
lib/DFserver.cf	File	rw-r-----	Exact
lib/DFsetup	File	rw-rw----	Minimum
lib/DFsetup.backup	File	rw-rw----	Minimum	This file contains the previous version of the study setup and is overwritten as part of the initialization process of DFsetup.
lib/DFtips	File	rw-rw----	Minimum
lib/DFvisit_map	File	rw-rw----	Minimum
lib/DFccycle_map	File	rw-rw----	Minimum	These remaining files in the study lib directory are optional.
lib/DFcplate_map	File	rw-rw----	Minimum
lib/DFcterm_map	File	rw-rw----	Minimum
lib/DFcvisit_map	File	rw-rw----	Minimum
lib/DFedits	File	rw-rw----	Minimum
lib/DFlut_map	File	rw-rw----	Minimum
lib/DFmissing_map	File	rw-rw----	Minimum
lib/DFpage_map	File	rw-rw----	Minimum
lib/DFqcproblem_map	File	rw-r-----	Minimum
lib/DFqcps.prolog	File	r--r-----	Minimum
lib/DFqcsort	File	rw-rw----	Minimum
lib/DFraw_map	File	rw-rw----	Minimum
lib/QCcovers	File	rw--rw-----	Minimum
lib/QCmessages	File	rw--rw-----	Minimum
lib/QCtitles	File	rw--rw-----	Minimum
lut	Directory	rwxr-x---	Minimum
pages,pages_hd	Directory	rwxr-x---	Minimum
pages/????, pages_hd/????	Directory	rwxr-x---	Minimum	These are the directories, organized by year and week of year, in which the CRF images are stored.
pages/????/???????, pages_hd/????/???????	File	rw-r-----	Exact
reports	Directory	rwxr-x---	Minimum	If users are permitted to install their own study-specific reports, these permissions will need to be rwxrwx---.
reports/QC	Directory	rwxrws---	Minimum
reports/QC/*-??????	File	rw-rw----	Minimum
reports/QC/QC_LOG	File	rw-rw----	Minimum
reports/QC/QC_NEW	File	rw-rw----	Minimum
reports/QC/SENDFAX.log	File	rw-rw----	Minimum
reports/QC/SENDFAX.qup	File	rw-rw----	Minimum
reports/QC/internal	Directory	rwxrwx---	Minimum
reports/QC/sent	Directory	rwxrwx---	Minimum
reports/QC/sent/*-??????	File	rw-rw----	Minimum
work	Directory	rwxrwx---	Minimum
work/DFvisit.dates	File	rw-rw----	Minimum
work/DFX_*	File	rw-rw----	Minimum
work/DF*.drf	File	rw-rw----	Minimum
work/DF_QCupdate.log	File	rw-rw----	Minimum

The work directory for a DFdiscover study includes a mixture of temporary files created by DFdiscover and temporary files created by users.

Files that have names beginning with DFX are created by the DFdiscover DF_XXkeys report. They are overwritten each time that DF_XXkeys or DF_QCupdate is executed. In most circumstances, they should be left alone. However, if disk space is at a premium they can be deleted, as they will be re-created the next time the reports are run.

The other temporary files that might be found in the work directory will be specific to each DFdiscover installation. You will have to use your own discretion in deciding which files to delete. As a general guideline, files with the following attributes are good candidates for deletion:

The reports directory for a DFdiscover study includes study specific reports as well as Query Reports. The Query Reports are stored in a further QC sub-directory of the reports directory. Reports that are created by DF_QCreports are stored in this QC sub-directory and then are moved to a further QC/sent sub-directory when they are successfully faxed to investigators. If reports are created by DF_QCreports but are never subsequently faxed out, they will be left in the QC sub-directory. Periodically check the files in the QC sub-directory of the study reports directories for such reports and delete them if they are out of date. If there is any doubt, this step should be coordinated with the staff member responsible for creating Query Reports for the study.

When preparing to close-out a study or archive a copy for interim analysis, the following issues need to be considered:

A minimal set of steps for making an archive copy of the primary database records might follow the scenario in Example 12.6, “Making an archive copy of the primary database records for study 254”.

# mkdir -p /opt/archive254/data
# cd /opt/studies/val254
# tar cf - lib | ( cd /opt/archive254; tar xpf - )
# foreach p ( `DFlistplates.rpc -s 254` 0 511 )
? DFexport.rpc -s primary 254 $p /opt/archive254/data/exp$p
? end
# tar cf /dev/rmt/0 pages

Rarely, a user may encounter the message 'image not available' in the DFexplore Review Images dialog. Before retrieving a lost image file, attempt to determine the cause of the problem and log it. Periodically review the log to look for any systematic problems that might be correctable.

The DF_ICrecords report verifies the integrity of data records for all or specified plates in the database. It does this by confirming that each record has the correct number of fields defined by the plate definition in the study setup. Additionally, DF_ICrecords performs the following checks on each record in the specified data files:

The latter check detects more than one primary record for a set of keys and also detects secondary records that have no primary.

Executing this report with the -d option creates a DRF named ICrecords.drf that contains a record for each data record that fails one or more of the above checks. Using Select-By Data Retrieval File DFexplore is used to correct each problem record detected by DF_ICrecords. After resolving the problems, re-execution of DF_ICrecords will generate no error output.

In addition to the DF_ICrecords report, the shell-level utility, DFcmpSchema, are used to more stringently examine each record. DF_ICrecords ensures that the database structure is consistent with DFdiscover requirements. DFcmpSchema ensures that the database content is consistent with the study schema.

The DF_ICimages report verifies that each data record in a study database references a CRF image in the study pages directory, and conversely that each CRF image in the study pages directory is referenced by exactly one data record.

In most cases, the DF_ICimages report should be run with the -x option which forces the report to execute with the database in a read-only state. Without this option, the database is in a read-write state that allows the database state to change while the report is being run. The end result may be that DF_ICimages indicates problems with are present because they are timing related.

If the DF_ICimages report detects a record that references a missing CRF image, follow the steps in Retrieving lost CRF images.

If the DF_ICimages report detects a CRF image that is not referenced by a data record, two resolution methods are possible:

Finally, if DF_ICimages detects a CRF image that is referenced by two or more data records, DFexplore is used to review all of the involved records and delete all but the correct primary (or secondary) record.

The DF_ICqcs report:

The DF_ICqcs report includes the -r option that causes the report to attempt to repair problems resulting from un-referenced queries and final records having unresolved queries. Inconsistencies are resolved by deleting all un-referenced queries. On final records, the unresolved queries are marked as resolved.

Multiple queries that reference the same data field can be resolved by using DFexplore to delete all but one of the duplicate queries.

Each of the client applications communicates with the DFdiscover server using HTTPS on port 443. This port must be open on any firewalls between the local computer and the study server.

This is industry-standard technology that encrypts the bi-directional communication using a 'certificate of trust' provided by the server. It is the same technology used by banks and the majority of secure, global web services.

You can visually confirm that the communication is secure. After logging-in to DFadmin select Help > Certificate Info and look for the green checkmark.

There are various log files that are maintained by DFdiscover that can be periodically truncated. In truncating these files it is important to maintain the file permission and ownerships that were in place before the file was truncated. Also, you should choose between completely clearing all of the log messages or maintaining a context of the most recently written log messages. In the examples below, both methods are indicated.

DFserver.rpc.251[27239]: start on teamserver at Mon Jan 22 17:23:37 2018
DFserver.rpc.251[27239]: exit at Tue Jan 23 09:29:53 2018

DFedcservice.101[21877]: jack@localhost deleted all study data Fri Dec 1 12:00:09 2017
DFedcservice.101[21877]: jack@localhost deleted all study data and setup info Fri Dec 1 12:01:19 2017
DFedcservice.101[21877]: jack@localhost deleted study from datafax registry Fri Dec 1 12:02:58 2017

Certain log files contain information that is relevant to a DFdiscover installation over its entire history. These log files should not be pruned.

HylaFAX provides a detailed log of all transactions that is very useful in debugging faxing problems. The information contained in these log files includes the remote fax machine number, the speed and encoding method used to transfer the fax and information about the duration and success or failure of each transmission. These log files need to be cleaned up periodically, and HylaFAX provides two scripts to accomplish this.

The first of these, faxcron, truncates the log files, and the second, faxqclean, is responsible for purging job description and old document files that are left over after a fax request has completed. Both of these scripts are normally run automatically by the UNIX cron facility.

To ensure that scripts have been correctly configured on your machine you will need to log in as root (or have a super-user perform these steps) and execute the following commands:

# crontab -l > mycronjobs
# more mycronjobs

If you see lines containing faxqclean and faxcron, the scripts are already correctly installed and no further action is necessary. If they do not appear, edit the mycronjobs file and add the following lines to the end of the file:

25 23 * * * /opt/hylafax/sbin/faxqclean
0 3 * * 0 /opt/hylafax/sbin/faxcron

which executes the faxqclean script every day at 11:25PM and the faxcron script every Sunday at 03:00AM. Save the file and then inform cron of the changes with the command:

# crontab mycronjobs

It may be necessary to increase the frequency of the script execution for a very high-volume site. In such a case, it can occur that the partition containing the HylaFAX logs (typically /var) will fill with log files leaving no space for normal system operation.

Periodic maintenance of a DFdiscover system as described in this chapter is a preventive measure that can save many hours or days of corrective or restorative work. It also gives DFdiscover users a feeling of confidence that the system is always available and running smoothly. Done regularly, this maintenance should require no more than 30 to 45 minutes per week.