Chapter 6. Batch Edit checks

DFbatch needs a batch control file as input. This must be created by the user in advance of invoking DFbatch. The purpose of the batch control file is to specify criteria that select records from the study, optionally name edit checks to be executed (by default all edit checks referenced by variables of selected records are executed), and state actions to be applied as the edit checks are executed.

<?xml version='1.0'?>
<BATCHLIST version='1.0'>
    <BATCH name="simple">
        <TITLE>This is a very simple batch</TITLE>
        <DESC>This batch executes edit check aeCoding on all
        records that are currently at level 2.</DESC>
        <ACTION>
            <APPLY which="none"/>
            <LOG which="data msg qc" file="simple_out.xml"/>
        </ACTION>
        <CRITERIA>
            <LEVEL include="2"/>
            <EDIT>aeCoding</EDIT>
        </CRITERIA>
    </BATCH>
</BATCHLIST>

The first thing that you will notice is a new language. The batch control file is specified in eXtensible Markup Language, XML. In XML parlance, the input file is a document. It turns out that the log output from DFbatch is also written in XML. There are many public domain tools (written in Java, Perl, C, etc) that are to parse and transform XML. Some web browsers are even able to display XML directly.

XML is a markup language and the markup is in elements and attributes. The extent of an element is marked with tags, namely a start tag and an end tag. The start tag and end tag for an element have the same name but different notation. The start tag is denoted with <TAG> and the matching end tag is denoted with </TAG>. The terminology is important: for the element named TAG, the start tag is <TAG> and the end tag is </TAG>.

The content of an element is the data between the start and end tags. The relationships in the content are achieved by nesting elements.

In the example,

are elements. Each document must have exactly one element, called the root, within which all other elements are nested. In the example, BATCHLIST is the root element. It is an error for any text to appear after the end tag of the root element.

Nested elements must be properly balanced or the document will be invalid, meaning that it cannot be parsed.

<A><B></B></A>

<A><B></A></B>

Elements that have no content are called empty elements. Empty elements are denoted with a single tag that represents both the start tag and the end tag. The notation for an empty tag is <TAG/>. In the example,

are empty elements. Empty elements are typically used to convey information via their attributes.

Data about elements can be kept in nested elements or, if the data is itself not structural, in attributes. In the example,

The example control file contains a single BATCH named simple. The batch has a brief TITLE and a more verbose description in DESC. These two elements are present for documentation purposes only and do not influence the processing.

The records to be processed are selected by the CRITERIA element. In this example, only those records that have a current validation level equal to 2 and reference the aeCoding edit check are selected. For each selected record (those records that have a validation level of 2), the data fields are traversed in tab order and the edit check aeCoding is executed at every data field that references it by name through at least one of the variable's plate enter, field enter, field exit, or plate exit attributes. Other edit checks are not executed. Any actions that cause messages to be generated, queries to be added, or data field values to be changed are logged to the file simple_out.xml. The changes are not however applied to the database as is indicated by the which attribute of the APPLY element.

Of equal importance to the behavior that is stated, is the behavior that is not stated. In particular,

The example control file must be saved to a named file that is accessible to the DFbatch program. There are two possible locations:

For file naming conventions, we recommend that the control file use the name of the batch as the basename with a _in.xml extension (meaning input XML file). For example, a control file named simple for study 254 which is stored on the server where study 254 has a STUDY_DIR of /opt/studies/val254 would be saved to /opt/studies/val254/batch/simple_in.xml.

To process the control file, DFbatch needs to login to your DFdiscover server the same way DFexplore does when you login. If you need to use a proxy server to access the internet, DFbatch automatically uses the same proxy settings as DFexplore. If you need to change these settings, use the Proxy Setting window in DFexplore to do this. Next, you need to tell DFbatch the name of your DFdiscover server, your username and your password. You can do this using -S, -U and -C command line options, by setting DFSERVER, DFUSER and DFPASSWD environment variables or by using DFpass. DFbatch is invoked with the command:

% cd /opt/studies/val254/batch
% DFbatch -S server.somedomain.com -U datafax -C passwd 254 -i simple_in.xml

The login, study number and control file arguments can appear in any order. For more invocation options, see Invoking DFbatch.

The result of the command is the execution of the batch and the creation of the log file, /opt/studies/val254/batch/simple_out.xml on the server (as requested by the file attribute of the LOG element). The log file records the details of the batch execution but is not post-processed in any way.

A more common way of running DFbatch is to process the batch, applying any actions and creating the log file as before, and then immediately post-processing the log to create a viewable HTML page. This is achieved with -p xsl as in:

% cd /opt/studies/val254/batch
% DFbatch -S server.somedomain.com -U datafax -C passwd  -p xsl 254 -i simple_in.xml -o somedir/simple.html

The option requests post-processing via a default XSL transformation (supplied with DFbatch) and the result is written to simple.html, typically in a directory that can be viewed from a web browser. The HTML output is explicitly re-directed to an output file at execution time (rather than making that file an attribute of the input control file) to allow for the various web-publishing infrastructures that are present at DFdiscover installations.

Alternatively, an existing log file can be transformed at any time into an HTML file using DFstyle, a companion program to DFbatch. The invocation of DFstyle requires a log file created by a previous invocation of DFbatch (not an input control file).

% cd /opt/studies/val254/batch
% DFstyle -p xsl simple_out.xml > somedir/simple.html

The exit status of DFbatch reflects the success of the command. If the exit status is 0, DFbatch executed successfully. Any other exit status indicates that an error occurred. The text of the error will appear in one of the last elements of the log file.

% /opt/dfdiscover/bin/DFbatch -S server.somedomain.com -U datafax -C passwd 254 -i simple_in.xml
% echo $status
0

Certain characters have special meaning to an XML parser and hence cannot appear directly in an XML document. This includes the input control file. To use one of these five special characters in an XML document, the corresponding entity must instead be used.

<?xml version='1.0'?>
<BATCHLIST version="1.0">
    <BATCH name="special">
        <TITLE>Run checkElig &amp; checkDemo</TITLE>
        <DESC>Among other things these edit checks test
        that the subject&apos;s age is &gt;= 20 and &lt;= 50.
        </DESC>
        <!-- note that &, <, >, ', and " can appear inside
        comments because comments are not parsed -->
        ...
    </BATCH>
</BATCHLIST>

Record selection criteria are similar to those found in the By Data Fields retrieval option of DFexplore. Retrieval criteria for records are by: site ID, subject ID (not subject alias), visit, plate, level, status, creation date, modification date. These map respectively to the elements: SITE, ID, VISIT, PLATE, LEVEL, STATUS, CREATE, MODIFY. These criteria are specified as nested elements of the CRITERIA element. If any element is omitted (or empty), the records to be selected are not constrained by this element. For example, if PLATE is omitted (or empty), there is no restriction on plate and hence all plates are retrieved, subject to the other criteria. Retrieval by center and pattern are currently excluded. Where multiple elements are given as criteria, the selected records must satisfy the inclusion criteria of each element, that is, the selected records are the intersection set of the sets of selected records satisfying each element individually.

Each element is expected to appear at most once. If an element (accidentally) does appear more than once, the last element overrides any previous occurrences.

Each criterion allows a value, range, and/or lists of both. These are expressed as the value of the include attribute as in:

include="val,min-max,val2"

<CRITERIA>
    <STATUS include="final"/>
    <VISIT include="1-5,10,20-29"/>
</CRITERIA>

The CRITERIA element accepts one optional attribute, sort. The value of the attribute determines what sort order, if any, is applied before processing to records that meet the selection criteria. The value is constructed from one or more sort keys (precedence for multiple leys is left to right) from the list: id, visit, plate where multiple keys are separated by ; and each key is preceded by either + to indicate that the key is sorted in ascending order, or - to indicate that the key is sorted in descending order.

<CRITERIA sort="+id;-visit;-plate">
....
</CRITERIA>

It is also possible to select records by an existing DFdiscover Retrieval File (DRF). This is specified with the IDRF empty element and the file attribute as in:

<IDRF file="test.drf" />

<IDRF file="sub_folder/test.drf" />

<IDRF file="/STUDY_DIR/drf/test.drf" />

The use of IDRF is mutually exclusive of the other selection criteria and it is an error for the two to appear together in one batch's CRITERIA. Additionally, records selected by the IDRF element are processed in the order that they appear in the file, ignoring any value assigned to the file attribute of the parent CRITERIA.

By default, all of the edit checks that are referenced by data fields on the records selected by the criteria, are executed.

It is possible to include only certain edit checks to be executed by naming them in EDIT elements within the CRITERIA element. Records and fields are still traversed in the same manner but only those edit checks that match one of the included names are executed. This can be useful, for example, when a new edit check is introduced and requires testing.

<CRITERIA>
    <PLATE include="5" />
    <EDIT>aeCoding</EDIT>
</CRITERIA>

The EDIT element is the only element that is allowed to repeat within CRITERIA. A single EDIT element may also list multiple comma or space separated edit checks by name in the element body.

<CRITERIA>
    <EDIT>aeCoding</EDIT>
    <EDIT>checkInit</EDIT>
    <EDIT>sigLookup</EDIT>
</CRITERIA>

<CRITERIA>
    <EDIT>aeCoding,checkInit,sigLookup</EDIT>
</CRITERIA>

When EDIT elements appear in the criteria, DFbatch optimizes the retrieval by automatically excluding from selection those plates that contain no variables which reference any of the named edit checks. For example, if the edit check named checkElig is referenced only by variables on plate 2, only records from plate 2 are selected even if PLATE does not appear in the criteria. However, if the PLATE element is present and includes a plate other than 2, the resulting set of selected records will be empty as there can be no records that reference both the checkElig edit check and are from a plate other than 2.

Edit checks are capable of changing (or assigning) data field values, adding queries to fields, adding and deleting missing page queries for records other than the current record, and generating information, warning, and error messages. This leads to three important categories:

In an interactive environment, the user has the ability to visually review and confirm or cancel each of these actions. In DFbatch this is not possible. Instead the batch control file must carefully state which actions will be allowed to proceed unattended and which ones will be canceled. This is specified in the ACTION element and its APPLY, LOG, and ODRF child elements.

<ACTION>
  <APPLY when="changes|all" level="1|2|3|4|5|6|7"
  which="none msg data qc"/>
  <LOG when="changes|all" which="none msg data qc" file="logfile_out.xml"
  mode="create|write" share="yes|no" history="yes|no" />
  <ODRF when="changes|all" which="none msg data qc" file="outfile.drf"
  mode="create|write" share="yes|no"/>
</ACTION>

6.3.1.4.1. The APPLY element

This element controls what categories of actions are applied back to the database.

	Use with care
	This is the only element that controls whether or not DFbatch can request irreversible database changes to be made and hence it must be used with care. If you are uncertain about the behavior of a batch control file or the edit checks that it may execute, use <APPLY which="none"/>.

The attributes of APPLY have the following semantics:

• when.  This attribute must have a value of all or changes. If the value is all, every selected record is sent back to the database as an update, even if the record contains no differences from its previous state. If the value is changes, only those records that contain changes are sent back to the database for update.

• level.  This attribute must be a single integer in the range of legal validation levels, 1 through 7 inclusive. It indicates the validation level to assign to records when they are sent back to the database for update. This causes DFbatch to behave in a manner equivalent to Validate mode in DFexplore. If this attribute is not specified, the validation level of any record sent back for update is not changed. This is equivalent to Edit mode in DFexplore.

• which.  This attribute has a value that is one or more space delimited categorical keywords from the list: none, data, msg, qc. data indicates that all records with changed data fields should be updated back to the database. qc indicates that all queries added via dfaddqc() or dfaddmpqc(), modified by dfeditqc() or deleted via dfdelmpqc(), should update the database. msg is intended for logging purposes only and is present here for symmetry only. none indicates that no changes to data or queries should be updated back to the database. This is the recommended attribute value for this attribute in the context of APPLY.

For the current implementation of DFbatch, the input file must contain only the elements and attributes that have been described and are valid for the document type definition. The presence of unknown elements or attributes will cause reading of the batch to fail, resulting in the batch not being processed. This will likely change in a future release.

For additional descriptions of each element and the attributes (and their purpose) that are valid for each element, consult BATCHLIST Element Reference.

DFbatch is ideally a batch process, run during off-hours. In the UNIX environment this is easily accomplished with the cron program. To use cron, simple include the needed DFbatch command-line equivalents in the crontab file. For additional information on using cron, refer to the UNIX man pages.

0 23 * * * /opt/dfdiscover/bin/DFbatch -S idemo44.datafax.com -U datafax -C passwd 254 -i test_in.xml

	DFbatch exit status
	When executing DFbatch several times with a sequence of control files (as might be done in a shell script), it is recommended to check the exit status of each execution before continuing with the next. Refer to DFbatch exit status.

Recall that the root document element, BATCHLIST, can contain one or more nested BATCH elements. The option -b batchname(s) can be used to select for processing specific batches by their name. For example, the command:

% DFbatch -S idemo44.datafax.com -U datafax -C passwd -b simple 254 -i test_in.xml

processes only the batch named simple from the control file, independent of how many other batches are defined. If the control file defines only the batch named simple then execution of DFbatch with and without -b simple are equivalent. However, if the control file defines three batches named simple, hard, and, difficult, it would be possible to process only a subset of the defined batches with a command like

% DFbatch -S idemo44.datafax.com -U datafax -C passwd -b "simple difficult" 254 -i another_in.xml

which would process the two batches simple and difficult while skipping the batch named hard.

When multiple batches appear in the control file, and multiple batches are processed, default ordering specifies that the batches be processed in the order that they appear in the file. Using -b it is possible to alter the processing order at invocation time without altering the control file.

Consider the following skeleton of a control file.

<?xml version="1.0"?>
<BATCHLIST>
        <BATCH name="b1">
                ...
        </BATCH>
        <BATCH name="b2">
                ...
        </BATCH>
        <BATCH name="b3">
                ...
        </BATCH>
</BATCHLIST>

Invoked in the default fashion,

% DFbatch -S idemo44.datafax.com -U datafax -C passwd 254 -i simple_in.xml

DFbatch will process batches b1, then b2, and finally b3. Order of processing can be altered on an ad-hoc basis through use of -b, as in the following example that processes batch b3 first, and then b1, skipping b2:

% DFbatch -S idemo44.datafax.com -U datafax -C passwd -b "b3 b1" 254 -i simple_in.xml

	Note
	If non-default ordering is commonly achieved with this method, it is recommended that the control file be edited to permanently re-order the batches.

The log information recorded by a batch execution is also in XML format. This makes it amenable to further processing, filtering, or transformation (also called styling). Of course, post-processing is only meaningful if the control file uses a LOG element and the value of the which attribute is not none; otherwise, there is no log information available to post-process.

For this release, the only post-processing that is supported by DFbatch is transformation of the log output via XSL. This is accomplished with one or more XSL style sheets that are able to transform the log information into HTML without affecting the log file itself. HTML is the most common output format but is certainly not the only one. An XSL transformation could just as easily create another XML file, plain text output, or even PDF. The simplest method for specifying post-processing is at DFbatch execution time with the command:

% DFbatch -p xsl study control_file > htmlfile

This command processes, in the same manner as already described, the batch in control_file in the context of study study. When the processing completes the log information is immediately post-processed with the default XSL transformation to create an HTML view of the log information that is stored to htmlfile. Note that the log file is not changed which allows post-processing to re-occur at a future time with the same XSL transformation, or a different XSL transformation to achieve different views of the same log. This post-processing of XML via XSL (or other) transformation is the key ingredient that allows DFbatch to separate content creation from presentation.

DFbatch comes with a default file for XSL transformation that is stored in /opt/dfdiscover/lib/xsl/batchlog.xsl and is referenced by the file /opt/dfdiscover/lib/stylesheets.xml. It is possible to create other XSL transformations (and future versions of DFbatch will be available with more) and incorporate them into /opt/dfdiscover/lib/stylesheets.xml. In such a case, it is possible to post-process the log information with an alternate, non-default, XSL transformation using the command:

% DFbatch -p XSL=name study control_file > htmlfile

which applies the XSL transformation named name to the log file, instead of the default transformation. Note that the case of the option is important: -p xsl requests the default transformation, -p XSL=name requests a specific transformation.

So far, we have only seen how to post-process a log file at the time of DFbatch execution. This unfortunately does not de-couple log creation from presentation as DFbatch must be run again to re-do the post-processing. De-coupling log creation from presentation requires an additional program, DFstyle.

The DFstyle program can be invoked at any time to apply a transformation to an XML file (not just a batch log file). DFstyle is invoked with either:

% DFstyle -p xsl XML file > htmlfile

which applies the default transformation, or

% DFstyle -p XSL=name XML file > htmlfile

which applies the transformation named by name. Notice that XML file can be any XML file, although in the context of DFbatch, it is an existing log file (not control file) from a previous execution. Also notice that the study number is not supplied to DFstyle - it does need the context of the study number to transform the log file.

Using DFstyle in conjunction with DFbatch, it is now possible to independently execute a batch to create a log, and then transform that log information into another format for viewing, thus achieving the de-coupling of content creation from presentation.

% DFbatch -S idemo44.datafax.com -U datafax -C passwd -p xsl 254 -i /opt/studies/val254/batch/example_in.xml \
-o /opt/studies/val254/batch/htmlfile

% DFbatch -S idemo44.datafax.com -U username -C passwd 254 -i /opt/studies/val254/batch/example_in.xml
% DFstyle -p xsl example_out.xml > htmlfile

If multiple XSL transformations are defined, it also possible to create multiple presentations of the same log file.

% DFbatch -S idemo44.datafax.com -U datafax -C passwd -p xsl 254 -i /opt/studies/val254/batch/example_in.xml \
-o /opt/studies/val254/batch/htmlfile1           (1)
% DFstyle -p XSL=secondtransform example_out.xml > htmlfile2 (2)
% DFstyle -p XSL=thirdtransform example_out.xml > htmlfile3  (3)

Provides a quick synopsis of the element. The content of the synopsis varies according to the nature of the element, but may include any or all of the following sections:

Describes the semantics of the element in detail. Typically contains the following sub-sections:

Provides examples of proper usage for the element.