DFimport.rpc

DFimport.rpc
Prev	3.4. Alphabetical Listing	Next

DFimport.rpc — Import database records to a study database from an ASCII text file

Synopsis

DFimport.rpc [-v] [-N] [-R] [ [-n] | [-q] | [-c] ] { [-a] | [-m] | [-r] } [-s] {#} {string}

Description

DFimport.rpc imports records from the input file into a DFdiscover study database. To use DFimport.rpc users must have permission to import data, defined by the study administrator in role permissions. To import new records (using the -a or -m options) users must have create permission and to replace existing records (using the -r or -m options) users must have modify permission. Remember that create and modify permission may be specified by level, site, subject, assessment and plate. DFimport.rpc will reject any records it cannot import with an error message indicating that the user does not have the necessary permissions.

When reading records from the input file, each record must be delimited by a newline ('\n') character. Additionally, each record can be at most 4095 characters in length. If 4095 characters are read without encountering a newline, the record is truncated at 4095 characters. It will subsequently fail record checking and be rejected.

DFimport.rpc may be used in a shell script stored in the study reports directory and run in the DFexplore Reports View, or in a shell script stored in the study ecbin directory and run dfexecute in an edit check. In these cases the permissions of the DFexplore user running the report or edit check will be applied. Thus the behavior and output may differ depending on the permissions of the user who runs the script.

If the same script is run from the UNIX shell the permissions associated with the UNIX login name apply. Thus, if a user has both UNIX and DFdiscover login accounts, and these accounts have different permissions, the user may get different results when they run the script in the UNIX and DFexplore environments.

DFexplore uses the environment variable DFUSER to implement user permissions. It is possible to set and export this variable in the shell script to fix the permissions to those of any specified user, thus rendering the results constant for all users despite their specific DFexplore permissions.

Input records can be new data records that need to be reviewed in Fax View, new or previously entered data records for direct entry to the study database, or metadata records (data queries or reasons). It is not possible to mix these different record types in a single import operation. The type of records being imported is specified using one of the following options:

-n : new records to be added to the new fax queue
-q : queries for specified data fields
-c : reasons for specified data fields

If none of these options is specified, the records must be new or replacement data records to be imported into the study database according to the plate number specified in the fifth field of each data record.

In addition to the record type options, there are 3 import mode options (add, replace and merge) to specify whether records are new and being added to the database (-a), replacement records (-r), or a combination requiring a merge operation during which new records are added to the database and existing records are replaced (-m).

To locate existing records in the database, DFimport.rpc searches for a record with matching keys (id, visit, plate) and image ID. This will always result in the identification of a record uniquely, if it exists, independent of whether the record has primary or secondary status. Dependent upon the record's presence in the database, the requested import mode will result in either an 'Add' or 'Replace' operation as shown in Table 3.9, “Deriving operations from import mode”.

Table 3.9. Deriving operations from import mode

Import mode	Exists	Does not exist
`-a`	**error	Add
`-r`	Replace	**error
`-m`	Replace	Add

If the operation is not in error, then the primary or secondary status of the import record is considered subject to the following rules:

Table 3.10. Add operation

Import record status	Existing record status	Ending State
primary	none	imported record becomes primary
secondary	none	**error
primary	primary	existing primary becomes secondary; imported record becomes primary
secondary	primary	imported record is added as secondary
primary	missing	**error

Table 3.11. Replace operation

Import record status	Existing record status	Ending State
primary	secondary	**error
secondary	secondary	imported record replaces existing
primary	primary	imported record replaces existing
secondary	primary	**error

Imported records are verified against the study data dictionary if -v is given. Without this option, imported records are only verified for the correct number of fields.

Formatting Records for Import

	Formatting Records for Import
Records must be correctly formatted for the DFdiscover plate to which they will be imported. The standard field delimiter, `\|`, must appear between data fields. The first 7 fields must include valid values for: status, level, image ID, study, plate, visit and subject ID respectively, and the last 3 fields must have valid values for: status, creation and modification. The only exception is for data records being imported to the new fax queue (using the `-n` option), for which the subject ID and visit keys may be blank. The third field, image ID, must be unique for all data records in the study database. This field contains the image identifier if the record has an associated image file, or a raw record identifier if there is no image. When importing new records that have no image (e.g. lab data from an external source) this field should contain a placeholder image ID, `0000/0000000`. Specifying `-R`, will instruct DFimport.rpc to generate a new raw record identifier and insert it in the record, replacing the placeholder. When importing metadata records (queries and reasons) the image ID field must also contain `0000/0000000`, but this value is not converted to a unique identifier by DFimport.rpc (`0000/0000000` is valid in metadata records). A trailing delimiter is required at the end of all data records but must not be present on metadata records, i.e. queries (plate 511) and reasons (plate 510). Thus, care should be taken not to use the `-p` option, which ensures there is a trailing pipe on exported records, when using DFexportrpc; to export queries and reasons if the intention is to re-import them into the database.

Records must be correctly formatted for the DFdiscover plate to which they will be imported. The standard field delimiter, |, must appear between data fields. The first 7 fields must include valid values for: status, level, image ID, study, plate, visit and subject ID respectively, and the last 3 fields must have valid values for: status, creation and modification. The only exception is for data records being imported to the new fax queue (using the -n option), for which the subject ID and visit keys may be blank.

The third field, image ID, must be unique for all data records in the study database. This field contains the image identifier if the record has an associated image file, or a raw record identifier if there is no image. When importing new records that have no image (e.g. lab data from an external source) this field should contain a placeholder image ID, 0000/0000000. Specifying -R, will instruct DFimport.rpc to generate a new raw record identifier and insert it in the record, replacing the placeholder.

When importing metadata records (queries and reasons) the image ID field must also contain 0000/0000000, but this value is not converted to a unique identifier by DFimport.rpc (0000/0000000 is valid in metadata records).

A trailing delimiter is required at the end of all data records but must not be present on metadata records, i.e. queries (plate 511) and reasons (plate 510). Thus, care should be taken not to use the -p option, which ensures there is a trailing pipe on exported records, when using DFexportrpc; to export queries and reasons if the intention is to re-import them into the database.

If the validation level of an imported record is higher than the user's maximum write level, it is adjusted down to the maximum, a warning message is generated, and the record import proceeds.

If an input record has a # or a newline in the first column position, DFimport.rpc treats the entire record as a comment and skips over it.

If DFimport.rpc completes without error, it echoes the number of records imported and the number of warning messages and exits with status 0. If DFimport.rpc encounters errors, which are unrelated to the records being imported, it exits with a message to stderr and a positive exit status. If DFimport.rpc encounters errors with the records it is importing, it writes each offending record to stdout, a problem description to stderr, and increments a counter of failed records. Upon exit, DFimport.rpc sets its exit status to the failed record counter. If all records are successfully imported, the exit status is 0.

	Note
DFimport.rpc does not complete successfully if the study server has been disabled or put into read-only access mode. If the study server is in exclusive access mode, only the user who put it into this mode can import data. If the study is in restricted access mode, only study and DFdiscover administrators are able to import data.

Note

DFimport.rpc does not complete successfully if the study server has been disabled or put into read-only access mode.
If the study server is in exclusive access mode, only the user who put it into this mode can import data.
If the study is in restricted access mode, only study and DFdiscover administrators are able to import data.

Options

-v	Verify that the imported records match the database record format as specified in the study data dictionary. It is highly recommended that this argument be used under most circumstances. Without this option, minimal record checking is performed on the number of fields (must be at least 7), the record status, and the record validation level. The verification performed by `-v` is equivalent to the consistency checking performed by the DFdiscover utility program DFcmpSchema when run with the `-v` option.
-N	Test that the data records verify correctly but do not import them.
-R	Convert `0000/0000000` in field 3 to a new raw record identifier before importing data records to the study database. This option cannot be used when importing metadata records (queries and reasons).
-n, -q, -c	Imported record type: if none of these options is specified the input file must contain data records to be imported into the study database according to the plate number field in each data record. Specifying: `-n`, New: add records to the new record queue, `DFin.dat`. When importing to `DFin.dat`, the import mode must be `-a` or `-r`. `-q`, queries: add records to the Query database, `DFqc.dat`. When importing to `DFqc.dat`, the import mode must be `-a` or `-r`. `-c`, Reason for change notes: add records to the reasons database, `DFreason.dat`. When importing to `DFreason.dat`, the import mode must be `-a` or `-r`.
-a, -m, -r	Import mode (required) specifies one of three possible actions: `-a`, Add: fail if a record with matching keys and image ID already exists in database. `-m`, Merge: if a primary record with matching keys exists in the database, replace it if the image ID is the same, make it secondary if the image ID is different, otherwise proceed as if adding. `-r`, Replace: fail if a record with matching keys and image ID does not exist in the database, otherwise replace it.
-s	Skip plate arrival triggers. This option can only be specified in conjunction with the `-n` (new record) import option. By default, plate arrival triggers (defined in Plate View in DFsetup) are executed when data records are imported to the new record queue. The combination of `-n -s` options allows new records to be imported without executing plate arrival triggers. Plate arrival triggers are only executed if the record import is successful. If a trigger fails, a warning message is written to the DFdiscover error log.
`#`	The study database number into which the records are to be imported (required).
string	The input file containing the records to be imported (required), or `-`, in which case the input records are read from standard input.

Exit Status

DFimport.rpc exits with one of the following statuses:

0	The command was successful.
24	The user does not have the necessary permission to execute the command.
36	The required command-line arguments were not present or were incorrectly specified.
> 0	The command failed because the study schema or the input file could not be read, the database server could not be contacted, or communication with the database server failed.

Limitations

DFimport.rpc attempts to enforce database integrity as best it can. As a result, the following constraints are enforced:

DFimport.rpc does not allow the import of type 21 (missing page), 22 (overdue visit) and 23 (edit check missing page) queries for which there is a primary record (missing, final, incomplete) in the database. Attempts to do this will result in a message that the record already exists.
DFimport.rpc will delete any corresponding type 21 (missing page), 22 (overdue visit) and 23 (edit check missing page) queries when a primary record (missing, final, incomplete) is imported into the database.
DFimport.rpc does not allow the import of new query records with category codes that are not defined in the Query Category Map in DFsetup. However, it does allow modification or deletion of records with undefined category codes.
DFimport.rpc can be used to import data records that contain fields from an e-signature module but each of the e-signature fields will be blanked during the import. It is not possible to programmatically add/replace data records that have pre-filled e-signature fields. Such records must be reviewed in DFexplore after the import so that they can be e-signed.
DFimport.rpc does not allow the import of queries for which the primary record is locked. Attempts to do this will result in a message that the record is currently in use.
DFimport.rpc does not allow the import of data values for choice and check fields that do not exactly match one of the codes defined for those fields in the study schema.
DFimport.rpc will not import secondary records with a raw or zeros image ID (e.g. 0922R00023001 or 0000/0000000). The only supported function for secondary records is to reference additional images attached to the primary data record. Thus secondary records must have an image ID that points to an image file.

However, there are situations where a knowledgeable user may want to import records out of order, for example, so that referential integrity is initially violated, and then subsequently restored. The following operations are permitted as they allow a knowledgeable user to fully utilize DFimport.rpc.

DFimport.rpc allows the import of queries for which there is no primary record in the database. This is to allow an out-of-sequence import of queries followed by their data records in a subsequent import. The correct sequence of steps is to first import the primary records followed by the queries.
DFimport.rpc allows the primary record to be imported with a delete status, effectively deleting the primary record and all secondary records, queries and reasons associated with it. If the primary record is deleted, its secondary records and metadata will also be deleted to avoid having orphaned secondary, query and reason records in the database. During deletion, the record currently in the database is compared with the record being imported. All fields (other than status) must match for the operation to succeed. The number of fields can be different than the current definition for a given plate. This allows for the deletion of possibly corrupt records with field counts that differ from the current plate definition by exporting the record, changing its status to delete, and importing the record in question.
DFimport.rpc allows query and reason records to be imported with a delete status, effectively deleting the queries and reasons. All fields (other than status and validation level) must match for the operation to succeed. The deletion of queries and reasons does not delete the primary record to which they are attached.

Prev	Up	Next
DFimageio	Home	DFlistplates.rpc