CDS/ISIS manual appendix F, G and H
This is partial scan of CDS/ISIS manual (appendix F, G and H, pages
257-272) which is than converted to text using OCR and proofread.
However, there might be mistakes, and any corrections sent to
dpavlin@rot13.org
will be greatly appreciated.
This digital version is made because current version available in ditial form doesn't contain details about CDS/ISIS file format and was essential in making Biblio::Isis module.
This extract of manual has been produced in compliance with section (d) of WinIsis LICENCE for receiving institution/person which say:
The receiving institution/person may: (d) Print/reproduce the CDS/ISIS manuals or portions thereof, provided that such copies reproduce the copyright notice;
This section describes the various files of the CDS/ISIS system, the file naming conventions and the file extensions used for each type of file. All CDS/ISIS files have standard names as follows:
nnnnnn.eee
where:
nnnnnn
is the file name (all file names, except program names, are limited to a maximum of 6 characters)
.eee
is the file extension identifying a particular type of file.
Files marked with *
are ASCII files which you may display or print. The
other files are binary files.
System files are common to all CDS/ISIS users and include the various executable programs as well as system menus, worksheets and message files provided by Unesco as well as additional ones which you may create.
The name of the program file, as supplied by Unesco is
ISIS.EXE
Depending on the release and/or target computer, there may also be one
or more overlay files. These, if present, have the extension OVL
.
Check the contents of your system diskettes or tape to see whether
overlay files are present.
All system menus and worksheets have the file extension FMT and the names are built as follows:
pctnnn.FMT
where:
p
is the page number (A for the first page, B for the second, etc.)
c
is the language code (e.g. E for English), which must be one of those provided for in the language selection menu xXLNG.
t
is X for menus and Y for system worksheets
nnn
is a unique identifier
For example the full name of the English version of the menu xXGEN is
AEXGEN.FMT
.
The page number is transparent to the CDS/ISIS user. Like the file extension the page number is automatically provided by the system. Therefore when a CDS/ISIS program prompts you to enter a menu or worksheet name you must not include the page number. Furthermore as file names are restricted to 6 characters, menus and worksheets names may not be longer than 5 characters.
System menus and worksheets may only have one page.
The language code is mandatory for system menus and standard system worksheets. For example if you want to link a HELP menu to the system menu EXGEN, its name must begin with the letter E.
The X convention is only enforced for standard system menus. It is a good practice, however, to use the same convention for menus that you create, and to avoid creating worksheets (including data entry worksheets) with X in this position, that is with names like xXxxx.
Furthermore, if a data base name contains X or Y in the second
position, then the corresponding data entry worksheets will be created
in the system worksheet directory (parameter 2 of SYSPAR.PAR
) rather
then the data base directory. Although this will not prevent normal
operation of the data base, it is not recommended.
System messages and prompts are stored in standard CDS/ISIS data bases. All corresponding data base files (see below) are required when updating a message file, but only the Master file is used to display messages.
There must be a message data base for each language supported through the language selection menu xXLNG.
The data base name assigned to message data bases is xMSG (where x is the language code).
System tables are used by CDS/ISIS to define character sets. Two are required at present:
ISISUC.TAB
*defines lower to upper-case translation
ISISAC.TAB
*defines the alphabetic characters.
Certain CDS/ISIS print functions do not send the output directly to the
printer but store it on a disk file from which you may then print it at
a convenient time. These files have all the file extension LST
and
are reused each time the corresponding function is executed.
In addition CDS/ISIS creates temporary work files which are normally
automatically discarded at the end of the session. If the session
terminates abnormally, however, they will not be deleted. A case of
abnormal termination would be a power failure while you are using a
CDS/ISIS program. Also these files, however, are reused each time,
so that you do not normally need to delete them manually. Work files
all have the extension TMP
.
The print and work files created by CDS/ISIS are given below:
IFLIST.LST
*Inverted file listing file (produced by ISISINV)
WSLIST.LST
*Worksheet/menu listing file (produced by ISISUTL)
xMSG.LST
*System messages listing file (produced by ISISUTL)
x.LST
*Printed output (produced by ISISPRT when printing no print file name is supplied)
SORTIO.TMP
Sort work file 1
SORTII.TMP
Sort work file 2
SORTI2.TMP
Sort work file 3
SORTI3.TMP
Sort work file 4
SORT20.TMP
Sort work file 5
SORT2I.TMP
Sort work file 6
SORT22.TMP
Sort work file 7
SORT23.TMP
Sort work file 8
TRACE.TMP
*Trace file created by certain programs
ATSF.TMP
Temporary storage for hit lists created during retrieval
ATSQ.TMP
Temporary storage for search expressions
mandatory files, which must always be present. These are normally established when the data base is defined by means of the ISISDEF services and should never be deleted;
auxiliary files created by the system whenever certain functions are performed. These can periodically be deleted when they are no longer needed.
user files created by the data base user (such as display formats), which are fully under the user's responsibility.
Each data base consists of a number of physically distinct files as indicated below. There are three categories of data base files:
In the following description xxxxxx
is the 1-6 character data base
name.
xxxxxx.FDT
*Field Definition Table
xxxxxx.FST
*Field Select Table for Inverted file
xxxxxx.FMT
*Default data entry worksheet (where p is the page number).
Note that the data base name is truncated to 5 characters if necessary
xxxxxx.PFT
*Default display format
xxxxxx.MST
Master file
xxxxxx.XRF
Crossreference file (Master file index)
xxxxxx.CNT
B*tree (search term dictionary) control file
xxxxxx.N01
B*tree Nodes (for terms up to 10 characters long)
xxxxxx.L01
B*tree Leafs (for terms up to 10 characters long)
xxxxxx.N02
B*tree Nodes (for terms longer than 10 characters)
xxxxxx.L02
B*tree Leafs (for terms longer than 10 characters)
xxxxxx.IFP
Inverted file postings
xxxxxx.ANY
*ANY file
xxxxx.STW
*Stopword file used during inverted file generation
xxxxxx.LN1
*Unsorted Link file (short terms)
xxxxxx.LN2
*Unsorted Link file (long terms)
xxxxxx.LKl
*Sorted Link file (short terms)
xxxxxx.LK2
*Sorted Link file (long terms)
xxxxxx.BKP
Master file backup
xxxxxx.XHF
Hit file index
xxxxxx.HIT
Hit file
xxxxxx.SRT
*Sort convertion table (see "Uppercase conversion table (1SISUC.TAB)" on page 227)
yyyyyy.FST
*Field Select tables used for sorting
yyyyyy.PFT
*Additional display formats
yyyyyy.FMT
*Additional data entry worksheets
yyyyyy.STW
*Additional stopword files
yyyyyy.SAV
Save files created during retrieval
The name of user files is fully under user control. However, in order
to avoid possible name conflicts it is advisable to establish some
standard conventions to be followed by all CDS/ISIS users at a given
site, such as for example to define yyyyyy
as follows:
xxxyyy
where:
xxx
is a data base identifier (which could be the first three letters of the data base name if no two data bases names are allowed to begin with the same three letters)
yyy
a user chosen name.
The Master record is a variable length record consisting of three sections: a fixed length leader; a directory; and the variable length data fields.
The leader consists of the following 7 integers (fields marked with * are 31-bit signed integers):
MFN
*Master file number
MFRL
Record length (always an even number)
MFBWB
*Backward pointer - Block number
MFBWP
Backward pointer - Offset
BASE
Offset to variable fields (this is the combined length of the Leader and Directory part of the record, in bytes)
NVF
Number of fields in the record (i.e. number of directory entries)
STATUS
Logical deletion indicator (0=record active; 1=record marked for deletion)
MFBWB
and MFBWP
are initially set to 0 when the record is
created. They are subsequently updated each time the record itself is
updated (see below).
The directory is a table indicating the record contents. There is one directory entry for each field present in, the record (i.e. the directory has exactly NVF entries). Each directory entry consists of 3 integers:
TAG
Field Tag
POS
Offset to first character position of field in the variable field
section (the first field has POS=0
)
LEN
Field length in bytes
The total directory length in bytes is therefore 6*NVF
; the BASE
field
in the leader is always: 18+6*NVF
.
This section contains the data fields (in the order indicated by the directory). Data fields are placed one after the other, with no separating characters.
The first record in the Master file is a control record which the
system maintains automatically. This is never accessible to the ISIS
user. Its contents are as follows (fields marked with *
are 31-bit
signed integers):
CTLMFN
*always 0
NXTMFN
*MFN to be assigned to the next record created in the data base
NXTMFB
*Last block number allocated to the Master file (first block is 1)
NXTMFP
Offset to next available position in last block
MFTYPE
always 0 for user data base file (1 for system message files)
(the last four fields are used for statistics during backup/restore).
The Master file records are stored consecutively, one after the other,
each record occupying exactly MFRL
bytes. The file is stored as
physical blocks of 512 bytes. A record may begin at any word boundary
between 0-498 (no record begins between 500-510) and may span over two
or more blocks.
As the Master file is created and/or updated, the system maintains an
index indicating the position of each record. The index is stored in
the Crossreference file (.XRF
)
The XRF
file is organized as a table of pointers to the Master file.
The first pointer corresponds to MFN 1, the second to MFN 2, etc.
Each pointer consists of two fields:
RECCNT
*MFCXX1
*MFCXX2
*MFCXX3
*XRFMFB
(21 bits) Block number of Master file block containing the record
XRFMFP
(11 bits) Offset in block of first character position of Master record (first block position is 0)
which are stored in a 31-bit signed integer (4 bytes) as follows:
pointer = XRFMFB * 2048 + XRFMFP
(giving therefore a maximum Master file size of 500 Megabytes).
Each block of the XRF
file is 512 bytes and contains 127 pointers. The
first field in each block (XRFPOS
) is a 31-bit signed integer whose
absolute value is the XRF
block number. A negative XRFPOS
indicates
the last block.
Deleted records are indicated as follows:
XRFMFB < 0
and XRFMFP > 0
logically deleted record (in this case ABS(XRFMFB)
is the correct block
pointer and XRFMFP
is the offset of the record, which can therefore
still be retrieved)
XRFMFB = -1
and XRFMFP = 0
physically deleted record
XRFMFB = 0
and XRFMFP = 0
inexistent record (all records beyond the highest MFN
assigned in the
data base)
New records are always added at the end of the Master file, at the
position indicated by the fields NXTMFB
/NXTMFP
in the Master file
control record. The MFN
to be assigned is also obtained from the field
NXTMFN
in the control record.
After adding the record, NXTMFN
is increased by 1 and NXTMFB
/NXTMFP
are updated to point to the next available position. In addition a new
pointer is created in the XRF
file and the XRFMFP
field corresponding
to the record is increased by 1024 to indicate that this is a new
record to be inverted (after the inversion of the record 1024 is
subtracted from XRFMFP
).
Whenever you update a record (i.e., you call it in data entry and exit with option X from the editor) the system writes the record back to the Master file. Where it is written depends on the status of the record when it was initially read.
This condition is indicated by the following:
On XRF
XRFMFP < 512
and
On MST
MFBWB = 0
and MFBWP = 0
In this case, the record is always rewritten at the end of the Master
file (as if it were a new record) as indicated by NXTMFB
/NXTMFP
in the
control record. In the new version of the record MFBWB
/MFBWP
are set to
point to the old version of the record, while in the XRF
file the
pointer points to the new version. In addition 512 is added to XRFMFP
to indicate that an inverted file update is pending. When the inverted
file is updated, the old version of the record is used to determine the
postings to be deleted and the new version is used to add the new
postings. After the update of the Inverted file, 512 is subtracted from
XRFMFP
, and MFBWB
/MFBWP
are reset to 0.
This condition is indicated by the following:
On XRF
XRFMFP > 512
and
On MST
MFBWB > 0
In this case MFBWB
/MFBWP
point to the version of the record which is
currently reflected in the Inverted file. If possible, i.e. if the
record length was not increased, the record is written back at its
original location, otherwise it is written at the end of the file. In
both cases, MFBWB
/MFBWP
are not changed.
Record deletion is treated as an update, with the following additional markings:
On XRF
XRFMFB
is negative
On MST
STATUS
is set to 1
As indicated above, as Master file records are updated the MST
file
grows in size and there will be lost space in the file which cannot be
used. The reorganization facilities allow this space to be reclaimed by
recompacting the file.
During the backup phase a Master file backup file is created (.BKP
).
The structure and format of this file is the same as the Master file
(.MST
), except that a Crossreference file is not required as all the
records are adjacent. Records marked for deletion are not backed up.
Because only the latest copy of each record is backed up, the system
does not allow you to perform a backup whenever an Inverted file update
is pending for one or more records.
During the restore phase the backup file is read sequentially and the
program recreates the MST
and XRF
file. At this point alt records which
were marked for logical deletion (before the backup) are now marked as
physically deleted (by setting XRFMFB = -1
and XRFMFP = 0
.
Deleted records are detected by checking holes in the MFN
numbering.
The CDS/ISIS Inverted file consists of six physical files, five of
which contain the dictionary of searchable terms (organized as a
B*tree) and the sixth contains the list of postings associated with
each term. In order to optimize disk storage, two separate B*trees are
maintained, one for terms of up to 10 characters (stored in files
.N01
/.L01
) and one for terms longer than 10 characters, up to a maximum
of 30 characters (stored in files .N02
/.L02
). The file CNT
contains
control fields for both B*trees. In each B*tree the file .N0x
contains
the nodes of the tree and the .L0x
file contains the leafs. The leaf
records point to the postings file .IFP
.
The relationship between the various files is schematically represented in Figure 67.
The physical relationship between these six files is a
pointer, which represents the relative address of the record being
pointed to. A relative address is the ordinal record number of a record
in a given file (i.e. the first record is record number 1, the second
is record number 2, etc.). The file .CNT
points to the file .N0x
,
.N0x
points to .L0x
, and .L0x
points to .IFP
. Because the
.IFP
is a packed file, the pointer from .L0x
to .IFP
has two
components: the block number and the offset within the block, each expressed
as an integer.
.CNT
fileThis file contain two 26-byte fixed length records (one for each B*tree) each containing 10 integers as follows (fields marked with * are 31-bit signed integers):
IDTYPE
B*tree type (1 for .N01
/.L01
, 2 for .N02
/.L02
)
ORDN
Nodes order (each .N0x
record contains at most 2*ORDN
keys)
ORDF
Leafs order (each .L0x
record contains at most 2*ORDF
keys)
N
Number of memory buffers allocated for nodes
K
Number of buffers allocated to lst level index (K < N
)
LIV
Current number of index levels
POSRX
*Pointer to Root record in .N0x
NMAXPOS
*Next available position in .N0x
file
FMAXPOS
*Next available position in .L0x
file
ABNORMAL
Formal B*tree normality indicator (0 if B*tree is abnormal, 1 if B*tree
is normal). A B*tree is abnormal if the nodes file .N0x
contains only
the Root.
ORDN
, ORDF
, N
and K
are fixed for a given generated system.
Currently these values are set as follows:
ORDN = 5
; ORDF = 5
; N = 15
; K = 5
for both B*trees
+--------------+ | Root address | +-------|------+ | .CNT file | ------------- | .N0x file +-----------V--------+ | Key1 Key2 ... Keyn | Root +---|-------------|--+ | | +-----+ +------+ | | +----------V----------+ +---------V----------+ 1st level | Key1 Key2 ... Keyn | ... | Key1 Key2 ... Keyn | index +--|------------------+ +-----------------|--+ | : : +-------+ | | +--V------------------+ +---------V----------+ last level | Key1 Key2 ... Keyn | ... | Key1 Key2 ... Keyn | index +---------|-----------+ +---------|----------+ | | | | ------------- | | .L0x file +---------V-----------+ +---------V----------+ | Key1 Key2 ... Keyn | ... | Key1 Key2 ... Keyn | +--|------------------+ +--------------------+ | | ------------- | .IPF file +--V----------------------------------+ | P1 P2 P3 ..................... Pn | +-------------------------------------+
Figure 67: Inverted file structure
The other values are set as required when the B*trees are generated.
.N0x
filesThese files contain the indexes) of the dictionary of searchable terms
(.N01
for terms shorter than 11 characters and .N02
for terms longer
than 10 characters). The .N0x
file records have the following format
(fields marked with * are 31-bit signed integers):
POS
*an integer indicating the relative record number (1 for the first record, 2 for the second record, etc.)
OCK
an integer indicating the number of active keys in the record
( 1 <= OCK <= 2*ORDN
)
IT
an integer indicating the type of B*tree (1 for .N01
, 2 for .N02
)
IDX
an array of ORDN
entries (OCK
of which are active), each having the
following format:
KEY
a fixed length character string of length .LEx
(LE1 =10
, LE2 = 30
)
PUNT
a pointer to the .N0x
record (if PUNT > 0
) or .L0x
record
(if PUNT < 0
) whose IDX(1).KEY = KEY
. PUNT = 0
indicates
an inactive entry. A positive PUNT
indicates a branch to a hierarchically
lower level index. The lowest level index (PUNT < 0
) points the leafs in
the .L0x
file.
.L0x
filesThese files contain the full dictionary of searchable terms (.L01
for
terms shorter than 11 characters and .L02
for terms longer than 10
characters). The .L0x
file records have the following format (fields
marked with *
are 31-bit signed integers):
POS
*an integer indicating the relative record number (1 for the first record, 2 for the second record, etc.)
OCK
an integer indicating the number of active keys in the record
(1 < OCK <= 2*ORDF
)
IT
an integer indicating the type of B*tree (1 for .N01
, 2 for .N02
)
PS
*is the immediate successor of IDX[OCK].KEY
in this record (this is used
to speed up sequential access to the file)
IDX
an array of ORDN
entries (OCK
of which are active), each having the
following format:
KEY
a fixed length character string of length LEx
(LE1=10
, LE2=30
)
INFO
a pointer to the .IFP
record where the list of postings associated with
KEY
begins. This pointer consists of two 31-bit signed integers as
follows:
INFO[1]
*relative block number in .IFP
INFO[2]
*offset (word number relative to 0) to postings list
.IFP
fileThis file contains the list of postings for each dictionary term. Each list of postings has the format indicated below. The file is structured in blocks of 512 characters, where (for an initially loaded and compacted file) the lists of postings for each term are adjacent, except as noted below.
The general format of each block is:
IFPBLK
a 31-bit signed integer indicating the Block number of this block (blocks are numbered from 1)
IFPREC
An array of 127 31-bit signed integers
IFPREC[1]
and FPREC[2]
of the first block are a pointer to the
next available position in the .IFP
file.
Pointers from .L0x
to .IFP
and pointers within .IFP
consist of two
31-bit signed integers: the first integer is a block number, and the
second integer is a word offset in IFPREC
(e.g. the offset to the
first word in IFPREC
is 0). The list of postings associated with the
first search term will therefore start at 1/0.
Each list of postings consists of a header (5 double-words) followed by the actual list of postings (8 bytes for each posting). The header has the following format (each field is a 31-bit signed integer):
IFPNXTB
*Pointer to next segment (Block number)
IFPNXTP
*Pointer to next segment (offset)
IFPTOTP
*Total number of postings (accurate only in first segment)
IFPSEGP
*Number of postings in this segment (IFPSEGP <= IFPTOTP
)
IFPSEGC
*Segment capacity (i.e. number of postings which can be stored in this segment)
Each posting is a 64-bit string partitioned as follows:
PMFN
(24 bits) Master file number
PTAG
(16 bits) Field identifier (assigned from the FST
)
POCC
(8 bits) Occurrence number
PCNT
(16 bits) Term sequence number in field
Each field is stored in a strict left-to-right sequence with leading zeros added if necessary to adjust the corresponding bit string to the right (this allows comparisons of two postings as character strings).
The list of postings is stored in ascending PMFN
/PTAG
/POCC
/PCNT
sequence. When the inverted file is loaded sequentially (e.g. after a
full inverted file generation with ISISINV), each list consists of one
or more adjacent segments. If IFPTOT <= 32768
then:
IFPNXTB/IFPNXTP = 0/0
and IFPTOT = IFPSEGP = IFPSEGC
.
As updates are performed, additional segments may be created whenever
new postings must be added. In this case a new segment with capacity
IFPTOTP
is created and linked to other segments (through the pointer
IFPNXTB
/IFPNXTP
) in such a way that the sequence
PMFN
/PTAG
/POCC
/PCNT
is maintained. Whenever such a split occurs
the postings of the segment where the new posting should have been inserted
are equally distributed between this segment and the newly created segment.
New segments are always written at the end of the file (which is maintained
in IFPREC[1]
/IFPREC[2]
of the first .IFP
block.
For example, assume that a new posting Px
has to be inserted between P2
and P3
in the following list:
+----------------------------+ | 0 0 5 5 5 | P1 P2 P3 P4 P5 | +----------------------------+
after the split (and assuming that the next available position in .IFP
is 3/4) the list of postings will consist of the following two segments:
+----------------------------+ | 3 4 5 3 5 | P2 P2 Px -- -- | +--|-------------------------+ | +--V-------------------------+ | 0 0 5 3 5 | P3 P4 P5 -- -- | +----------------------------+
In this situation, no new segment will be created until either segment becomes again full.
As mentioned above, the posting lists are normally stored one after the
other. However, in order to facilitate access to the .IFP
file the
segments are stored in such a way that:
the header and the first posting in each list (28 bytes) are never split between two blocks.
a posting is never split between two blocks; if there is not enough room in the current block the whole posting is stored in the next block.
UNESCO has developed and owns the intellectual property of the CDS/ISIS software (in whole or in part, including all files and documentation, from here on referred to as CDS/ISIS) for the storage and retrieval of information.
For complete text of licence visit http://www.unesco.org/isis/files/winisislicense.html.