Thoughts on CP Directory Management

From Kris' internal archive of documents written for customers...

(Note: this page is reformatted to match the limited subset of HTML permitted on this site -- the original file with Kris' original formatting is attached at the bottom of the page.)

The CP directory

The CP directory contains records to define Virtual Machines. Maybe the most important parts of the directory are the minidisks. MDISK cards define how DASDs are divided. Minidisks should normally not overlap each other, as this can result in dataloss. At the other hand, sometimes it is required to have access to a whole DASD (for example: ICKDSF and DIRECTXA), then we define a fullpack MDISK, in overlap with all other MDISKs on that volume (for example: MAINT 123 on the VM residence pack).

Object and Source format of the CP directory

The CP directory exists in two formats: the source format is 1 or a series of CMS files; the object format of the directory is what CP reads, it is stored on CP-owned packs on areas allocated as DRCT. The DIRECTXA command reads the source format and writes the object format on free sections in the DRCT area(s). The DIRECTORY record in the source directory defines on which virtual dasd the object directory will be written, usually the address is 123.

The DIAG 84 instruction allows a direct update of sections in the object directory. CP does not need to read the DRCT area(s) each time it consults the directory: it stores the object directory in an ESA address space.

Clustered source directory The source directory can be in clustered format, in which case there is a main CMS file that defines which other CMS files need to be included in order to compose the complete directory. This format was created to diminish the IOs required to manage the source directory. For example: adding a minidisk in a source directory of a million lines, requires the rewrite of this million lines; with the clustered format, only the main and one imbedded file must be rewritten.

On our largest system, the non-clustered directory contains 25000 records (490 4K blocks). The main file of clustered version has 2800 lines (54 4K blocks) and 16 imbedded files, each about 1600 records (32 4K blocks).

Dirmaint

Dirmaint is billable SW to manage the CP directory. Dirmaint manages the source format of this CP directory. Dirmaint R5 always uses a clustered source directory. The source directory is stored on DIRMAINT 1DF as USER DIRECT and CLSTnnnn CLUSTER. Daily -and when DIRM USER BACKUP or DIRM BACKUP is issued- DIRMAINT creates a non clustered copy of the directory as USER BACKUP on DIRMAINT 1DB. DIRMAINT uses both DIRECTXA and and DIAG 84 to update the object directory.

Mapping minidisks

As it is crucial to avoid inadvertent overlaps, various tools exist that read the directory and create a map of the MDISKs:

DIRMAP
Was part of IPF, later CUF, which were charged extra. Since z/VM standard in CMS. Disadvantages: no major ones, except that it does not support the clustered source directory. It ends with non-zero rc when it detect non-fullpack overlaps. It supports control files to exclude volumes etc.

DISKMAP
Free in CMS since many years. Disadvantages: reports fullpack minidisks as overlaps, hence inadvertent overlaps may remain unnoticed. When overlaps are found it still ends with rc=0. Does not know size of DASDs, hence it does not report free space at the end of volumes. Conclusion: forget this tool.

DIRM DIRMAP
Command of DIRMAINT. Disadvantages: reports fullpack minidisks as overlap; does not report gaps when a fullpack overlaps the whole pack, unless using the EXCLUDE option. Advantage: it can work with the clustered source directory.

DRM
Tool written by Kris Buelens, uses XEDIT and DIRMAP. Sometimes called "DIRMAINT for poor people", available on VM's download library. Besides these there is also DIRM USED and DIRM FREE, but they don't provide a nice-looking map.

Dummy minidisks

Most space used in VM is located on minidisks, CP itself though was never using minidisks to store/retrieve data. Consequently there is no technical requirement to define these CP areas as minidisks in the CP directory.

Since VM/ESA V2 R1, CP uses CMS minidisks (by default: MAINT CF1, CF2, and CF3) as place for the CP nucleus, SYSTEM CONFIG file, LOGON logos, and LOGON messages. One can use CP Q CPDISK to see which CMS minidisks CP currently uses. As these are normal minidisks, they must be defined in the CP directory and we don't discuss them further.

The other CP areas need some attention. It is common practice to define dummy minidisks in the directory to indicate that these areas are not free. Most CP areas are defined to CP in the ALLOCATION MAP, an area in cylinder 0 of CP-OWNED packs, written by ICKDSF. Here an overview of all CP areas:

• Warm Start and Checkpoint Start areas
  
  Both areas are defined in the SYSTEM CONFIG file. Usually one uses minidisks of USER $SYSWRM$ and USER $SYSCKP$ to protect them.
• Object Directory
  
  Defined to CP by allocation type DRCT. Usually minidisk(s) of USER $DIRECT$ protect it.
• Spool and Paging space
  
  Defined to CP by allocation type SPOL and PAGE. Usually minidisks of USER $SPOOL$ and USER $PAGE$ protect it.
• Space for Temporary Minidisks
  
  Defined to CP by allocation type TDSK. Usually minidisk(s) of USER $TDISK$ protect it.
• CSE area
  
  With VM's CSE function one can share minidisks amongst different VM systems (CSE does not bring concurrent R/W sharing for CMS minidisks). To make CSE work, CP needs on each shared volume a map to record which VM has minidisks on which cylinders. The place of this CSE map is defined to CP in the SYSTEM CONFIG file. You should protect these areas too; we use minidisk(s) of USER $CSE$ to protect them.
• Disk label, Alloc map, IPL records
  
  Last but not least, cylinder 0 contains the volume serial of the DASD. If you define a minidisk at cylinder 0, the volume serial of the minidisk and that of the real pack is the same. When the user of the minidisk changes the label, the real volser is changed too. Often this is not desired, and one defines a dummy minidisk to cover cylinder 0.
  
  For CP-owned packs, also the allocation area is in cylinder 0 and should be protected. The VM residence pack (and maybe other packs) have IPL records in cylinder 0 that should not be overwritten by some end-user. To protect cylinder 0, one usually uses USER $ALLOC$.

There is no mechanism to assure that the location of these dummy minidisks matches the actual location used by CP, this remains your responsibility. CP only protects some its areas: it refuses LINKs to a minidisk that overlaps with PAGE, SPOOL, or TDISK.

Conventions for minidisk addresses and fullpack overlays

As seen above, quite some minidisks are defined in the CP directory to protect space, or to have a convenient fullpack overlay. Some convention may be useful.

• Minidisk addresses
  
  The minidisk addresses of these dummy minidisks have no importance at all. We opted to use as minidisk address the real device address of the pack, which is helpful in some cases.
• Fullpack overlays
  
  The CP directory delivered with VM contains a few fullpack overlays: MAINT 123 (resident), MAINT 124, MAINT 125, SYSDUMP1 123 And sometimes even more. And, to make it easy to work with ICKDSF, or DDR, one may define extra fullpack minidisks.
  
  Having many fullpack minidisks at somewhat random addresses/users is not ideal:
  • When using minidisk based backups, you will have to exclude these fullpack minidisks from your backups. Otherwise the volumes with a fullpack overlay are backed up more than once (through the fullpack overlay and through the real minidisks).
  • You have to define these various fullpacks in some control files to have them excluded. For example DIRMAINT's EXTENT CONTROL file.
  • When the addresses are somewhat meaningless, it is hard to know which address to enter on a LINK command when seeking access to a volume.
  • When adding a new fullpack overlay, you must not forget to add the new one to the various exclude lists.
  Our solution to these problems is:
  • Define all fullpack overlays in one user, named FULLPACK, and this user has only fullpack minidisks.
  • As address for these fullpacks overlays, use the real device address, this makes your LINK commands easier to code.
  • Exclude user FULLPACK from your backup SW.
  • Code user FULLPACK as excluded in other control files, like DIRMAINT's EXTENT CONTROL.
  • Replace all MDISK records of users needing access to a fullpack overlay by a LINK. For example for MAINT and DIRMAINT: LINK FULLPACK rdev 123 MW.
  Life is easier once this is done; a new fullpack overlay is only a change to FULLPACK.
  
  There is however one exception to this general setup: suppose the real device address of the VM residence pack changes. If you adhere to our rule you must adapt the MDISK address of that volume in user FULLPACK. Consequently you also should change the LINK records pointing to that volume, but this ain't practical. Therefore, one extra rule:
  • In user FULLPACK, define two fullpack minidisks for the VM residence pack: one MDISK with address 123 and one with the real device address. Then you code once and forever LINK FULLPACK 123 123 MW for user MAINT and DIRMAINT.
  • Similar precautions are taken for spooling packs: VMSPOOL needs a LINK to it. So the spool packs get an extra MDISK definition in user FULLPACK with addresses 124, etc.

DIRMAINT and protection of CP areas

DIRMAINT Rel 5 introduced an automatic protection of some CP areas. It uses the information obtained by CP Q ALLOC to create dummy occupation blocks for Spool, Page, Directory, and T-disk areas. It reports those in its maps as .SPOOL., .PAGE., .DRCT., and .TDISK. respectively. This information is refreshed when a DIRM RLDEXT is issued.

Hence one can conclude it is no longer needed to maintain our dummy minidisks in USER $SPOOL$, USER $PAGE$, USER $DIRECT$, and USER $TDISK$. I have the following elements to vote against this:

• DIRMAINT has no similar mechanism for the other CP areas (like CSE, Checkpoint and Warmstart).
• When the allocation of an existing CP-owned pack is dynamically changed (e.g. extra cylinders are allocated as spool), CP's Q ALLOC command will still display the old allocation information until you can VARY OFF/ON the pack or until the next IPL. During that time, DIRMAINT can not possibly reflect the new allocation map in its occupation blocks, and will consider these new cylinders as free ... unless you add a dummy minidisk for them ...
• If you keep the dummy minidisks of USER $SPOOL$ and co., DIRMAINT will flags those as overlaps. So you should use one or the other, not both.
• When you use non-DIRMAINT tools to create minidisk maps (like DIRMAP or DISKMAP) those areas are reported as free space after removal of the dummy minidisks.

Therefore, we decided to stop using DIRMAINT's .xxxx. entries. A first -but failing- attempt is coding

:EXCLUDE.

.SPOOL. *

.PAGE. *

.DIRECT. *

in EXTENT CONTROL. DIRMAINT however ignores the exclude request for the .xxx. entries. A working solution is the removal of CLASS D from DIRMAINT's privilege classes (so that it can no longer execute CP Q ALLOC) and issuing DIRM RLDEXT.

Extra notes and tools

• FULLPACK sizes
  
  Since DASDs are emulated, it is not uncommon to have 3390 (or 3380) with non-standard sizes. A standard 3390 Mdl 1 has 1113 cylinders, but you might have some left-over space of for example 400 cylinders. For DIRMAINT, you can define the size of each volume in EXTENT CONTROL.
  
  DIRMAP at the other hand dynamically guesses the size of the volumes by matching the highest cylinder allocated to a minidisk with a hardcoded table of known models. (DIRMAP's online help explains: HELP CMS DIRMAP) When you have DASDs with non-standard sizes, add these sizes in FULLPACK DEFINES and use DIRMAP's FULLPACK option. Then proceed as explained above by defining these volumes as fullpack minidisk under user FULLPACK. Do not code MDISK xxxx 3390 000 END volser as then DIRMAP still has to guess how big the volume is, code MDISK xxxx 3390 000 mysize-1 volser.
  
  As DRM (see later) also uses DIRMAP, the same work as above has to be done. In addition, you must tell DRM that it should use the FULLPACK option on DIRMAP commands. Update the :dirmapOpts. tag in DRM NAMES to become for example :dirmapOpts.EXCLUDE FULLPACK.
• DIRBKP EXEC
  
  We daily run the DIRBKP EXEC in MAINT at 6AM on all our VM systems. It is run for various purposes:
  • It keeps 9 backup copies of the directory on MAINT B91 and in a central SFS: SFS72:MAINT.VMxxxxxx. Weekly a TERSED copy of the directory is saved on MAINT 191 and kept for 1 year.
  • It checks that some crucial users and minidisks are not removed from the directory, such as MAINT 190 and 298, VTAM, TCPIP and TCPMAINT xxx.
  • It checks the fullpack overlays in user FULLPACK: each dasd ATTACHED to SYSTEM must have a fullpack minidisk, with the real device address as virtual address. Anomalies are automatically fixed.
  • It tries to set the the real device address as virtual address on all the minidisks of the $xxxx$ users.
  • When some problems are detected an e-mail is sent to the to the addresses defined in :nick.VMSTOR in SUPPORT NAMES.
    
    An extra task of DIRBKP is executed at our VM SW installation system. It extracts all directories found in SFS72:MAINT.VMxxxxxx and checks the minidisk definitions on shared packs.
  • The VSHRxx disks are shared between all VM systems: DIRBKP checks that each VM system has the same MDISKs on these packs. b(When minidisk addresses of MDISKs on VSHRxx cannot be identical, DIRBKP would signal this as problem. Therefore, you can code $DIRBKP$ as read password and a "key" as write password. DIRBKP will then use this "key" instead of the minidisk address for the comparison.
  • Some other disks are shared between two VM systems (branch VMs and Infocenters). As soon as DIRBKP detects a volume appears in two (or more) VM directories, it expects to find the same minidisks on all VM systems sharing that volume. This is an automatic process, the shared volsers don't need to be included in some control file.
  • To make life easier for the DIRBKP EXEC, we make an SFS alias for most recent backup copy of the directory as VMxxxxxx DIRECT-0 in SFS72:MAINT.ALL_LAST_DIRBKP, and this for each VM system. So, if you only need to look at the most recent directory, issue FILELIST * * SFS72:MAINT.ALL_LAST_DIRBKP. else issue FILELIST * * SFS72:MAINT.VMxxxxxx.
• DRM and DRMAC
  
  The DRM tool was created to make minidisk allocations safer and easier for VM systems without DIRMAINT. DRM has some nice features that remain useful even when you've got DIRMAINT. We will not explain all DRM's features, consult DRM's online help: HELP CMS DRM. DRM is available on VM's download library too.
  
  The DRMAC exec makes it easier to start DRM: it links the required minidisks and then starts DRM, at the end, the linked minidisks are detached again. As we've got DIRMAINT, DRMAC uses the USER BACKUP file found on DIRMAINT 1DB. If no-one else issued a DIRM USER BACKUP, DRMAC will find the backup file taken by DIRBKP at 6AM. If you need a recent copy, run DIRM USER BACKUP before starting DRMAC, or, while in DRM, enter DIRM in XEDIT's command line and press PF4 (Refresh). b(You should also know that changes made to the USER BACKUP file in DRM will never be placed online: you must use DIRMAINT commands instead.
  
  DRM helps you with some DIRMAINT commands: you can enter some minidisk related commands in XEDIT's prefix area on the record you like to work with. For example, you can enter DMD on an MDISK card and DRM will place a DIRMAINT command in XEDIT's command-area. You can change the command and press enter to execute it. DRM prepares the least dangerous version of the DIRMAINT command. For example, for a DMDISK, it prepares DIRM FOR xxx DMDISK NOCLEAN KEEPLINK, this way no data is directly lost when you press enter and the DMDISK is sent to DIRMAINT. If you really mean it, you should change NOCLEAN into CLEAN and probably remove KEEPLINK before pressing enter.
  
  DRM helps you also with RACF commands: it executes or prepares RACF commands related to the record on which you enter the command. Here a list of special DRM prefix commands:
  • LK D
    
    On MDISK or LINK records: LINKs to the indicated minidisk, ACCESSes it, starts FILELIST and DETACHes it (the D parameter stands for DETACH, which implies ACCESS and FILELIST first. LK MD is similar, but the M stands for Mult, so we LINK in M-mode).
  • RA
    
    Prepares RACF commands related to the indicated MDISK, LINK or USER.
  • LU and RL
    
    Executes a RAC LISTUSER or RAC RLIST VMMDISK command for the indicated USER or MDISK.
  • AMD, CMD, DMD, TMD, CHV, PURGE
    
    Prepares the corresponding DIRMAINT command for the indicated USER, MDISK, or LINK record.
  • DE and DE L
    
    Starts DIRME or DIRME LOCK for the indicated USER record.
• DIRME
  
  DIRME, part of the DRM package, is a tool for people that don't know many DIRMAINT commands, or that want to view directory entries quickly. It can be used for two kinds of things: user entries and DIRMAINT control files. First for user entries: You issue DIRME userid and are placed directly into an XEDIT session with the directory entry for userid; it is meant for consultation. When you issue DIRME userid LOCK you can apply changes and DIRME will send issue a DIRM FOR userid REPLACE at the end. While this is very easy, you should better learn how to use the native DIRMAINT commands to make the changes, because:
  • Native DIRMAINT commands often require less machine processing: if you change a user's password with DIRM FOR user PASSWORD xxx, DIRMAINT and CP can honor this change with very few I/O operations. If this is done through DIRME (which results in DIRM FOR xxx REPLACE), DIRMAINT must rewrite the whole directory.
  • Each change made by a DIRMAINT command can be traced back in DIRMAINT's console or TRANSLOG file. A DIRM REPLACE leaves traces too, but it will be impossible to know what has been changed. When something goes wrong, it is useful to know who changed what.
  • When you manually add minidisks using DIRME, you can create overlapping minidisks. Very dangerous. Therefore, when DIRME sees you changed MDISK allocations, it requests a DIRM USED: when DIRMAINT sends a second RDR file, named DIAGNOSE USEDEXT, overlaps where found.
  To use DIRME you must be authorized to link to DIRMAINT 1DF. DIRME can also be used as a PIPE stage in EXECs to extract the directory of entry one or more userids. Refer to the introduction comments in the DIRME EXEC.
  
  The second use of DIRME is to consult/update some selected DIRMAINT control files. With native commands you are supposed to issue:
  
  DIRM SEND fn ft

RECEIVE the fn ft from your RDR

XEDIT the file

DIRM FILE fn ft

DIRM RLDxxx
  
  Now you can simply issue DIRME fn ft and you are placed in XEDIT. When you changed the file, DIRME issues a DIRM FILE command and prompts you for an optional DIRM RLDxxx. This DIRM RLDxxx command is required to make DIRMAINT recognize the changed file(s). DIRME recognizes the following DIRMAINT control files: CONFIG DATADVH, AUTHFOR CONTROL , EXTENT CONTROL, fn PROTODIR DIRMAINT DATADVH, and DVHNAMES DATADVH. Other files can be easily supported by changing the files= string in DIRME EXEC. When you forgot the exact name of a control file, simply enter something like DIRME XX XX and DIRME reveals the names of the supported files.
  
  Whereas using DIRME to update user entries is less performant than using native DIRMAINT commands, there is no such drawback when updating DIRMAINT control files.
• DIRENT
  
  DIRENT is a tool we found on VM's download library. This tool allows to reconstruct directory entries from CP object directory. It will XEDIT the reconstructed entry. To use it, you must have a link as 123 to the fullpack minidisk where a CP object directory resides.
  
  Our BUSERVER uses this when minidisks must be copied (with DFSMS or DDR) from one VM system to another. Using this method makes that BUSERVER surely gets the most recent copy of the entry and so it obtains the location of the minidisks it needs to copy. As BUSERVER doesn't like the directory entry be presented in XEDIT, we copied the DIRENT EXEC into DIRENTF EXEC which stores the entry in a CMS file on the A-disk instead of storing in XEDIT. b(If some day the DIRENT tool would no longer work,we know the following, less easy, method could be used: BUSERVER could issue a DIRM to VMxxxxx FOR xxxx GET and wait for the entry sent via a RDR file.

Using Passwords as content description

The last information we want to present here is what we do with the minidisk passwords. As we use RACF as security manager, the three passwords one can code on MDISK records have no longer any real meaning. At the other hand, it is always nice to document in the directory which minidisks are used for what. Usually one inserts some comment records.

After using DIRMAINT's minidisk management commands for a while, these precious comments are no longer placed next the the MDISKs they describe. And often the comments get backlevel. Here an example extracted from our CT system:

*** MAINT 595 : GCS10A ***

*** MAINT 596 : GCS10E ***

*** MAINT 59E : GCS EXECS STARTUP DISK ***

MDISK 0596 3390 1498 2 VCTRES RR

MDISK 059E 3390 1490 3 VCTRES RR

*** MAINT 298 = VTAM 191 : VTAM DEFINITIONS ***

*** MAINT 198 = VTAM 198 : VTAM DEFINITIONS BIS ***

*** MAINT 29A : VTAM/ESA V4.2 28/06/96 ***

MDISK 0298 3390 904 40 VCTRES RR

* OLD 298 ENLARGED 24/06/99

MDISK 229A 3390 2508 17 VCT002 RR

*** MAINT 59F : RSCS

* MAINT 343 /434 - SSP DISKS (SEE PW FOR RELEASES...)

MDISK 0434 3390 2806 40 VCTRES MR

MDISK 0343 3390 184 14 VCTRES MR

* SYSTEMS SERVER MDISK

* NETVIEW 2.2 23/06/97 MDISK 334 = RUN DISK OF NETVIEW

MDISK 0334 3390 2454 145 VCT018

MDISK 0190 3390 287 107 VCTBKP RR ALL ZVM440 JUN-2005

Realy a mess: some commented minidisks even no longer exist or the MDISK record got placed many lines away from the comment. As with RACF the last 3 words on MDISK cards are no longer used as minidisk passwords, we decided to use them as content description. These 3 words will always move with the MDISK card when DIRMAINT touches them. When MDISKs are deleted, the comment vanishes too.

Here an extract of more recent minidisks from the same same directory entry:

MDISK 0190 3390 287 107 VCTBKP RR ALL ZVM440 JUN-2005

MDISK 029A 3390 3215 17 VCT024 RR ALL ZVM440 JAN2005

MDISK 159F 3390 3243 13 VCT021 RR RSCS320 SEP1999

MDISK 0490 3390 1 107 VCT024 RR ALL ZVM440 JAN2005

MDISK 0595 3390 159 1 VCTRES RR ALL GCS20A MAY2004

MDISK 049D 3390 1466 102 VCT018 RR ALL ZVM0202 OCT2002

MDISK 059F 3390 1676 13 VCTSP1 RR RSCS320 NOV2002

MDISK 0B91 3390 1630 20 VCT021 MR RMAINT WMAINT MMAINT

MDISK 0493 3390 2148 167 VCTBKP RR ALL ZVM0401 MAY2004

MDISK 019D 3390 1024 146 VCT024 RR ALL ZVM0401 MAY2004

MDISK 019E 3390 1785 190 VCTSP1 RR ALL ZVM440 JUN-2005

MDISK 049E 3390 159 190 VCT024 MR ZVM440 JAN2005

MDISK 0193 3390 2158 167 VCTSP1 RR ALL ZVM440 JUN-2005

As opposed to true comment lines, these passwords can be maintained with DIRMAINT commands, hence they can also be updated by execs that perform product installations like DISTFILE. When adding a new minidisk: DIRM FOR xxx AMDISK ..... RR PWS rpw wpw mpw or when updating an existing minidisk: DIRM FOR xxx MDISK RR rpw wpw mpw.

I always prefer to use the second method, even when my installation adds new minidisks: if the minidisk to install on would unexpectedly already exist, the AMDISK command fails, and the passwords coded on it do not get set on the existing MDISK. So in my installation jobs, I code both an AMDISK and an MDISK command: the MDISK command will in any case set the passwords.

Some precautions remain: the comments cannot become longer than 8 chars and one may like to keep ALL as read password for some minidisks like MAINT 190, just in case some day one runs a VM without RACF.

Last updated on 18 Nov 2005. B) 2005 KBC/ICS/VMTeam

Attachments:
directory.zip