Flex Family Manager Software Changes

Internal change only.

Change in params to RULES_ENVIRON.

The Flex Usage command has long been a better way to do this. With the recent addition of wastage info, there is no good reason to use Analyse and Summarise in Familymanager, so they have been deleted with a reduction of nearly 1100 lines of source code.

Load:Flat <familyindex> in Flex Inquiry is a better way to do the DISCARD function, so it has been de-implemented in FamilyManager, saving about 5% in the source and object file sizes.

Setting the JOBSUMMARYTITLE attribute in a job which rune Familymanager would cause the copy job generated by Familymanager to fail syntax.

The Jobsummarytitle used by the generated jobs will now be the JobSummaryTitle of FamilyManager's parent with the job name appended.

Internal Metalogic change.

Clean Up.

Use Include/Meta

Make use of addition of Arylen in PD Object.

Tapes created by Familymanager currently have prefixes of FLEXCOPY or the name of the family being dumped. A new parameter (PREFIX) is now alowed which will override the default. The specified prefix must be 10 or fewer characters and be valid for tape names. The Prefix parameter is also valid in Mergetape.

Ex ("DUMP DEV,PRODUCTION PREFIX DEVPROD")
 or
    ("DUMP = PREFIX=HOSTA_ALL")

Update to use FLEXLIB's new FILEKIND handling.

If the default report type was not set to ManagerEmail or UserEmail and the report type for one or more users was an email type it was possible for Familymanager to fault with a string protect error. This fault has been corrected.

The rules block backup timestamp is now correctly updated in runs where only one family is specified. Previously it was only updated on multi backup runs.

The Rules Block modified timestamp is no longer updated when the backup timestamp is changed.

A new option HIDEDONTBACKUP can be added to the parameter list passed to FamilManager. If this option is present then files not backed up do to a DONTBACKUP rule will not be listed in user reports.

Typically the same files are reported as 'not backed up' in every FamilyManager run and can make it difficult to recognise more significant actions in the reports.

Summaries will still report the number of files 'not backed up'.

When generating a very large copy job which had to be split (60000- 100000 files depending on file title length) , the generated job would fail syntax. This problem started with version 510.11.

If too many files had titles close to 80 characters long, the calculation used to determine when to split a job due to WFL size restrictions was inacurate. This could cause the generated job to fail due to WFL limits.

When running for two or more Families the totals, on family reports, no longer include values for other Families.

If only Summary Reports were emailed during a FAMILYMANAGER run and not printed, an empty SUMMARY print file with an associated Prints request would remain on the system. This behaviour has been corrected; this print file will now be automatically purged if the Summary Reports are only emailed.

Change the 'From' in emailed reports to have a 'Friendly name' of

  FMGR <hostname> to make it easier to filter emailed reports

Fix as String Protect error which could occur in a RULES TEST FAM .. command if a number of users had no significant rulings.

The Email summary report now rcorrectly reports file counts for genartaions > 0;

FamilyManager reports can now be directed to email. For sites without Supervisor,the Metalogic Mail Library can be configured via Marc if the Mail directive is set up.

  The Marc command DIRECTIVE MAIL = *METALOGIC/MAILLIB/DIRECTIVE sets
  up the Directive.

  Ex. MAIL STATUS
      MAIL HELP

Two new Rules Options have been added:

                                       
 MANAGEREMAIL  - Sends User reports in the form of HTML email                   
                 messages to the Rules Manager.  If the Rules Manager           
                 usercode has an email entry in the UserData file               
                 then that address is used otherwise the Usercode of            
                 the rules manager is used as the address.  If                  
                 set as a family option then the  FamilManager                  
                 summary report is sent as email instead of being               
                 printed.                                                       
 USEREMAIL     - Sends User reports as an HTML email to the User.  If           
                 the user has an email entry in the Userdata file               
                 then that address will be used otherwise the                   
                 Usercode is used as the address.                   

If running with the DEBUG option, appending to tapes or after detection of WFL syntax errors, FAMILYMANAGER will lock the generated WFL COPY jobs on the dumped family. The names of these files were inconsistent and did not reflect the type of backup performed nor the medium used. Also, where multiple FAMILYMANAGER jobs were running in series, it was possible for CDIMAGE and CD dumps to occasionally fail because of job file overwrite problems.

The COPY jobs now have the following naming convention using backup command type, the output media, the name of the disk family being handled and creation date:

   JOB/<backup type>_<output medium>/<family>/<date>/<reel>

For example:

   JOB/BACKUP_FARM/DEV/2006336/0
   JOB/DUMP_CDIMAGE/WORK/2006335/2
   JOB/BACKUP_TAPE/FLEXCOPY12345/2006330/1

FamilyManager now supports the new format Directory object.

Volume based Backup attributes now work for family TAPE or when the non-Flat SW2 option is set.

FAMILYMANAGER will now pass on additional information about APPEND runs for tape. The tape serial numbers designated for the APPEND, if explicitly selected by the user or chosen by FLEX itself, are passed to the MERGETAPE utility in the Flex CHECKFILE. This allows MERGETAPE to determine the correct average megabyte per file rating, used to assign weighting in a MERGETAPE run, by directly reading the LIBMAINDIR file for the appended tape.

Internal Metalogic change.

If backing up the DL Libmaintdir family and appending to an earlier tape it was possible for the generated copy job to hang attempting to copy the libmaintdir file for the tape being used. Libmaint dir files relating to the current copy are now excluded from the copy.

The restriction in WFL on the total length of a Copy statement can force Familymanager to split a dump into multiple tapes even though the physical capacity of the tape has not been reached. If TapeLength is set to 0 (Infinite) and the specified density is capable of appending then Familymanager will make the subsequent copies append to the original tape. This new option is MCP dependent. It is not available on MCPs < 4.8 and on 4.8 it must be 48.nnn.175 or later. On MCP 4.9 it must be 49.nnn.49 or later. MCPs 5.0 and above are fine.

A new option APPEND has been added. The APPEND option is only valid for Tape destinations. If set then Familymanager will search for the most appropriate volumes to be used for this. If the duplicated option is set then the most recent backup with the same volume prefix will be used. If duplicated is not set then the second most recent matching volume will be used.

Ex. RUN *METALOGIC/FLEX/FAMILYMANAGER("DUMP DEV APPEND")

When using Append either by specification, or if automatic due to a job split, then LIBMAINTDIR files become very important for recovery. Although is is possible to recreate LIBMAINTDIR files from a tape, it can take a very long time. Libmaintdir files should therefore be backed up, as should a copy of the current catalog.

There is an MCP imposed limit on the size of a WFL Job. For Dump or Backup runs where a large number ( > 200000 ) of files are to be copied it was possible to exceed this limit. If a duplicated copy was requested, with the singledrive option set then two copies were done in the one job which reduced the limit to nearer 100000 files.

Familymanager now monitors files to be copied against the Job limit and will split the generated job if the limit is about to be exceeded.

If a generated job fails with syntax errors a waiting entry is now created. The name of the waiting entry is METALOGIC/FAMILYMANAGER/FAILURE with an RSVP indicating the name of the Job which failed. Once the failure has been noted an AX can be entered of the waiting entry.

Previously, specifying options after a SERIALNO OUT option would give errors in the options scanning. This is no longer the case.

This change fixes a problem in the Mergetape run in a Familymanager generated job. The problem would occur if both Duplicated and a SERIALNO OUT clause with two serial numbers was used. Mergetape would report, for each file copied, that the file was missing on Vol #1.

Fixes a problem with SERIALNO OUT vs FARM and CDIMAGE Jobs

A new parameter allows FamilyManager to append to existing tapes using the Library/Maintenance option "LIBMTAPPEND=TOEND".

The syntax is:

--- SERIALNO --- OUT --- <serialno> ------------------------------->
                                      |                        |
                                      +--- , --- <serialno> ---+

The second serial number is needed if appending to 2 tapes on a DUPLICATED run. FamilyManager will flag an error if it cannot find the LIBMAINTDIR for a specified serial number.

Examples:

SERIALNO OUT 33
SERIALNO OUT A00044 , BKUPMT

Using SERIALNO OUT is incompatible with RETRIES options other than NONE Setting RETRIES to another value will flag an error.

In line with MergeTape, FamilyManager now uses only MegaBytes for tape lengths.

FamilyManager may now make a LIBMAINTDIR file when copying to tape, if a new Default option is set appropriately. See FLEX/Inquiry notes for details.

FAMILYMANAGER will now verify that the COPYWRITE library is available and active, before commencing all backup operations to CD, CD image or Disk Farms. The following messages may be seen:

'Missing GETCOPYWRITESTATUS entrypoint in MAGUS'
'COPYWRITE library is inactive or not installed'
'COPYWRITE DSS functionality is DISABLED'

The first message indicates an unlikely incompatibility problem with an older version of MAGUS. If COPYWRITE is already installed, then it may be necessary to re-install using the COPYWRITE utility e.g. from CANDE:

U *METALOGIC/COPYWRITE REINSTALL

Previously, FAMILYMANAGER would ignore the first keyword after a CDIMAGE clause in the parameter though CDIMAGE=<alt.family> would work correctly. For example,'DUMP DEV CDIMAGE DUPLICATED' would cause the DUPLICATED option to be discarded; this problem is now fixed.

FAMILYMANAGER will now correctly handle overrides of COMPARE, RETRIES, SAVEFACTOR and SCRATCHPOOL, provided by its parameter string. These settings were scanned correctly but subsequently ignored prior to the job creation processing.

Version 500.07 inadvertently introduced an Invalid Index error when dumping to Tape. This has been corrected.

For DUMP and BACKUP activities to CD disk images (using the CDIMAGE modifier, FAMILYMANAGER will use the family specified by the Magus config variable FLEX_CWIMAGE as image file destination. It is now possible to override this setting using the following modified syntax:

--- DUMP --- <Family> --- CDIMAGE ---+-----------------------+-- ...
               |                     |
               +-- = <Alt.Family> ---+

When an alternate CD image is specified, the CD images are generated as normal on the default image family and then automatically COPY&BACKUP by CopyWrite to the alternate family. Copywrite removes the image from the original family but leaving a non-resident file entry pointing to the alternate family. This is referred to in Metalogic software as a 'HARD LINK'. You may specify as many as alternate families as required (as long as the family is volumed).

In this example, the default image family is DISK and an alternate image family is called ALTIMAGE. After creation of the image, a PD of the image on each family appears as:

PD *DEV/CI0017F2004142A

FILE *DEV/CI0017F2004142A ON DISK (PROMBURNERDATA)
NOT RESIDENT
FAMILY SERIAL NUMBER: 000505
ENTRY 1 (PROMBURNERDATA):
CYCLE: 1 VERSION: 0
SECURITY: PRIVATE
TIMESTAMP: Friday, May 21, 2004 (2004142) AT 11:52:15
LASTACCESS: Friday, May 21, 2004 (2004142) AT 11:52:02
BACKUP MEDIA IS: PK
SERIAL: 000600
BACKUP MEDIA IS: PK
SERIAL: 000600

On ALTIMAGE (pack serial number 600):

PD *DEV/CI0017F2004142A ON ALTIMAGE

FILE *DEV/CI0017F2004142A ON ALTIMAGE (PROMBURNERDATA)
CREATION DATE= Friday, May 21, 2004 AT 11:52:01 BST
LASTACCESS DATE= Friday, May 21, 2004 AT 11:52:01 BST
ALTER DATE= Friday, May 21, 2004 AT 11:52:01 BST
ATTMODIFY DATE= Friday, May 21, 2004 AT 11:52:16 BST
BACKUP DATE= Friday, May 21, 2004 AT 11:52:01 BST
COPYDEST DATE= Friday, May 21, 2004 AT 11:52:01 BST
COPYSOURCE DATE= Friday, May 21, 2004 AT 11:52:01 BST
EXECUTE DATE= Friday, May 21, 2004 AT 11:52:01 BST
READ DATE= Friday, May 21, 2004 AT 11:52:01 BST
TOTAL SECTORS: 88,896 (5,556 PER AREA)
SECURITY = OWNER *:RWX, GROUP <none>:NO, OTHER:NO
(PRIVATE - USAGE: READ/WRITE) LOCKEDFILE
FAMILY SERIAL NUMBER: 000600
ENTRY 1 IS RESIDENT, NOT CATALOGED (PROMBURNERDATA):
CYCLE: 1 VERSION: 0
SECURITY: PRIVATE
TIMESTAMP: Friday, May 21, 2004 (2004142) AT 11:52:15
NO ARCHIVE BACKUP RECORD FOR FILE

The MERGETAPE utility has been changed to support this new feature, please see MERGETAPE DNote 500.02 for more information.

This change allows the FEET=0 option in Familymanager to override a non zero default tape feet value.

In addition when FEET=0 is specified the correct file size will be used when calculating the average file size value used by Mergetape. Previously when FEET=0 was used a value of 1 was used for every file.

Version 500.05 inadvertently introduced an Invalid Index error when dumping to Tape. This has been corrected.

The change described in DNote 500.02 caused the SERIALNO assignment for a FARM copy to be assigned an empty string which caused the COPY to fail. Disk FARM copies will now again correctly allocate a serial number.

The change applied in Dnote 490.06 had the side-effect of causing CD dumps to fail with INVALID INDEX at 56735490. This problem is now fixed.

For CDIMAGE and CD backup runs, FAMILYMANAGER would unconditionally create a SCRATCH volume entry for the impending backup even if there were no eligible files for copying. This meant that spurious SCRATCH entries were permanently left in the Volume Library. This behaviour has been corrected.

Internal change only.

When dumping to CDIMAGE on MCPs < 4.8 it was possible for the generated Job to fail with a syntax error 'COPY OR ADD STATEMENT TOO LONG'. This problem has been corrected.

Attempts to remove files with 'long filenames' would give an attribute error and the file would not be removed. This problem has also been corrected.

During FAMILYMANAGER's initialisation phase, the availability of the FLEX RULES file and the setup of the RULES manager usercode and accesscode will now be verified. Previously, FAMILYMANAGER would not give any warnings if these essentials features were not established.

If the RULES has not been set up or is unavailable:

	'RULES file is not valid or does not exist' 

If both the RULES Manager usercode and its alternate usercode have not been setup:

	'RULES Manager USERCODE is NOT assigned' 

	'RULES Manager ACCESSCODE is NOT assigned'

When backing up files to a Disk Farm the TARGET task attribute is set so that CopyWrite performs the Copy, even if NFT is configured.

When producing backups to PC Disk Farms using the FARM function, FAMILYMANAGER would always ignore the DUPLICATED modifier. So, if DUPLICATED was set e.g.

RUN *METALOGIC/FLEX/FAMILYMANAGER ("DUMP DEV FARM(META1) DUPLICATED")

The above run would cause FAMILYMANAGER/COPYWRITE to generate a single reference for each dumped file instead of 2. This, in turn, caused MERGETAPE to fail, assuming that the second backup reference had failed. This problem is now resolved.

The internal version of FAMILYMANAGER has been changed to reflect MCP 49.1 compatibility.

Previously, FAMILYMANAGER would unconditionally give a run-time error processing a CD, CDIMAGE or FARM request unless all the Magus config variables were set up. For example, a CDIMAGE run would fail if the FLEX_CWHOST config (used for CD production) was not set up. This behaviour has been corrected and FAMILYMANAGER will only check the appropriate config items, as set up by the INSTALL utility.

The parameter FARM ( <hostname>) will cause Familymanager to backup files to the Copywrite disk farm on the PC specified by <HostName>.
Ex.
RUN *METALOGIC/FLEX/FAMILYMANAGER("DUMP DEV FARM(BACKUPPC)")

FAMILYMANAGER will now ALTER disk containers generated by Copywrite so that the LOCKEDFILE attribute is always set. This helps protect these files from accidental removal.

During a DUMP to CD, the change applied by 480.07 caused the familyname being dumped to be incorrectly written to the Flex CHECKFILE. This caused the MERGETAPE run, which uses Copywrite to catalog the files on the CD, to use an incorrect FAMILYOWNER for each file. This problem is now fixed.

During a DUMP to CD, the change applied by 480.07 caused the familyname being dumped to be incorrectly written to the Flex CHECKFILE. This caused the MERGETAPE run, which uses Copywrite to catalog the files on the CD, to use an incorrect FAMILYOWNER for each file. This problem is now fixed.

Using the keyword DUPLICATED in a FAMILYMANAGER CD image run causes the creation of a second, duplicated backup reference for each file dumped. During the MERGETAPE run to verify the backup references, MERGETAPE would display 'Failure on second backup' producing warning messages for each file indicating that the second backup reference was incorrect. This problem is now fixed.

On MCP's prior to 4.8 there is a restriction on the total length of a COPY statement when copying to CDIMAGE. This could cause Familymanager generated copy jobs to fail with a syntax error.

Familymanager now takes this limit into consideration and will split the job when required. The created job is now locked if any syntax error occurs and a message indication the name of the Job in question is displayed.

When copying to CD image the default behaviour was to generate one CD image but to set both backup references to that image. This has been changed. Now only one reference to the image is created. To get old behaviour it is necessary to set the DUPLICATED option in the parameter to Familymanager.
Ex.
RUN METALOGIC/FLEX/FAMILYMANAGER("DUMP DEV CDIMAGE")
will create one reference for the files copied.
RUN METALOGIC/FLEX/FAMILYMANAGER("DUMP DEV CDIMAGE DUPLICATED")
will create two references to the image for the files copied.

Previously backups to CDIMAGE created the image under the directory *FLEXCDIMAGE. If backups were done from various families it was very difficult to determine which family a particular image was for. The convention has been changed and the directory name now matches the family name. So, if PACK1 is backed up to CDIMAGE the directory is *PACK1/...

This change allows Familymanager to dump to CD or CDIMAGE on a system where the Flex defaults are still set to Tape. CD or CDIMAGE can be added to the DUMP parameter.

Ex: ("DUMP MYPACK CDIMAGE").

The only prerequisite is that four config variables are set up using U META/INSTALL CONFIG Select 7 for Full Flex and then change each of FLEX_CWHOST,FLEX_CWIMAGE
FLEX_CDSERIALNO and FLEX_CWSERIALNO.
FLEX_CWHOST should be the host name of your system.
FLEX_CWIMAGE should be the family where you want to keep CD images.
FLEX_CDSERIALNO at present should be CD0001 (It will be updated by FLEX
FLEX_CISERIALNO at present should be CI0001 (It will be updated by FLEX

Dumping to CDIMAGE requires the additional Metalogic product COPYWRITE.

This change does some preliminary work to allow Dumps to Copywrite Disk Farms.

Any file with a blocksize < 6 characters would cause an Analyse or Summarise run to fault with a divide by zero.

The summary by date for the Analyse and Summarise command was wildly inaccurate if any user code did not have files which had been accessed on the day of the run. In extreme cases Familymanager would fault with a resize error.

Both faults have been corrected.

This patch only changes the version to 48. Version 470.07 is still valid for MCP 48.

FAMILYMANAGER will now correctly handle files with long names on system running with SYSOPS LONGFILENAMES set. Also, the DISCARD function will now work correctly establish its internal sort files and process files whose FILESTRUCTURE is STREAM.

When running Familymanager on three or more families it was possible for the generated copy job to use the wrong familyname for one of the families. This would happen if there were no files to be copied on the first of three consecutive families

Previously, when dumping only to CD or CD images, the JOBBUILDER processes would unconditionally fail with INVALID OP when generating any continuation volumes. This meant that only the first volume (name suffixed by "A") would be created successfully. This problem is now fixed.

FAMILYMANAGER automatically excludes certain file kinds from being backed up at any time. For example, files such as DBDATA, JOBDESC, SYSTEMDIRECTORY etc. However, despite popular belief to the contrary, these files were NOT excluded from rules such as RIP, REMOVE and ZIP.

In the interest of file safety, all such files will now be exempt from *any* Flex rule during a FAMILYMANAGER run.

CD and CDIMAGE are now allowed as modifiers in the parameter to Familymanager. If CDIMAGE is specified the Metalogic CopyWrite product is required, and files are backed up to a Copywrite CD image. Both backup references are set to the same image. The config variable FLEX_CWIMAGE is used to specify the family where the images are to be kept, as described in Dnote 450.02. The Config variable FLEX_CDSERIALNO is used to assign serial numbers to the image. Thus if FLEX_CDSERIALNO=I00000 the next serialno would be I00001 and FLEX_CDSERIALNO would be updated to I00001. The CD images are entered in the volume library as pseudo packs so PV PK I00001 would give details of the image.

Example run:

     RUN METALOGIC/FLEX/FAMILYMANAGER("DUMP DEV CDIMAGE") 

The Duplicated attribute is deimplimented on MCP 4.6. Consequently Familymanager will no longer accept Duplicated in a file equation. Use the Duplicate option the task parameter instead.

This change is required if running on MCP 4.7 or later

Version 450.98 inadvertently left the TADS compiler option set. This does not effect the results produced by FamilyManager but does impose a processor overhead. The option has been reset.

It was possible for Familymanager to fail with a seg array error when processing file names of 14 levels each of 17 characters.

Prior to this release a Forget rule would not forget backups for a resident entry.

This has now been corrected.

The Archive Log count lines have been eliminated from the standard reports, and the more efficient FLEXFORM masks are now used.

FAMILYMANAGER will now treat files with LOCKEDFILE set as if they were RIP. If the file satisfies a ZAP, REMOVE, or RELEASE rule it will be removed. SYSTEMFILE files are now included in the INUSE report.

FAMILYMANAGER used to ask the file search for all attributes except the header, the GUARDFILE, and the USERINFO. Because of this GUARDFILE USERINFO, and SINGLEPACK did not work.

This has now been fixed by using information gathered by OPAL and stored in the rules blocks, which will allow new attributes to be implemented and more efficient searches.

Previously, for a multi-family run with the DUPLICATED run-time option set, FAMILYMANAGER would ignore the setting of the SINGLEDRIVE Flex DEFAULTS option, unconditionally requesting two drives.

This could be easily detoured by including the keyword SINGLEDRIVE in the run-time parameter, as discussed in Dnote 450.09, but is now fixed.

Further, the key word DUPLICATES is now permitted as a synonym for DUPLICATED in the parameter options.

To avoid the need to use File Equation to override default settings more options have been added to the parameter to Familymanager.

The Syntax Diagram now looks like:

For a Boolean option where neither ON nor OFF is specified, ON is assumed.

A Display, listing the action to be taken and the Families to be used has been added. A second Display lists the active options.

This is a temporary fix to allow the syntax DUPLICATED OFFSITE SELECT in FamilyManager. Previously the run would proceed but only DUPLICATED would be set, OFFSITE and SELECT would be ignored.

A subsequent change will improve parsing of commands and return an error if any part of the parameter is invalid. At present the program continues with a 'best guess' which can hide problems.

When dumping files to CD using CopyWrite, certain configuration variables are required. If they are not set up the generated copy fails with syntax errors. The variables are now checked during initialisation, and Familymanager aborts if they are not present.

FLEX_CWHOST should be set to the host name of the system where the CD image file should be sent to be burned.

FLEX_CWIMAGE is the name of the family where CD image files are stored.

FLEX_CDSERIALNO is a six character string representing the next serial number to be used for CD output.

This problem was caused by code inserted to detour an MCP bug. The bug was corrected in MCP 4.2 and had the effect that every FamilyManager run would switch the order of the backup references of each file scanned. When the 'SELECT TRUE' clause was used this could result in the oldest backup being retained and the latest being replaced by the new backup. The 'detour' code has now been removed.

This change also support the change to FLEX 450.03 which allows zero to be used for tape length. A tape length of zero is treated as an infinite length tape.

450.03 introduced a bug with the last usercode on a family - now fixed.

FAMILYMANAGER now checks to make sure that the length of the WFL COPY statement for the output tapes does not exceed its maximum.

This patch protects against an INVALID INDEX problem that may have been seen when FAMILYMANAGER was processing a specific family within a multi-family list.