Maillib Software Changes

The change described by DNote 530.08 addressed a problem where data packets sent to a SMTP server starting with the '.' character caused

-554 errors.  Unfortunately, this fix did not protect all variations of

the problem especially with simple data streams (i.e not file attachments). These issues have been corrected.

The ^OP READRECEIPT action now works correctly if supplied with a valid email address.

The ^ATTACH and ^INCLUDE actions will now handle attachment file names which have nodes that are quoted and include lower-case characters.

The LOAD NICKNAMES command will attempt to load the file from the family to which MAILLIB is SL-ed. Previously, only family 'DISK' was used.

MAILLIB will now recognise and correctly handle GZIP attachments; PDF attachments are now assigned the correct mime Content-Type of 'application/pdf'.

Support has been provided for the NICKNAMELIST and MYSELF(TRANSNO) SYSTEM ML attributes as described in SUPERVISOR DNote 540.54.

A new MAIL command, TRANSNO, allows the user to conveniently search the Metalogic MAILLIB logs for a specific email transaction number. Since this command uses the LOG FIND facility, additional log filtering can be appended to the command.

 -- MAIL --- TRansno --- <TransNo> ----+----------------+--------|
                                       +-- <log spec> --+

For example:

    MAIL TRANSNO 1234

       ---- MAILLIB Log at 17:03 ----
       ---- Search 'All' ----
       ---- 15/06/10 LOG ----
    16:46:05 Snd:07025:Sent OK to:Ian.Patterson@metalog.com
    16:46:03 Att:07025:Attached *TEST/FILE ON DEV (864.0 Kb)
    16:46:02 Rcv:07025:To:Support@metalog.com,Subject:Test

    MAIL TRANSNO 5678 BACK 2 DAYS
    MAIL TR 1234 +

Internal change to allow MAILLIB functionality with ASAP SUPERVISOR.

MAILLIB will now not give the attribute error 'RAWFILE.RELEASEID STRING DID NOT IN PERIOD OR NUL' @ 53548900 when queuing a SMTP file to disk because of server unavailability. The attribute error would only occur if the email subject contained any double-quote characters.

Also, MAILLIB will now automatically attempt to re-send any currently queued emails during its initialization phase. Previously, this was not done until 180 seconds after MAILLIN had started.

Emails that have been queued because of -987 errors, "Connection refused by server" will now be automatically retried by a NUDGE command or queued email poll.

If a warning email was sent to a recipient that had already exceeded its local storage on the mail server, this email would have been rejected and MAILLIB would have attempted to send another warning, eventually causing MAILLIB to fault. This behaviour has been corrected.

This change also addresses several minor problems with MAILLIB command handling and logging. MAILLIB will now report the correct number of active SMTP connections in the STATUS command response.

When both Primary and Alternate Mail Servers are specified, a timing hole existed where emails could be lost if the primary server went off-line whilst the internal status of the secondary was incorrect. MAILLIB now properly protects all references to internal server information and this problem has been fixed.

Previously, MAILLIB would have attempted to handle many stream file attachments without mime-encoding them. File types such as .ZIP and container files are detected but other types such as .PDF were not handled correctly. The problem with certain PDF files has been observed to cause MAILMAN fault #14 errors at MAILLIB sequence numbers 59770000 and 59860000.

MAILLIB will now unconditionally encode *all* stream file attachments: this will prevent problems with unusual file content. PDF files are now detected correctly and '.PDF' will be added to the attachment name if required.

Previously, MAILLIB did not ignore 4"0A" characters when these were present in the text body of the email (i.e not ^INCLUDE or ^ATTACH content). This problem resurfaced after recent changes to support the PDF implementation and is now fixed.

Because there are various reasons for any email transaction to be queued by MAILLIB, the MAILMAN and QUICKMAIL entry points will now return a flag in bit [47:1] indicating that the email has been saved to disk. This means that the user can take appropriate action to reprocess the email instead of the caller attempting to retry what maybe already a lost cause.

Previously, MAILLIB did not correctly handle an email request that had been rejected by a server because its own maximum number of SMTP connections had been exceeded. In such cases, the mail server forcibly closes the SMTP connection immediately but MAILLIB did not detect this for 60 seconds, eventually marking the failure as a 'Read Timeout'. To compound the problem, MAILLIB was unable to reroute the email to the alternate server (if assigned) because the SMTP port file was not set up correctly.

Now, MAILLIB detects the immediate forced close immediately and will properly use the alternate server, if available. If the alternate is unavailable, the email will be queued to a *SMTP file on disk with the error value -987 and is eligible for reprocessing later.

 12:18:48 Que:01228:Message Queued [Closed] (-987):Connection refused by server 

To address the wider issue of too many concurrent MAILLIB SMTP requests requests flooding the email server, a new control mechanism has been implemented to allow a limit to be enforced. A new command, MAXSMTP, is now available to control this setting: the default value is 10 connections but can take any value between 1 and 100. The syntax is:

   --- MAIL --- MAXsmtp -----+------------------+----------|
                             +-- <integer> -----+

   MAIL MAXSMTP 5

   --- MAIL Library response ---
   Maximum concurrent SMTP connections is now 5

When an email request is received by MAILLIB which will exceed the MAXSMTP limit, MAILLIB unconditionally holds up the request for 30 seconds before retrying. If the limit would still be exceeded, the request is queued with a result of -997 (not -999) and a *SMTP disk file is created. The MAILLIB log will show entries such as:

  10:18:59 Msg:2900:Max connections exceeded:mail queued
  10:19:02 Que:02900:Message Queued test@metalogic.eu.com (-997):Too
           many SMTP connections:mail queued

The -997 SMTP files are treated in the same way as normal queued files and MAILLIB attempts to reprocess them during the next 'nudge' cycle.

Also, additional status information is now provided by the MAIL STATUS command showing corrected transmitted bytes values and destination details such as alternate server or SMTP file. MAILLIB now correctly applies the ALTPORTNO setting to the alternate mail server instead of previously using the value assigned to the primary.

Previously, using the QUICKMAIL entrypoint to process email requests would split very long lines of text into 1000-character segments. In such circumstances, QUICKMAIL would incorrectly drop the first character from every split line except the first. This problem has now been fixed and, in addition, the maximum number of characters before line-wrapping occurs has been increased to 5000.

MAILLIB now supports CRAM-MD5 as a method for SMTP authentication. This challenge-response authentication mechanism is widely used by many SMTP servers and is significantly more secure than the PLAIN and LOGIN methods also offered by MAILLIB.

The syntax of the AUTH command has been modified to support CRAM-MD5:

  MAIL AUTH CRAM-MD5

      --- MAIL Library response ---
  SMTP Authentication is set to 'CRAM-MD5'
   Username and Password are NOT set!!!

If a password has not been previously assigned, as shown above, then:

  MAIL AUTH USER TEST/PASSWORD

      --- MAIL Library response ---
  SMTP Authentication is set to 'CRAM-MD5'
    Username is 'TEST'
    Password has been assigned

As before, if the username and password (aka secret key) are incorrect, the mail server will reject any all email transmissions.

13:29:30 Err:02095:Server report #535 5.7.8 Authentication failed

Note that, at this time, MAILLIB only supports 23-characters passwords.

In rare circumstances, it was possible for a queued SMTP file to be rejected, when subsequently processed, with a 'Bad message text' error (-989) which indicates a corrupt SMTP file. This problem has been fixed.

The change described in DNote 540.01 could cause unexpected line wrapping in a generated PDF file if the streamed line of text included any '\','(' or ')' characters.

When creating PDF files, MAILLIB will now wrap individual lines of streamed data (delimited by carriage-return) instead of using truncation. Note the default line width is 90-character for stream file input; this may be overridden to landscape mode (132 characters) by using the ^ATWPDF (wide-PDF) modifier or by equating the ATWPDF logical file with the MAILER utility. Generally, MAILLIB will now handle line lengths of 20,000 characters in a stream attachment instead of the previous limit of 1,000.

The PDFWRITER entrypoint has been exported for use by other Metalogic software.

The handling of Virtual printing requests that exceed site-defined User limits has been changed. Previously, MAILLIB's Virtual Print server would have refused the request, marked the request as EXCEPTION and sent a simple email to the original recipient.

This behaviour has been changed. Virtual Printing is now handled in the same way as normal email attachments such as those sent by the MAILER utility. This means that only those files that exceed the users quota are rejected, other files will be handled as normal. The rejected files are not removed and can, optionally, be printed at a later time if the message size limits are relaxed for that user or printed by another user.

As with normal rejected attachments, Virtual Mail requests that exceed limits will send detailed messages to the recipient and the MAILLIB postmaster. See MAILLIB DNote 530.12 for more details.

Please note that MAILLIB will always use the physical size of each printer backup file when calculating user limits although this will not match the actual attachment size when delivered to the email client. This is not done because of the additional overhead that would be incurred.

The message size calculations used by User Limits to control large file attachments has been improved. In particular, attachments which require additional message content are now adjusted by 5% though this is only of significance for large files.

Also, MAILLIB will now retain '%' characters in ASCII stream file attachments instead of incorrectly translating them to carriage returns.

Previously, MAILLIB could have corrupted certain HTML attachments by unexpectedly splitting a line of HTML in the wrong place causing the file to be incorrectly rendered by the email client. Typically, such errors would occur after 3000 characters had been read from the source. This problem has been fixed.

MAILLIB now supports several modes of SMTP authentication, allowing access to protected SMTP mail servers. In this implementation, only LOGIN and PLAIN modes are supported and there is no SSL or TLS security available for encrypting the login so these protocols are not available.

A new MAIL command called AUTH is now available and requires MAILLIB to authenticate all email requests with both primary and alternate mail servers. In a later release, it will be possible to assign individual authentication at the server level.

   ----  AUTH --------+----------------------------+----|
                      +----   -   -----------------+
                      +---- LOGIN -----------------+
                      +---- PLAIN -----------------+
                      +---- USER  --- <User/Pass>--+

If AUTH is set to PLAIN or LOGIN, an AUTH USER command must also be used to establish security. MAILLIB will always expect a password to be provided after the user and must be separated by '/'.

   MAIL AUTH PLAIN
   MAIL USER TEST/test
   MAIL AUTH -

Appropriate steps within MAILLIB have been taken to secure the password. Note that the USER specification is case-sensitive. The authentication settings can be cancelled at any time by using the AUTH

- command.

The generation of PDF attachments for blocked DATA files is now handled correctly. Previously, for some such files, the attachment would have been corrupted.

PDF is now permitted as a default file extension for all unqualified attachments (i.e. those that have not been assigned an alternative name or cannot be converted to PDF. This setting may be changed using the ATTACH command e.g.

  MAIL ATTACH PDF

The recent change to support PDF creation caused queued email handling, used when the mail server is unavailable, to fail. The usual SMTP files created on disk would generate corrupted files, usually with no message content, when the server came on-line. This problem has been corrected.

The changes applied by DNote 530.26 inadvertently caused INCLUDE files with sequence numbers to retain those numbers inside the email body. MAILLIB has always stripped sequence numbers, regardless of symbol FILEKIND, so the previous handling of INCLUDE files has been restored.

The MAILLIB library now has the capability to create Adobe PDF files either by streaming source files as email attachments or by creating the files directly on the MCP system. At this time, the implementation is by no means complete and there are various enhancements planned.

With regards to the mail system, PDF attachments can be created from any symbolic or readable stream file. The data is streamed directly in the email content so no local PDF file is created on the MCP system. Instead of using the ^ATTACH directive with, for example, the Opal MAIL function, a PDF variant is available:

  ^ATTPDF *SYMBOL/MYSOURCE ON DEV

The email attachment file name will be automatically assigned an extension of .PDF to allow the file to opened by the Adobe Reader utility. All PDF files, by default, will open with page thumbnails visible in the Bookmarks navigation panel.

According to file specifications, MAILLIB will attempt to determine the best page formatting and font size to be used. Currently, only the font COURIER NEW is permitted but it will be possible to override this. If necessary, MAILLIB will switch the page set up to Landscape instead of Portrait (e.g. allowing better presentation of 132-character backup files). At this time, only A4 page formatting is supported.

Regarding STREAM files, MAILLIB currently uses a default line width of 90 characters and any lines longer than this are truncated inside the generated PDF file. This temporary behaviour will be corrected in the next release.

The PDF files are uncompressed and not linearized - this means that they are not optimized for fast loading by network or internet access (via a browser). Both linearization and, possibly, compression of PDF files will be available in later releases.

PDF files can also be generated from printer backup files by the MAILLIB Virtual Server. At this time, the only way to print files direct to PDF is to set up a new PrintS device:

  PS CONFIG MAILPDF SERVER "PRINTTOPDF IN SL MAILLIB"

By changing the PRINTDEFAULTS DESTINATION setting of a MCS session or WFL job, all generated backup files will be created with the .PDF extension. Additional controls will be available later allowing the use of PRINTS attributes such as PAGECOMP and FORMID to signal a PDF request.

A new public entrypoint in the MAILLIB library allows PDF files to be created by any external program. This entrypoint is shown below:

  Boolean Procedure CREATEPDF(INPUT,PDF);
  File INPUT;
  Array PDF[0];
     Library MAILLIB;

The CREATEPDF entrypoint accepts a symbolic blocked file as source and requires the name of the output PDF file from Words[3] onwards in simple DisplayForm name format. There are no security restrictions in force, normal MCP file security applies.

  File INPUT(KIND=DISK,TITLE="SYMBOL/MYSOURCE.");
  Array PDF[0:21];
  Replace Pointer(PDF[3]) By "PDF/"48"7F""MYSOURCE.PDF"48"7F"".";
  CREATEPDF(INPUT,PDF);

At this time, CREATEPDF does not yet handle stream files.

Several issues with the handling of non-ASCII codesets (such as LATIN1ISO) whilst generating file attachments have been addressed. For email body text, MAILLIB will now apply "iso-8859-1" and "iso-8859-2" character sets for LATIN1ISO and LATIN2ISO codeset configurations respectively instead of the general "us-ascii". No other character sets are yet supported.

The Mail Help command indicated that there was one more line of help than there actually was. This could in some cases cause Supervisor to fault with a bounds error.

The changes described by DNotes 530.19 and 530.20 did not handle all cases of carriage-returns embedded in either printer backup or symbolic files. THis caused stricter mail servers to possibly reject emails with a -501 error e.g. Stray CR character in the message. MAILLIB will now ensure any spurious CR characters are translated before data transmission.

The MODIFY command will now show the requested transaction number in error messages if the transaction is unknown; previously, the last transaction in the mail queue would have been displayed. Emails that have been queued by MAILLIB because of a -530 error ("Relay to this address is prohibited") may now be the subject of a MODIFY command to change the recipient email address.

Previously, on MCP releases with the new SYSOPS option SYSTEMLOGGING, it was possible for MAILLIB's Virtual print server to fail with the following error when printing a CANDE or MARC session after a log-off or SPLIT.

 "Print aborted. Request 45194:No email address from USERCODE or NOTE"

The conditions for this problem are shown below:

1. The SYSOPS option SESSIONLOGGING is set to CONDITIONAL or UNCONDTIONAL.

2. PS DEFAULT JOBSUMMARY is set to either CONDITIONAL or UNCONDITIONAL.

3. The split session has the appropriate NOTE and DESTINATION PrintS settings but the session USERCODE
does not have an email address assigned.

This problem is now resolved.

The change described by DNote 530.19 caused multiple embedded carriage returns within a single text line to be converted to '?' characters when processed by MAILLIB. This problem caused presentation issues particularly for HTML emails and is now fixed.

Previously, MAILLIB was not handling stray CR or LF characters that may have appeared in some printer backup files. This caused some SMTP servers that used strict checks on the presence of these characters to fail email transmission with 501 SMTP errors. This change has been extended generally for all email handling not only backup files.

The change described in DNote 530.12 caused the SUPERVISOR OPAL attribute PRIMARYSERVER to return a corrupted string; this problem has been corrected.

Previously, using an ^ATTACH directive in a message body to process a directory which had filenames with nodes longer than 17 characters caused MAILLIB to omit those files from the attachment processing. This problem is now fixed.

Further, MAILLIB now better handles some SEQDATA attachment or include files that have embedded null characters in character position 80-84 of each record. This caused certain stricter SMTP servers to return a

-501 error "There is a NUL character in the message".

The MAILER utility now also better supports attachments and includes that process long filenames. For example, passing an ^ATTACH request using TASKSTRING would have previously failed if quoted levels appeared in the attachment filename/directory.

To assist with directory handling in general, MAILER will now accept two new run-time file equations: ATTACHDIR and INCLUDEDIR. When present, MAILLIB will append '/=' to the file title before passing it to MAILLIB.

For example, the following MAILER file equations:

    FILE ATTACH=(ADMIN)FILE/1             maps to

    ^ATTACH (ADMIN)FILE/1                 MAILLIB file request

To process a file directory:

    FILE ATTACHDIR=(ADMIN)FILES ON DEV    maps to

    ^ATTACH (ADMIN)FILES/= ON DEV         MAILLIB file request

INCLUDEDIR can be used instead of ATTACHDIR in the above examples and maps them onto their ^INCLUDE equivalents.

The recent changes applied to improve 552 errors caused MAILLIB to ignore BASE64 mime-encoding for both container and zip file attachments. Although the attachment would appear correct in the client email, the data would be corrupted. This problem is now fixed.

Versions of MailLib > 530.08 could sometimes fail with the error:

*METALOGIC/MAILLIB/CONFIG I/O ERROR: END OF FILE @ (22888015)

This problem has been corrected.

The changes applied by DNote 530.12 caused any STREAM file attachment to be corrupted if the individual file was greater than 1000 characters. This problem has been fixed.

The usercode of the caller will now be shown in any MAILLIB-generated

-552 error emails for both the Postmaster and intended recipients(s).

This is a major change in the way that -552 errors (server storage exceeded) are handled by MAILLIB. Also, User Limit controls by usercode are now better handled and provide better information both to the Postmaster and recipients when an email exceeds its limit.

Previously, a -552 error was retryable allowing the site to modify the queued transaction or correct the storage problem for that mailbox on the mail server. This behaviour has been changed; -552 errors are now not retryable and will be automatically rejected; no SMTP file is created.

MAILLIB will now notify email both Postmaster and intended recipients with an email when a -552 error has been encountered. The email provides detailed information about the failure, including attachment and include file details. An example email is shown below:

 --
 Subject:MAILLIB Alert! SERVER storage exceeded
         for ian.patterson@metalog.com [-552]

 The following email exceeded Storage limits on Mail Server DELL8500

   To:ian.patterson@metalog.com
   Subject:Test
   Estimated Mail size: 1273.3 Kb
   Transaction No: 4902
   Maillib returned: -552

 The above email transaction was Rejected

 #1  *SOURCE/NEWTAPE ON DEV   (1273.3 Kb)
 -> Attachment was processed ok
 --

For emails that exceed Maillib User Limits restrictions, the recipient receives the email intact but with warning messages indicating the reasons that any files/includes have been skipped or truncated. In addition, the Postmaster will also receive an advisory email similar to that for a -552 error.

In the event that a large email is sent to an offline server and the email is not restricted by MAILLIB User Limits, MAILLIB is obliged to retain the *SMTP file but when the server subsequently becomes available and fails the queued email with a -552 error, MAILLIB will now also automatically remove the associated *SMTP file. Both system messages and entries in the MAILLIB log will show this behaviour:

10:48:24 Snd:04906:Sent OK to:ian.patterson@metalog.com,
         POSTMASTER@METALOGIC.EU.COM
10:48:24 Rcv:04906:To:ian.patterson@metalog.com,Subject:MAILLIB Alert!
         SERVER storage exceeded for ian.patterson@metalog.com [-552]
10:48:24 Err:04904:*SMTP file removed due to SERVER storage limit exceeded
10:48:24 Err:04904:Rejected ian.patterson@metalog.com (-552):Requested
         action aborted:Exceeded storage
10:48:24 Err:04904:Server report #552 Message too large - contact your
         administrator
10:48:14 Msg:Checking SMTP mail Queue (2 messages)

The MAIL SHOW command has been enhanced to show attachment and include file details when a transaction number is provided as a modifier to SHOW:

  MAIL SHOW 4912

      --- MAIL Library response ---
      --- Searching for Queued Email  #4912 ---
  #4912 *SMTP/SUPERVISOR/20081126/"ian.patterson@met"/04912 1303.4 Kb
    To:ian.patterson@metalogic.eu.com release@metalog.com
    Subj:'Attach *SOURCE/NEWTAPE ON DEV'
    Reason: Queued (-999)
    #1 : File *SOURCE/NEWTAPE ON DEV     (1273.3 Kb)
      -> Attachment processed ok

 Each file will show status information about each attached or
 included file.

MAILLIB will now indicate adjusted file sizes when an attachment has been MIME-encoded. This has an effect for User Limit and Server storage limit considerations since MIME-encoded attachments are 33% larger than the original file length. Where appropriate, MAILLIB will indicate in the attachment details entry if a file has been encoded.

Lastly, MAILLIB would have faulted with a SEG ARRAY ERROR @29169630 when trying to MIME-encode certain stream file attachments This problem has been fixed.

The change described in DNote 530.10 regarding the stricter handling of the LOAD NICKNAMES command caused MAILLIB to fail to find the NICKNAMES file on MAILLIB's SL-ed family at initialisation time. This problem has been corrected.

Previously, if the mail server was unavailable and MAILLIB was forced to create SMTP recovery files, the following attribute error would be displayed if the emails' 'To:' address was derived from a loaded MAILLIB nickname:

 ATTRIBUTE ERROR:RAWFILE.PRODUCT STRING DID NOT END IN PERIOD OR NULL

This did not cause the SMTP file to lose its intended recipients but the recipients would not be shown in a MAIL SHOW QUEUE response. This problem is now fixed.

MAILLIB will now correctly handle a SMTPFAMILY that is set to DISK; creating SMTP files with the MAILER utility may have caused MAILLIB to fail to add them into its queue. The formatting of various error messages relating to SMTP file creation has been improved.

A new MAILLIB command called SMTP has been implemented. This command allows the site to control the location of the *SMTP disk files created for email requests that have been queued (e.g. because of Mail server unavailability). Previously the SMTP files would have been directed to the family specified in the MAILLIB SL, typically the H/L unit, which may be undesirable.

The command syntax is shown below:

  --- MAIL --- SMTPfamily ---+-------------+------------------|
                             +-- <family --+

The SMTP family is changed immediately and may only be used when the SMTP mail queue is empty (as seen by MAIL SHOW QUEUE). A default value will be automatically assigned using the SL family when this version of MAILLIB is first used. The current setting can always be viewed in a MAIL STATUS response. To assign an alternate family:

  MAIL SMTP CDIMAGE

      --- MAIL Library response ---
  Default family for SMTP files will be CDIMAGE

In addition, MAILLIB now checks all SMTP file writes to the SMTP family ensuring that sufficient disk space is available. In the event that MAILLIB is unable to complete the write of a SMTP file because of disk shortage, the email request is aborted and the following will be seen:

13:34:31 Err:04480:Unable to create *SMTP disk file on CDIMAGE 13:34:31 Err:04480:Rejected (-112):Insufficient Disk space for *SMTP 13:34:30 Att:04480:Attached *BD/0029353/0029355/000OUTFILE ON DISK 13:34:30 Que:04480:Message Queued [194] (-999):Mail Server unavailable 13:34:30 Rcv:04480:To:Ian.Patterson@metalog.com,Subject:Print files

         from "Session" #29353

New MAILLIB errors have been included for the above:

 -112  Insufficient Disk space for *SMTP file
 -113  I/O error writing *SMTP file
 -114  SMTP family <family> Unavailable of Off-line

If error -114 is returned, this means that the Mail server is offline and the disk family used to hold *SMTP files is also unavailable. In this serious situation, all email requests are rejected and Virtual printing will fail.

An intermittent error where messages seen in a MAIL LOG response were truncated by two characters has been fixed. The LOAD NICK command will now not accept any FROM <filename> specification; MALLIB will only ever load a nicknames file called *METALOGIC/MAILLIB/NICKNAMES on the SL MAILLIB family.

Previously, it was possible for the CODESET setting to be reset back to ASCII if an international codeset such as LATIN1ISO was active and a LOAD USER or LOAD ADDR command was subsequently issued. This sequence caused MAILLIB's configuration to be corrupted and, the next time that MAILLIB was restarted, CODESET would be reset. This problem has been fixed.

MAILLIB will now ensure that any included ASCII or EBCDIC files do not pass invalid characters such as nuls (4"00") in the data passed to the SMTP server. Previously, such rogue characters could have caused email rejection.

MAILLIB is now protected against the possibility of MEMORY resize faults by callers of the GETMAILINFO library entrypoint when very large amounts of mail log data are being returned. In particular, this problem caused SUPERVISOR to fault with REQUESTED MEMORY SIZE GREATER THAN 65535 WORDS when handling a very large MAIL LOG request (please refer to SUPERVISOR DNote 530.83 for additional information).

The maximum number of lines permitted within the body of a mail message sent by the Opal MAIL function or MAILLIB's QUICKMAIL entrypoint has now been increased from 3000 to 6000 lines.

MAILLIB will now correctly handle data strings, either in body text or file records, that have a "." in character position 1. Previously, in such situations, the '.' character would have been discarded by the SMTP server because MAILLIB did not conform to RFC 2821 requiring mail clients to insert an additional '.' character at the head of the sending text.

MAILLIB will now correctly handle stream files which have Unix-style end-of-line delimiters (hex 48"0A").

The MODIFY command has been enhanced to allow the 'To:' address of certain kinds of queued SMTP files to be replaced by one or more alternative addresses. Typically, this could be used for a queued email that has been rejected because the recipient mailbox is suspended or unavailable for some reason. Emails that are queued because of server unavailability (i.e. -999) cannot be modified with an alternate email address.

To change the email address of queued transaction #63136:

   MAIL MODIFY 63136 support@metalog.com

MAILLIB will immediately perform a NUDGE so that the modified emaill will be immediately reprocessed. If the revised 'To:' address also fails, a SHOW QUEUE command address will also show the revised address.

The original content of each modified email will remain intact though any 'Cc:' or 'Bcc:' headers will be ignored. Additional information will appear in the MAIL LOG:

12:44:08 Snd:63136:ReSent OK to:ian.patterson@metalog.com 12:44:08 Msg:Processing #63136 SMTP file 'To:ian.patterson@metalog.com' 12:44:08 Msg:Trans #63136 *SMTP/SYSTEM-NODE/20080729/"fake@metal"/63136

         modified

The <Return-path> header in the redirected message will appear as <RedirectedMail@mydomain.com> where mydomain.com is the MAILLIB's local default domain.

The change described in DNote 530.03 caused TEST emails on most mail servers to fail with a RELAY RESTRICTED -530 error. This was because of the attempted append of Maillib's default domain to unqualified addresses which consequently caused problems with 'From:' specifications with spaces in the field.

Because of this issue, the use of a 'From:' header to participate in the 'MAIL FROM:' SMTP exchange will now be disallowed unless a full qualified email address is used.

MAILLIB now includes the 'Originator-info' header for all email requests. This field identifies the MCP system origin and is usually of the form 'UserCode@TCPIP.Domain.com'; the usage of this address is now limited by MAILLIB.

If a 'From:' header has been provided by the caller and MAILLIB has validated that the address domain is acceptable (i.e. maps to MAILLIB's 'Default Domain' or the local TCPIP domain), then this 'From:' address will be used in the 'MAIL FROM:' protocol exchange with the mail server. This change allows certain mail servers to report errors back to addresses other than the PostMaster (which is used in all other cases).

Further, if a 'From:' header is an unqualified email address with no '@ <domain>' component, MAILLIB will now append its default domain setting, if present, to the address. Nicknames cannot be used with a 'From:' header.

This change allows SUPERVISOR to retrieve MAILLIB run-time and configuration information into a small attribute subset called SYSTEM MAIL as described in SUPERVISOR DNote 530.71.

On-line HELP for the CODESET function, previously omitted, is now available. The main HELP screen for MAILLIB has been updated to include all command variants.

The changed described in DNote 510.10 caused a 'Sender:' header to be inserted into the body of a MAILLIB generated email if the domain of an user-provided 'From:' email address did not match the TCPIP domain of the MCP system. This behaviour has been modified to now omit the 'Sender' header if the domain of the 'From:' address additionally matches that of the DEFAULT POSTMASTER address.

MAILLIB's on-line HELP has been updated to included information about the new LOG implementation, described by DNote 520.05.

The active MAILLIB log file will now be closed when MAILLIB is terminated by a QUIT or THAW. Previously, the file remained opened by the MAGUS library until that library was itself terminated.

If the PDEF SUPPRESS is set to TRUE or the MAILLIB control ^SIG NONE is used then MAILLIB will now inhibit the generation of 'Attached #' messages in the email body. MAILLIB was already inhibiting signatures and 'Included #' entries with SUPPRESS set to TRUE.

When calling the MailMan entrypoint directly an ^Include option in the last row of the string array was ignored. When processing Ascii HTML files <br> strings were added.

The logging of MAILLIB transactions and messages has been significantly changed. Instead of the SYSTEM/SUMLOG, MAILLIB will now log information into a dedicated file called *METALOGIC/MAILLIB/LOG on a nominated family configured using the INSTALL utility. By default, this family is set to that of DL LOG.

The LOG command has been enhanced to allow varied access to the MAILIB log files; searching by category, wild card text patterns or time and date ranges is permitted. The LOG command extensions are shown below:

                   +<<--------------------------------------------+
  --- MAIL -- LOG -+--+--------------------------+------------+---+---|
                      +-- <Category> -------------------------+
                      +-- <Time/Date> --+---------------------+
                      |                 +--  -  <Time/Date> --+
                      +-- FIND ---- <Text Pattern> -----------+

<Category>

As in previous versions of MAILLIB, <Category> is a 3-character text value that filters the log search by message category. These categories are MSG,SND,RCV,ERR,LGC,QUE,ATT,INC and CMD.

<Date/Time>

  ---- <hhmm> --+--------------+--------------------------------+-----|
                +- <dd/mm/yy> -+-+------------------------------+
                                 +-  -  <hhmm> --+--------------+
                                                 +- <dd/mm/yy> -+

The FIND modifier permits the searching of each log entry for the specified text. The FIND automatically encloses the target with "=" at both ends of the text and is not case-sensitive, therefore finding both lower-case and upper-case matches.

The log files are automatically released when the file size exceeds 15,000 records or the LC CLOSE command is used. The LC command allows any comment command to be written into the current log. The command category for any LC command is 'Lgc'. The LC command is shown below:

  --- MAIL -- LC --+--- <text> ------+----------------------------|
                   |                 |
                   +--- CLOSE -------+

MAILLIB will now correctly assign a default Alternate PortNo assignment when a MAIL ALTSERVER command is used. Previously, for MAILLIB CONFIG files created by earlier versions of MAILLIB, the misleading display message "Invalid SMTP Alternate Port No:25" would be displayed during MAILLIB initialisation even though an alternate server was not currently assigned.

For print requests handled by MAILLIB's Virtual Print server, the default signature and 'Include #' and 'Attached #' messages will not appear in the email body if the PRINTDEFAULTS attribute SUPPRESSED is set to TRUE.

MAILLIB licensing has now been extended for sites that do not have full SUPERVISOR but are licensed for the TRIM module.

The change applied by DNote 510.13 caused the CODESET implementation to be broken for sites using CCSVERSION other than ASERIESNATIVE. This meant that text using a local odesets would not be correctly translated in email attachments or within the text body. This problem has been fixed.

Previously, MAILLIB did not always eliminate invalid characters (e.g. nulls) from symbolic or text source files which could have caused CR-LF pairs to be dropped from any generated attachment. Now, MAILLIB will replace invalid characters by '?' in the message body, includes or attachments.

The change described in DNote 510.11 could have caused some corruption of email headers passed to the QUICKMAIL entrypoint when using the MAIL function from SUPERVISOR and FLEX. This problem has been fixed.

The MAILMAN entrypoint in the MAILLIB library is used to generate and send emails from various origins including SUPERVISOR, Virtual Mail printing and user programs such as MAILER. However, using the MAILMAN interface directly now needs to be simplified to allow the planned use of the MAIL function into FLEX Opal scripting.

To enable MAILLIB calls to be easier to use, a new entrypoint called QUICKMAIL has been implemented. This procedure has the following declaration in MAILLIB:

  Real Procedure QUICKMAIL(MAILHDRS,MAILTEXT,OPT);
  Value OPT;
  Ebcdic Array MAILHDRS, MAILTEXT [0];
  Boolean OPT;
     Library MAILLIB;

The MAILHDRS and MAILTEXT array are used in the same way that the Opal MAIL function operates; that is, MAILHDRS holds a string of email headers whilst MAILTEXT hold multiple lines of text for the mail body each line delimited by a CR (48"0D"). Both arrays should be terminated by a NULL (48"00") character. All header contents are validated by MAILLIB and error values are returned if an illegal header is seen.

The OPTION parameter allows the user to specify additional controls and operate as in the MAILMAN entrypoint. Please refer to the MAILLIB Reference manual for more information. The OPTION parameter does not require any mandatory settings.

Example ALGOL program:

  BEGIN
     LIBRARY MAILLIB(LIBACCESS=BYFUNCTION);
     Real Procedure QUICKMAIL(MYHDRS,MYTEXT,OPTION);
     Value OPT;
     Ebcdic Array MYHDRS, MYTEXT [0];
     Boolean OPT;
        LIBRARY MAILLIB;
     Ebcdic Array HDRS,TXT[0:29999];
     Boolean OPTION;
     Replace HDRS By "To:Support@metalog.com;Subject:QuickMail";
     Replace TXT  By "First line",48"0D",
                     "Second line of body text",48"00";
     QUICKMAIL(HDRS,TXT,OPT);
  End.

Previously, if MAILLIB encountered a FROM specification in an email, the address was checked to see if it belonged to the MCP's local TCPIP domain. If the TCPIP domain and email address domain did not match, even partially, the FROM was modified to append the originator in the form of <usercode>@<TCPIP domain>. This measure was employed to identify email spoofing from MCP systems using MAILLIB.

This behaviour has been changed. If a FROM address is used and its email address does not match the local TCPIP domain, the FROM address is now not modified and MAILLIB will include a 'Sender:' header in the email body. This 'Sender:' header allows spoofing to be protected since many email clients will show this field when the email is displayed. The change permit spoofed emails to be traced and sent to the outside world with an intact FROM address.

The new behaviour is shown below:

MAILLIB From TCPIP domain Mail 'From:' Mail 'Sender:' a@bcd.com bcd.com a@bcd.com a@zzz.com bcd.com a@zzz.com usercode@bcd.com bcd.com usercode@bcd.com

On earlier versions of MAILLIB, the FROM address for a@zzz.com in the above example would have been automatically changed to:

a@zzz.com [USERCODE@BCD.COM]

Also with this change, MAILLIB will now assign the Default Postmaster email address, if available, for the normal 'Reply-To header' if one has not been assigned. Previously, it was usual that the usercode- TCPIP domain constructed mail address would have been used.

MAILLIB will now automatically append the default domain address to any unqualified Reply (RFC 'Reply-To') address, as with other provided email addresses. Note that MAILLIB will permit only one email address to be assigned to a Reply-to field and that MAILLIB nicknames will not be recognised.

MAILLIB will now always split text message lines in 998 character chunks as dictated by RFC 1521. Previously, MAILLIB would not have handled such lines correctly, causing them to be truncated. This problem is now fixed.

If an ALTSERVER assignment was in force and MAILLIB's primary server rejected emails with certain errors (because of illegal mailbox, unknown user etc) then MAILLIB would attempt to send the mail using the alternate server. This behaviour has been corrected; at this time it has been decided that the secondary server will not be used in these circumstances.

Print requests with only a SUMMARY file present and no valid NOTE or usercode information (i.e. '*') were incorrectly printed by MAILLIB's virtual print server. Instead of marking the print file as EXCEPTION, this error caused the SUMMARY file to be subsequently removed and could therefore not reprintable. This problem is now fixed.

After assigning an ALTSERVER server assignment (see DNote 510.05), it was possible for the default SMTP port number assigned to the new mail server to be set to 0. Now, the default value of 25 will be assigned automatically when ALTSERVER is used; note that this value can be overridden by the ALTPORTNO command.

If MAILLIB shutdown was initiated using an AX QUIT, the MAILLIB library would persist in the mix indefinitely if programs other than Metalogic software were still linked to the library. This behaviour, which prevented users from linking to a new invocation of the library until the old copy had terminated, has now been fixed.

MAILLIB will now permit the specification of a secondary or alternate mail server allowing redundancy if the primary server has failed or is off-line. The server may be specified by the ALTSERVER command and its SMTP port is assigned with the ALTPORTNO modifier. These commands are syntactically similar to the existing SERVER and PORTNO commands.

------ ALTServer ---+--------------------------+----------|
                            +----- <IP address> ----+
                            +----- <host name> ----+
                            +----------    -   ----------+

------ ALTPortno ---+--------------------------+----------|
                            +----- <integer> -------+

When an alternate server is first specified using ALTSERVER, a default value of 25 is assumed for the SMTP port number. This may be overriden by the ALTPORTNO modifier. The alternate server may be cancelled at any time using the '-' modifier i.e.

MAIL ALTSERVER -

When an email to the primary server fails and a secondary server has been assigned, MAILLIB will automatically route the email request to the secondary server without creating an interim *SMTP disk file. If both servers fail, the email will then be written to disk. Note that the primary email server will *always* be used first even if the last email was known to have failed.

This change supports the formatting of messages returned to the new MAILLIB MARC Directive.

Previously, attaching DATA files with certain large MAXRECSIZE values with FRAMESIZE=8 could have caused a SEG ARRAY ERROR @ 59310000 in MAILLIB. This problem has now been fixed.

Internal metalogic change to reflect MCP 51.1/HMP 10.0 compatibility.

For systems running with SYSOPS LONGFILENAMES set, MAILLIB was not correctly handling files with levels greater than 17 characters. This problem caused multiple MCP warnings to be generated handling file attachments and giving unexpected PC file names. This behaviour has now been fixed.

For plain text file attachments, MAILLIB would automatically segment individual lines greater than 255 characters causing spurious CR-LF markers to appear in the attached file. This behaviour had the side effect of causing some characters to be dropped from the segmented lines.

MAILLIB will now correctly segment plain text attachments, without character loss, for lines greater than 3000 characters.

Previously, MAILLIB would unconditionally add the default file extension (e.g. '.TXT') to any attached MCP file, even if that file already has its own PC extension on the MCP system. This behaviour has been corrected; MAILLIB will only append the default document extension if the PC file attachment does not already have one. MAILLIB still provides special handling for files such as wrapped containers, zipped archives and will append .ZIP or .CON if needed.

Previously, MAILLIB could receive an UNEXPECTED END OF FILE error when generating a file attachment from a FILESTRUCTURE=STREAM file that had a MAXRECSIZE greater than the value of 1. MAILLIB will now handle these stream files as if they were blocked, writing records into the attachment with length of MAXRECSIZE characters. Note that the MCP handles blocked stream files in the same fashion.

If, when performing a TT MAIL TEST, the user inadvertently transmitted the whole page which included garbage data, MAILLIB could fault with a SEG ARRAY error.

MAILLIB will now correctly handle ABORT commands; previously, due to changes in DNote 500.06, ABORT was being treated as a MODIFY. Both these problems are now fixed. The ABORT command has been enhanced to allow a transaction number list or the keyword ALL:

                             +<<<--- , ------------+
                             |                     |
---- MAIL ----- ABORT ---+---+--- <transaction> ---+-+-------|
                         |                           |
                         +--------- ALL -------------+

If 'ALL' is used, then MAILLIB will delete all queued emails (as seen in a SHOW QUEUE response) but any currently active transactions are NOT affected; these must be aborted manually.

For example:

MAIL ABORT 12345,12346,12388
MAIL ABORT ALL

Previously, MAILLIB would fault with a SEG ARRAY error if a queued email, with any individual message header (e.g. 'To:', 'From:') longer than 1800 characters, was subsequently re-processed. This problem has now been fixed.

If the SYSOPS option BACKUPFSDFLT is set to STREAM, then the system will create printer backup files with a FILEKIND of PRINTFILE and EXTMODE of EBCDIC, but a FILESTRUCTURE of STREAM. With such files, MAILLIB did not attempt to translate characters from EBCDIC to ASCII, causing the emailed files to be unreadable. This behaviour is now fixed

Both FLEX Inquiry and FAMILYMANAGER will now be able to call MAILLIB library routines without the need for a valid SUPERVISOR licence key. Work is currently in progress to implement MAIL support in the above programs.

The CONTENT-DISPOSITION header had been accidentally deleted from the set up of attachments; this could have caused some email clients (e.g. Eudora 6.0) to include the file in the body of the email instead of attaching it. This header has now been restored.

The LOAD NICKNAMES command would occasionally fault with SEG ARRAY ERROR when handling nickname definitions which had short names but long descriptions. This problem is now fixed.

Previously, if the MAILLIB library received multiple TEST email requests in a short period of time, it was likely that only the first TEST would be handled correctly. This behaviour, which could occur on very busy systems or with slow TCPIP connections to the mail server, has now been corrected.

The header keyword 'Subject:' may now be truncated to 'Subj:' in a Virtual Mail server request.

Previously, when the TEST command was used with more than one email recipient, the command could have caused the SUPERVISOR call to fault with a SEG ARRAY error. This problem is now fixed and multiple recipients will be handled correctly.

The fix applied by DNote 490.08 caused ^INCLUDE and ^ATTACH directory commands to fail with GETSTATUS SOFTERROR 123. This unintended side-effect is now fixed.

The COPYBD option discussed in Dnote 490.06 did not work correctly for print requests created by non-privileged usercodes and the attempted COPY for each file would fail with security violations. This oversight has been corrected.

MAILLIB change 490.06 caused some command response headings to be incorrectly displayed. This problem has been corrected.

OUTLOOK marks the incoming email with a red exclamation mark.

Previously, MAILLIB did not preserve backup files printed by the Virtual Mail Server regardless of the settings of the PS DEFAULT PRINTCOMPLETIOn and PS DEFAULT PRINTRETENTION settings. This meant that print requests could persist in the PS SH COMPLETED list but their backup files would have already been removed. The only way to prevent this behaviour was to mark the request or individual files with SAVEBACKUPFILE set to TRUE.

For those sites wishing to preserve requests with PRINTRETENTION set, a new command has been implemented, OP, which will allow global MAILLIB options to be applied. Currently, only one option exists - COPYBD - which controls MAILLIB backup file handling behaviour.

--- MAIL -------- OP ------+--------------------------+--------|
                           |                          |
                           +--- - ---+---- COPYBD ----+
                           |         |
                           +--- + ---+

When the COPYBD option is set, all print files printed by MAILLIB's Virtual Mail server will be first copied as a new file with the prefix MAILBD as the first level of the name, unless this first level is a usercode. When MAILLIB has copied the file, the file will be printed and then automatically removed. The original backup file is then handled by the Print System according to the PRINTCOMPLETION and PRINTRETENTION and will be removed or retained for the appropriate time.

This procedure has been adopted because it is not practical to retrieve the system settings of PRINTCOMPLETEION and PRINTRETENTIOn because the mail server interface is not presented with these settings by the PRINTS software.

Two new ^OP variants are now available for use with the Supervisor OPAL MAIL function and programmatic calling of the MAILMAN library entrypoint.

^OP READRECEIPT will generate a 'Disposition-Notification-to' entry in the body of the email triggering the email client to request a read receipt, if supported by that email client.

The ^OP PRIORITY HIGH and ^OP PRIORITY LOW variants allow an email to be sent with an 'Importance' header of HIGH or NORMAL. A setting of HIGH may be interpreted in various ways by different email clients e.g.

OUTLOOK marks the incoming email with a red exclamation mark.

The new ^OP modifiers are shown below:

 --- ^  OP -------+--- READRECEIPT ----------------+---------------|
                  +--  PRIORITY ----+-- HIGH ------+
                                    +-- LOW -------+

As with all ^OP modifiers, the ^ must occupy the first character of any new line.

The MIME content specifications for the transmission of Unisys wrapped containers caused the attachment to become corrupted and unusable if sent to a Mac running Eudora. Although the incorrect specs did not seem to cause problems on PC's, the correct specifications have now been applied.

Also, for emails with HTML content in the body of the message, MAILLIB was not resetting its internal HTML flag by the time any attachments were being processed. This meant that, depending on the email client, there could appear instances of the string '<BR>' after the signature.

This problem is also fixed.

MAILLIB will now give a Read Timeout error whilst waiting for input from the specified mail server for more than 60 seconds. Previously, it has been reported that MAILLIB can hang indefinitely with a connection open to a mail server that has died or hung. In such cases, the TCPIP connection persisted and could not be closed unless the caller was DS-ed. Now, such hung emails will be timed-out after 60 seconds and will be marked with an error of -554, which requires a MODIFY command to allow re-processing.

Previously, the only way to assign an email address to any print request handled by the Virtual Mail Server was to assign a 'To:' header in the NOTE attribute. Now, if the 'To:' header is absent, MAILLIB will look at the EMAIL attribute associated with the owning usercode in the USERDATAFILE. For example, the MAKEUSER input for an existing usercode would be:

USER META EMAIL="BOBNIAN@METALOGIC.EU.COM"

If a 'To:' header is present in the NOTE attribute, this value will always override the USERDATAFILE setting.

MAILLIB will now correctly handle the printing to email of an empty backup file; previously, the Virtual Mail server would have failed with a I/O ERROR: END OF FILE and disabled the printer device. This problem is now fixed.

Also, it was possible for a SHOW QUEUE command to fault if any of the queued emails had a corrupted subject line. The cause of the corruption has been fixed.

The ^ATTACHBD and ^INCLUDEBD functions will now correctly process the *BD files for the given job number. Previously, MAILLIB would refuse to find any files in the directory. In addition, MAILLIB will now provide extra information in the email if a file search fails.

For programmatic calls of MAILLIBs' MAILMAN procedure, the 'To:' parameter (ML_TO) was not being preserved for further MAILMAN calls by the same program, which could cause these later calls to fail. The ML_TO parameter is now left untouched by MAILMAN.

MAILLIB now permits email addresses that include the apostrophe character as well as several other RFC-822 permitted characters that were previously omitted.

MAILLIB will now better handle certain types of emails that originally failed with an error that may be retried. Typically, this requires some form of operator action to resolve the problem ,e.g. a -552 error means that the server does not have enough storage for the mailbox specified in a 'To:', 'Cc:' or 'Bcc:' address. In this case, MAILLIB will now queue the email and it will appear 'Marked as exception' in the response to a SHOW QUEUE command.

Assuming that mailbox storage is now available, the email may be scheduled for re-processing by using a new command, MODIFY:
TT MAIL MODIFY 12345

where 12345 is the email transaction number. This resets the last to error flag and allows the email to be processed again.

Any queued email that has a 'retryable' error will appear in the response to a SHOW QUEUE command:

TT MAIL SHOW QUEUE

--- MAIL Library response ---
--- 1 Queued MAIL Messages ---
#1162 *SMTP/IPP/20030516/"MAILTEST@metalog."/01162 160.4 Kb
Subject:Print files from *SYSTEM/PRINT/BACKUP/PROCESSOR #709
Mail marked exception (-552)

Retryable errors include -552(mailbox storage exceeded), -554 (failed transaction),-452 (insufficient storage) etc.

Also, using the TEST command with multiple long email addresses could have caused MAILLIB to fault with a SEG ARRAY error. This problem is now fixed.

If the 'To:' part of a MAIL message header consisted of extraneous delimiters e.g.:

'To:ian@metalog.com,,bob@metalog.com,,,'

then MAILLIB would go into a processor loop scanning the multiple commas. This problem is now fixed; MAILLIB will ignore any extraneous delimiters(,;) in messages headers.

On systems where the TCPIP domain name consists of only one level (instead of 'a.b' or 'a.b.c'), any mail sent from that system, even with a full email address, would fail because the internally generated MAILLIB-supplied FROM address consists of the form:

test@hostname

which will be rejected by all mail servers. MAILLIB will always verify email addresses against the TCPIP domain name to prevent spoofing. Even an override of the FROM address does not work because MAILLIB does not permit an alternate REPLY or FROM if the domains are radically different. For this reason, it is desirable that the TCPIP default domain has multiple levels (e.g. POWEREDGE.METALOG.COM).

To resolve this issue, MAILLIB now determines if a single-level TCPIP domain exists; if so, MAILLIB will override this with the default assigned domain name (as given by MAIL DOMAIN command). This allows emails to be verified by MAILLIB and the mail server using a valid y domain.

The IPADDRESS command has been implemented to allow a site with multi-homed IP addresses, to configure a default MYIPADDRESS setting for all SMTP connections to the network mail server. The IP address must be valid for the Clearpath sending emails or indeterminate behaviour may result.

The syntax is shown below:

      ---- IPaddress ------+-----------------+------------------!
                           +--- <IP addr> ---+
                           +---   -   -------+                      

On occasion, even though a port connection has been established with the specified mail server, it is possible that the server would refuse to initiate the dialog. This caused the MAILLIB caller to hang indefinitely leading to subsequent requests issued to the same caller (e.g. Supervisor) queuing requests. It is not known why some servers remain in this state but because this problem has been observed with different mail servers and operating systems, it is likely to be a or TCPIP/networking issue.

MAILLIB will now wait 60 seconds for the dialog to commence after the SMTP connection has been made. If the dialog does not start, MAILLIB will automatically queue the email in progress and return a -999 result. The MAILLIB log will reflect a 'Read Timeout' error in such circumstances.

MAILLIB's default signatures have been modified slightly to now include the transaction number allocated during the email generation.

MAILLIB will now correctly show the last line (#900), if available, in any email sent by the OPAL MAIL function.

Some minor changes have been made to MAILLIB's debug facilities.

MAILLIB now has options to allow callers to change the PC name format of attached files and to control trailing spaces removal. These features can be utilised in several ways using a new ^ modifier and run-time file-equation.

The OP modifier has 4 settings:

    -- ^OP -+-- LONG ------+-------------------------------!
            +-- SHORT -----+
            +-- COMPRESS --+
            +-- NORMAL ----+
            +-- RESET -----+ 

^OP LONG
^ATTACH *DATA/TEXT/EXAMPLE ON DEV

would generate the file name 'DATA-TEXT-EXAMPLE.TXT' instead of 'EXAMPLE.TXT' which is the normal behaviour. This can also be provided by the MAILER utility by setting the FLEXIBLE attribute to any file-equated ATTACH file:

RUN *METALOGIC/MAILER("ian@metalog.com","TEST");
FILE ATTACH(TITLE=*DATA/TEST/EXAMPLE);
FILE ATTACH1(TITLE=*DATA/MYPC/NAME, FLEXIBLE);

In the above case, the long PC name format only applies to ATTACH1. The ^OP SHORT command is used to return to the normal single-level PC name.

Use of ^OP COMPRESS allows the removal of trailing spaces for file attachment an includes. In the case of ^ATTACHs, only DATA files are eligible for space removal whereas any blocked, text-based file used with an ^INCLUDE can be processed. This is because many text file kinds have sequence numbers which are preserved in MAILLIB attachments.

An ^OP NORMAL command disables the space removal capabilities. Note that, once set, any ^OP command will apply to all ^ATTACH and ^INCLUDE statements until it is disabled. Using the TRIMBLANKS file attribute, the MAILER utility can be persuaded to perform space removal for individual files:

RUN *METALOGIC/MAILER("ian@metalog.com","TEST");
FILE ATTACH(TRIMBLANKS,TITLE=*DATA/TEST/EXAMPLE);

The ^OP RESET command will return both controls to their original settings; please note that, currently, there is no way to alter the global behaviour of file-naming and space removal. Also, the Virtual Mail server facility is unaffected by both mechanisms though the actual TRIMBLANKS setting on a print file will be controlled by the PrintS System in the usual way.

For users calling MAILLIB directly, both these options can also be globally set using the OPTION parameter in the MAILMAN call. The following single bit fields controls are shown below:

LONGNAMESF = [0:1]
TRIMBLANKSF = [1:1]

For example, in ALGOL:

OPTION:=* & TRUE LONGNAMESF & TRUE TRIMBLANKSF;
MAILMAN(ML_TO,ML_CC,ML_BCC,ML_FROM,ML_REPLY,ML_SUBJ,ML_TEXT,OPTION);
....

Also, MAILLIB now retains the original case of any user-provided PC names assigned to an ^ATTACH command; previously, the name was always unconditionally up-cased.

Previously, it was possible for MAILLIB to reject the printing of a perfectly valid Prints Request with the error:

Rejected print req #7595 Job #37841 Maxlines exceeds 0.0 Kb

This only occurred if message limiting controls by usercode were in force and the NOTE attribute was not set up for the first file to be be processed (usually a SUMMARY file) by the Virtual Mail server. Depending on the level of PRINTS, it is not uncommon for a SUMMARY file to not have a NOTE even though other files in the request are fine.

Now, MAILLIB will unconditionally determine any usercode message limits by checking the owner of the SUMMARY file in the request, regardless of the presence of a NOTE entry.

Previously, MAILLIB was inocrrectly handling the attachment of blocked DATA files with FRAMESIZE=48; these files were being treated as if they had FRAMESIZE=8 and records were consequently truncated. This problem is now fixed.

At sites where MAILLIB uses alternate codeset translation, text files processed by ^ATTACH or ^INCLUDE will now be correctly translated. Previously, this was not being performed in all cases.

Some inconsistencies in the ^ATTACH handling of directories, with or without *, usercode or 'ON family' have been resolved.

Lastly, MAILLIB used to abort a Virtual Mail print request if any associated SUMMARY file did not have an assigned NOTE attribute. There are several ways this situation could arise: for example, if PS DEFAULT JOBSUMMARY was set to UNCONDITIONAL and a print request was generated by a SPLIT from a CANDE session. This problem is also fixed.

When an ^ATTACH request is processed for a directory of files, MAILLIB will now exclude any file that matches the original selected directory.

For example, ^ATTACH A/= will now omit the file 'A' if one exists but will process all other files in the same directory.

During initialization, MAILLIB attempts to load the default NICKNAMES file. If the file was not present, MAILLIB would erroneously report an OPERROR #2 display message. Since the presence of a NICKNAME file is optional, this behaviour has been suppressed.
Also, under certain circumstances, it was possible for print requests routed to MAILLIB's Virtual Mail system to be marked as EXCEPTION with an accompanying error message of:

"Rejected print request nnn:Total Lines exceeds 0.0 Kb"

This problem is now fixed and appropriate error messages are also now written into the Mail log.

The REBUILD NICKNAMES command was not correctly loading all nick names from the MAILLIB nicknames file. Loading a new file with this command may have caused an INVALID OP fault.

Also, MAILLIB would fault with an INVALID INDEX if the number of email recipients exceeded 100. Both these problems are now fixed.

Previously, the REQUESTNOTE attribute was used by the Virtual Mail server to allow users to override the default MAILLIB signature (by simulating a ^SIG operator). This feature has now been replaced.

Now, the PRINTCHARGE modifier should be used to override signature and REQUESTNOTE is now used to optionally supply an alternative subject. It should be noted that the signature title passed to PRINTCHARGE *must* start with a "^" or "]" character; any other text strings will be treated as conventional charge codes.

For example:

PRINTDEFAULTS=(NOTE="To:Support",PRINTCHARGE="^SUPPORT",                     
                  REQUESTNOTE="Test listings from live job");

Although a 'Subject:' header can be included in the NOTE attribute, REQUESTNOTE allows easier assignment of subject where NOTE is already pre-defined for the usercode in the Userdatafile.

The MAIL NEWS command has been deimplemented. The latest changes and fixes are fully documented in the *METANOTES/MAILLIB file supplied with the release and in documentation available via our website at http://www.metalog.com.

Lastly, MAILLIB will NOT now automatically assign a default 'Reply:' address if a valid 'From:' header has been already assigned. Previously, the default postmaster address would have been used automatically if no 'Reply:' had been assigned. Note that MAILLIB will only permit this behaviour if the 'From:' address is identified as a member of the default TCPIP domain. Users can still override this feature by using their own 'Reply:' headers.

MAILLIB will now recognise and process directory specifications passed by the ^ATTACH or ^INCLUDE commands; previously only single files were permitted. For example:

This will process all files in the FLEX directory under the caller's user usercode and family. As previously, MAILLIB will only ^ATTACH symbol files, containers, stream and zip files; ^INCLUDE will only handle stream and symbol files.

Please note that MAILLIB will handle as many eligible files exist in the directory - there currently is no enforced limit.

Also, a new MAIL command has been implemented called MINLINES:

  ---- MINlines -----+---------------+----------------------|
                     +-- <integer> --+ 

By default, all print file processed by the MAILLIB Virtual Mail server are automatically handled as file attachments, regardless of size. The MINLINES command allows the MAIL administrator to control this behaviour by assigning a minimum threshold value, in lines, for each print file. The number of lines is retrieved from the backupfile and if its value is less than the threshold, the file will be included instead.

By default, MINLINES is set to 0.

These controls are maintained using the LOAD USERLIMITS and LOAD ADDRESS commands:

-- MAIL -- LOAD -+-- USERlimits -+---+-------------------------+---|
                 +-- ADDResses --+   +-- FROM -- <filename> ---+ 

USERLIMITS command -> *METALOGIC/MAILLIB/USERLIMITS
ADDRESSES command -> *METALOGIC/MAILLIB/ADDRESSES

These files should reside in the same directory as the MAILLIB library codefile. If 'FROM' is used then any external file can be used. The filekind of the file can be any symbolic, data or text format. Unlike the implementation of nick names, the data from the USERLIMITS and ADDRESSES files are stored into the MAILLIB config and preserved over restarts.

The implementations are described below:

USERLIMITS file

Each record in the LIMITS file should appear as:

----+-- <Usercode> --+----+--- <Size in KB>----+-------------------|
+-- *DEFAULT* ---+ +--- UNLIMITED ------+
+-- SYSTEM-NODE -+

The <usercode> modifier should be a valid entry in the Userdatafile. The <Size in KB> can be any integer or real value and reflects the total permitted size for an individual email generated by the specified usercode.The *DEFAULT* modifier allows a 'default maximum' message size to be associated with all other usercodes *not* appearing in the USERLIMITS file. And, the SYSTEM-NODE identifier reflects any email generated by jobs with the '*' usercode. As part of this change, any emails generated by a '*' session used to be assigned a pseudo usercode of 'ODT' - this is now not the case.

Example:

META 4000
*DEFAULT* 1000
TAPELIB 2000

In the above case, all usercodes except META and TAPELIB, will be assigned a limit of 1 megabyte. If a *DEFAULT* entry does *not* appear in the USERLIMITS file then all other usercodes have UNLIMITED as their default.

When the LOAD USERLIMITS command has been performed, the revised limits active immediately and can be seen using the command:

TT MAIL SHOW USER

When in force, MAILLIB will attempt to calculate the size of the message including file attachments, includes and body text. With a mixture of include and attach, ^INCLUDE statements and body text are always processed first. MAILLIB will include a warning message into the body of the text if a file could not be processed because of limits.

The USERLIMITS mechanism also applies to email generated by the Virtual Mail server, In this case, MAILLIB uses the number of lines determined from each backup file, then multiplied by 132, to guess-estimate the msg size. Note that email clients such as Microsoft Outlook will often report emails as much larger than they really are.

ADDRESSES file

A similar mechanism exists for the ADDRESSES file which consists of a list of valid email addresses maintained by the MAIL administrator. MAILLIB will attempt to check each address for validity and, if a bad entry is found, the LOAD will be aborted.

MAILLIB will recognise certain wild-card characters to allow, for example, all users at a single domain to be permitted. The permitted wild-cards are '=', "#" and "?" which map respectively to any string (0..n chars), a single numeric character and a single character of any type.Example:

ian@metalog.com
bob@metalog.com
=@unisys.com
meta=@tesco.net

Here, the '=@unisys.com' entry will permit any email address in the unisys.com domain to be allowed. All other email addresses that do not match the above list will be rejected. By default, if no LOAD ADDRESSES command has been applied, all outgoing email addresses are considered valid.

The 'TT MAIL SHOW ADDR' command will show the current list of acceptable email addresses.

UNLOAD command

User limits and address valdation, described above, can be deactivated using the UNLOAD command:

 
-- MAIL -- UNLOAD -+-- USERLIMits ----+-----------------------------|
	               +-- ADDResses -----+

The selected control will be immediately disabled and apply to all new email requests. The change is also updated in the CONFIG file and applies across future restarts of MAILLIB.The usage of LOAD and UNLOAD commands are logged in the system SUMLOG and can be monitored.

When MAILLIB was processing a text of line, either from a file or by a direct call using the MAILMAN entrypoint, the call would fault with a SEG ARRAY error caused by text longer than 600 characters. This problem, which did not affect any Supervisor MAIL requests, is now resolved.

The transaction number of any failed or queued email was not being written into the SUMLOG in most cases. This oversight has now been rectified.

Further, under certain circumstances, MAILLIB might fail to find files specified by ^ATTACH requests if no 'ON <familyname>' was specified in the file title. Normally, MAILLIB uses the caller's user and family to locate the file. This problem would usually occur where multiple ^ATTACH or ^ATTACHBD requests were processed in the same MAILMAN call and would ultimately depend if any familynames were included in these requests. These problems are now fixed.

The CODESET feature implemented by patches 470.60 and 470.61 was inadvertently broken by recenet updates. This functionaility has now been restored.

DNote 470.63 inadvertently included extraneous command text in all emails that were generated by the Virtual Mail server. This text has now been removed.

After a restart of MAILLIB and the mail server was assigned to an IP address, MAILLIB would not correctly set up the SMTP port file attributes causing all emails to be queued. The problem could be detoured by assigning the mail server a resolvable TCPIP name or redefining the IP address as server This problem is now fixed.

This patch addresses a security issue with the MAILER utility. As a consequence, the MAILLIB library now has appropriate granulated privileges assigned to the codefile using the MP command.

MAILLIB will now allow ACCESSCODEs to be used as email recipients by the Virtual Mail server implementation. This is controlled by assigning the string "To:[ACCESS]" to the NOTE attribute instead of a simple email address. If the Virtual Mail server detects this and the task that created the print request has an ACCESSCODE, then this will be used as the email recipient.

For example, a print request is created by user META and Accesscode IPP and the NOTE attribute is set to "To:[ACCESS]". When printing, MAILLIB will consider the NOTE to be "To:IPP". For the email to be handled correctly, a default domain must have been previously set up so that the address is correctly expanded or IPP is a valid mailbox on the SMTP server or, lastly, IPP exists as a MAILLIB nickname entry.

After installation of MAILLIB 470.60, the default destination code set of ASCII would not have been correctly. This would have required manual TT MAIL CODESET ASCII command or a restart of the MAILLIB library (using TT MAIL QUIT and TT DELINK MAILLIB). If this was not done, sending any email would generate numerous translation warning messages. This problem has now been fixed; in addition, if MAILLIB detects any translation errors then a single warning message will be displayed and all further translation efforts will be omitted.

Previously, if MAIL SERVER was assigned to an IP address and then switched during the same session to a normal host name, any future emails would fail because MAILLIB would erroneously attempt to assign this hostname as an IP address. This problem is now fixed.

Lastly, if a new MAILLIB config was initialized, MAILLIB would not recognize any previously-assigned server provided by the Magus config variable MAIL_SERVER. This problem, which caused a default server name MAILSERVER to be assigned (easily overridden), has now been fixed.

MAILLIB will now, using CCS, allow a site to assign a default character code set to be used for translation purposes when sending mail form a Clearpath system. MAILLIB will now inspect the current CCS settings and determine the base code set in use on the system, for example, the SWISS CCSVERSION uses the LATIN1EBCDIC code set by default.

By default, where necessary, all EBCDIC-style data is translated to ASCII before being passed to the mail server. This default can be changed using the MAIL CODESET command:

   TT MAIL CODESET LATIN1ISO
   

Any CODESET changes are held in the MAILLIB config file and are held over restarts

MAILLIB can now use server names which include domain names and graphic characters such as "-" or "_". Previously MAILLIB expected server names to be simple alphanumeric strings

The ^ATTACH control function can now take an optional modifier allowing allows the user to determine the name of the attached file as it appears in a PC email client. The PC name should be a single-level name, ideally with a valid PC extension, should be encased with the control characters '[' and ']' and immediately follow the ^ATTACH modifier.

For example:

   ^ATTACH [MYFILE.LOG] *BATCH/FILE/LOG                                         
   ^ATTACH [RECORD.CSV] *SUPERVISOR/RECORD/MONDAY/4
   

For file attachments handled directly by the MAILER utility or the Virtual Mail server, the same behaviour can be implemented by the file equation of the FORMID attribute. FORMID must be set to a quoted string with an embedded '.' signifying a PC name. For example:

   RUN *METALOGIC/MAILER("SUPPORT@METALOPG.COM","MAIL FILES");                  
       FILE ATTACH(TITLE=DATA/CSVDATA,FORMID="DATA.CSV")

Assigning FORMID to a print request, with a DESTINATION set to MAIL, directly or by a PS MODIFY will have no effect. However, if you wish to use FORMID, it is important to set FORMID on the Virtual Mail printer device to *DONTCARE*; this allows PrintS to prevent any request to the device ignoring any FORMID:

   PS MODIFY MAIL FORMID="*DONTCARE*"
   

The changes described by Dnote 470.55 inadvertently broke the TT MAIL POSTMASTER command. Invariably, any use of the command would return a syntax error and corrupt any current post master setting. This fault is now fixed.

Also, if a queued mail, held on disk, was subsequently sent to the mail server but was rejected due to a bad email address, the SMTP disk file would remain on disk. Any subsequent re-scan of the SMTP directory by MAILLIB during a restart or in response to a MAIL REBUILD QUEUE command would cause the message to be re-processed each time. Now, MAILLIB will automatically change such SMTP files to BAD/SMTP

When using email addresses without a domain suffix (i.e. an address without a '@' component), MAILLIB was not attaching the default configured domain name which caused the email to be rejected. This problem is now fixed

MAILLIB will now accept comment information prefixed by the % character anywhere in a NICKNAMES file. All text following the % will be ignored.

For mail that could not be delivered (because of illegal email address, domain or unknown host), the bounced mail was being sent to the original Clearpath generated FROM address created from usercode and TCPIP domain name. This should only be the case if no POSTMASTER was established; if the POSTMASTER address is valid, this will be used as the MAIL FROM content in the mail dialog.

- ABORT command did not trap in-process emails that were being

- The SUSPEND&REJECT command can now be used when a SUSPEND is Nicknames or mailing list entries held in the MAILLIB NICKNAMES file can now be associated with a text description. For example, the mailing list SUPERLIST can be associated with the text "Supervisor Mailing LIST". This description is inserted into the message body as the "To:" header instead of, for example, SUPERLIST@METALOG.COM. The description must appear immediately after the name prefixed by a "=" and before the ":".

This patch addresses a number of faults that are now fixed:

  - ABORT command did not trap in-process emails that were being                
    read from a queued email.  ABORT could, in these circumstances,             
    fail to remove the *SMTP queued file
                                            
  - The SUSPEND&REJECT command can now be used when a SUSPEND is                
    already in force (and vice versa); no RESUME is required first
    

The description must appear immediately after the name prefixed by a "=" and before the ":". For example:

  STIRLING =Local Metalogic mailing list:                                       
     ian@metalog.com, bob@metalog.com;
     

MAILLIB was inadvertently disallowing lower case input via the TT MAIL command interface. Also, the number of nicknames loaded by a MAIL REBUILD NICKNAMES command, recorded in the SUMLOG display message, was usually incorrect.

Lastly, a nickname would not be correctly decoded if it was not the first address in the "To" list. This could caused the mail to be rejected with -550 error depending on the mail server. Also, MAILLIB will now insert text into the Comments header of the message if one or more MAILLIB nicknames have been decoded.