What's Up, DOCumentation? 2000 # 1

logo

February 2000

From:

Robert M. Green, President
David J. Greer, Vice President
Hans Hendriks, Editor pro tem

To: Users of Robelle Software

Re: News of the HP 3000 and of HP-UX, 2000 #1

What You Will Find in This News Memo:

MPE/iX 6.5 Is On the Way
Web Enabling Your HP 3000
The Computer is Missing! Tale of a Real Disaster Recovery
Qedit Scripting Solves System Administration Problems
Y2What?
TZ and the HP 3000's clocks (revisited)
Staff Spotlight: Jennifer Franklin
Suprtool Training March 13-14 in Oregon
Robelle Products, Problems, Solutions, Suggestions

Checking for Duplicate Records
Moving Big Data
Regular Expressions Do the Job

Another Happy User

This past year the Suprtool development team made two trips to HP in Cupertino to get very early access to MPE/iX 6.5, to test compatibility of the Robelle products with some of the changes CSY made to support files greater than 4Gb. Our visits were in collaboration with the CSY team, who identified Suprtool as the primary data extraction and manipulation tool for most of the HP 3000 sites requesting the enhancements that CSY included in this new version of MPE/iX. It was therefore critical that Suprtool support the new extended file structures by the time MPE/iX 6.5 reached Vab Prep status.

Suprtool version 4.3 and its associated programs (Suprlink and STExport) are capable of handling files greater than 4Gb. We have sorted, linked and exported files up to the 20Gb available on the test systems available, and have incorporated changes to support file sizes up to the new MPE/iX maximum of 128Gb. Suprtool version 4.3 is due for distribution in North America toward the end of April 2000, but if you need this capability now you can contact Robelle for the latest Suprtool pre-release.

With MPE/iX 6.5 no longer in VAB Prep status, the impending release of the latest version of MPE has an impressive list of new features, capabilities and most of all a huge increase in limits.

	Old Limit	New Limit
Main Memory	3.75Gb	16Gb
Disk devices per system	255	511
Disk devices per device class	255	511
DSTs per system	16384	65535
Ordinary fixed-length files	4Gb	128Gb
KSAM Files	4Gb	128Gb
VT Sessions	2000	2600
TIO Connections	2750	3300
TCP Connections	10240	20000
UDP Connections	4096	10000

Some of these limits are limited by the hardware you have, but the HP 3000 is definitely able to support your growth needs with this new version of the OS.

In order to support many of these new limits many other products, languages and utilities have also been enhanced:

FTP/iX supports large file transfers
C/iX now supports 64-bit Integers
Store/Restore now supports large files.

Over the next few editions of What's Up Doc we will investigate and report on more MPE/iX 6.5 enhancements, including the NEWCI command, the SDK for Java, LDAP, Apache, and much more.

[Neil Armstrong]

Web Enabling Your HP 3000

The CSY division of Hewlett Packard (the people who bring you the HP 3000) have published a White Paper called "Web Enabling Your HP 3000 Applications and Data Access". You can view the entire paper at http://www.businessservers.hp.com/solutions/internet/accesswp.html

The paper is approximately 50 pages long, so we will summarize some of the main points here. The paper identifies four major Web categories for HP 3000 applications:

"Webify" existing applications (leverage your existing investment by adding a browser interface to your current applications).
E-Commerce applications (typically those that involve multiple applications and which require fast time-to-market with a Web solution).
Enterprise solutions (where HP 3000 applications must fit into an overall corporate Web strategy).
E-services solutions (apps on tap such as Open Skies).

Depending on which category fits your situation, different software and solutions will be required to Web-enable your application. What is not mentioned in the White Paper is that HP suggests that to Web-enable either an HP 3000 application or an HP 9000 application, an N-tier approach should be used. In an N-tier implementation, there are at least three different computers that interact to provide a Web solution:

Client computer running a browser
Web server computer (the client connects to the Web server)
Database server (i.e., your HP 3000)

This solution is 3-tier, but other tiers (e.g., security servers, application servers, etc) could be added to the overall solution. There are a variety of technologies outlined in the White Paper that enable the middle tier (the Web server) to communicate in a bi-directional manner with the HP 3000.

If you are thinking of Web-enabling your HP 3000 application, we recommend that you review the White Paper. We are also available to share our extensive Web experience in helping you to find a solution for providing Web-access to your HP 3000.

[David Greer ]

The Computer is Missing!
Tale of a Real Disaster Recovery

The month of December was a busy one for the I.S. department at Robelle, as we were installing our new e-mail MS Exchange server.

All went fairly smoothly: setting up users, setting up and confirming a backup regimen, converting HP 3000-based e-mail data to The Microsoft Way, configuring the internet components, etc. We staged the process so that it took a leisurely two weeks to convert just about everyone. The last mail user, David Greer, would be converted after he came back from holidays in January.

Then disaster struck. The first Wednesday in January, thieves broke in to the Robelle office, stole a number of PC's, and our new e-mail server!

Because E-mail is core to our business, we had to act fast to fix our problem. We reverted all of the e-mail users back to using Internet Client mail connecting to our HP-UX POP server. We installed older machines on people's desks who were missing PC's. We contacted our Dell sales-rep right away, but the quickest that Dell could get replacements to us was in five to eight business days. You may recall that Robelle is situated in Western Canada. Dell assembles the PCs in Texas, then ships them by ground to Toronto. After being certified and Canadianized in Toronto, the systems are then shipped by ground out to the customer.

We then called a local supplier, Data Terminal Mart, who had put in a bid when we were initially looking for servers. DTM could get an HP Netserver out to us that evening. That sounded great! We went ahead and ordered the HP Netserver from DTM, while ordering replacement desktops from Dell.

The Netserver did arrive that evening, but too late for the local supplier's engineer to put all of the pieces together. He came back the next day and spent the morning configuring the RAID disk array, and then left us to install the operating system. The installation went very smoothly, up until the point of recognizing the DAT drive. The tech had installed it, but didn't test it. A quick inspection inside the cabinet showed the databus wire dangling from the DAT drive, sucking bits from air. After a quick call to DTM to verify where the wire should be plugged in, we were up and running. The Dell server and the HP server were almost identical in options, except that the HP came with a RAID disk array. There was also a difference in size: the HP NetServer weighs in at around 150 pounds, while the Dell was around 50. (A good thing, as it turns out.)

Disaster number two: Somehow, somewhen, during the installation of subsequent software on our original machine, our backup strategy for our e-mail server had been corrupted, and every single backup tape was blank. We'd been faithfully inserting backup tapes, but hadn't thought of regularly re- verifying that the tapes were still being written effectively. Groan! We re- configured our MS Exchange server and loaded everyone's e-mail back in, from Outlook ".pst" files saved before switching to MS Exchange, thus losing all messages from the three weeks we had used the new system. In retrospect, if the thieves had arrived a month later, things would have been much worse. One bright note: David Greer's e-mail hadn't been converted, so he didn't lose anything. :-)

We took extra precautions to ensure that the backup was working properly, and took a paranoid snapshot to move off-site, just in case the burglars came back.

We beefed up our physical security, installing additional burglar-proofing at the entrance to the office, and a steel accordion-style expanding security gate behind our regular front door.

The new desktops arrived within five days, and we were able to quickly configure them to the Robelle Common Operating environment. Things were back to normal, when ...

Disaster Stuck Again!

About three weeks later, thieves broke in again. They were in, out and gone in less than the five minutes it took the police to arrive. The bad guys crowbarred our new lock, broke the lock off the accordion gate, and stole off with four more desktops PCs: three fast Dell Optiplex machines, and a clunker P/166 that had been giving us trouble. This time they didn't take our e-mail server, probably because it was too heavy.

So we're once again replacing computers and beefing up our security. And of course we're verifying our nightly backups on our mail server.

[Ken Robertson]

Qedit Scripting Solves System Administration Problems

After our MS Exchange server disappeared, and we discovered that our NT backups were blank, it became obvious that we needed better control on those backups.

So I wrote the following QSL (Qedit Scripting Language) script to read the backup log files from our 3 NT servers, summarize them, then e-mail me the results.

The script is invoked from a nightly scheduled NT script which has 2 commands:

   qwin32.exe -r c:\robelle\qedit\user\scripts\backupcheck.qsl
   mailfile alpha paul "backup summary" c:\robops\log\backupck.log

The first command invokes Qedit for Windows and tells it to run the QSL script called backupcheck. The second command mails the summary file to me, using a WINBATCH script.

The QSL script is very simple:

opens a new empty file
```
   outfile=newfile();
```

opens the ntbackup.log file from one server

   file = open("\\alpha\c\robops\log\ntbackup.log");

calls a subroutine to save the lines with "Backup" and ":" in them

   Liststring(file,outfile) ;

The Liststring subroutine looks like this:

   sub ListString(file,outfile)

       list = {};
       list = list + file.fullfilename;
       list = list + "";

       outfile.insert(list);

       found = file.find( regexp:".*Backup.*:", startattop:true);

       repeat while found
           line = file.gettext( startline: file.lastfoundline );
           s = string(file.lastfoundline) + "??t" +  line[1];
           list = {};
           list = list + s;
           list = list + "";
           outfile.insert(list);
           writelog(s);
           if file.lastfoundline + 1 > file.linecount then
               found = false;
           else
               found = file.find( regexp: ".*Backup.*:", startline:
                   file.lastfoundline + 1 );
           endif
       endrepeat
   endsub

does the same for the other 2 servers

removes any null characters that NT backup insists on putting in its log files

   mynull = character(0);
   cleanup = outfile.find(string: mynull, entirefile: true, replacewith:" ");

saves the new summary file on one of the NT servers.

   Outfile.saveas( filename: "\\alpha\c\robops\log\backupck.log",
      forceoverwrite: true);

The complete QSL script can be viewed at http://www.robelle.com/support/qwin/backupck.txt

It's a good example of automating an operations task that requires some scripted text editing.

[Paul Gobes]

Y2What?

At Robelle, as with most of the world, New Year's Day came and went without any serious problems.

Suprtool logo Just a few calls during the first week of the year from customers who thought Suprtool would "automagically" know that "00" was greater than "99". For those customers we were able to show them the new $STDDATE function.

Another common call was from sites that were adopting the "AAMMDD" date format and needed to change their ITEM commands to reflect this.

As for own internal computers we had one software package that stopped, but after a quick call to the vendor, found out that they had already shipped us out a new version months ago. Within ten minutes it was installed.

We even had a one sharp customer (Rick Gilligan of CASE) find a Y2K problem in PROSE, the program that prints our manuals. It was printing the date as "04 Jan :0". Within a couple of hours Dave Lo had found and fixed that problem.

So, yes there was a lot of hype. But all the preparation work done by all of us in the computing industry made it into a non-event. Well done, everyone!

[Paul Gobes]

TZ and the HP 3000's clocks (revisited)

The article in the previous issue of What's Up Doc, regarding the TZ variable and the hardware and software clocks on the HP 3000, stirred up a few responses from our wise readership:

The article says that the only way to set the hardware clock is through the ISL CLKUTIL utility at boot time. This is not true; the SETCLOCK command also adjusts the hardware clock. Thanks to Gilles Schipper for testing and verifying this.
Jeff Vance pointed out that the sample job, if run daily (as mentioned in the article), would adjust the clocks on 1st April and 1st October, rather than the first Sunday of those months. The job should be run only on Sundays, or should include "and HPDAY=1" as a condition for execution.
The job shown was originally written by Shawn Gordon, and posted on HP 3000-L. Shawn's original post suggested running it weekly, on Sundays.

[Hans Hendriks]

Staff Spotlight:
Jennifer Franklin

Most customers who have had administrative dealings with Robelle probably know Jennifer Franklin. She has been handling customer accounts since 1989 and currently manages that department. Regular callers will have noticed that Jennifer has been notably missing in recent months. That's because she now also manages a brand new baby boy, Caleb, who is 6 months old. Jennifer has returned to work, but splits her time between working in the office and remotely from home.

She used to have other interests like playing baseball, doing yoga and working on projects around the house, but as anyone who's had a newborn knows, things change.

[Paul Gobes]

Suprtool Training March 13-14 in Oregon

The next off-site training session for Suprtool will be on March 13-14, 2000. Once again we will be hosted by Lund Performance Solutions in Albany, Oregon. Call Tammy at 1-888-ROBELLE or e-mail tammy.roscoe@robelle.com for details.

Topics to be covered by Hans Hendriks will include

Speed of Chain versus Get
Exporting Data to the World
Tables are like grocery lists
Linking data from multiple sources
Turbo-charging Quiz using Suprtool

Lund is holding System Performance Training around the same time. For more information call Lund Performance Solutions at 1-541-926-3800.

Robelle Products, Problems, Solutions, Suggestions

Checking for Duplicate Records

In December, Leonard Berkowitz posted the following question on the HP3000-L discussion list:

In Suprtool DUPLICATE ONLY RECORD:

If only one field is extracted to the output file, is the RECORD ONLY criterion applied only to that field or still to the entire dataset entry?

I replied as follows:

DUPLICATE ONLY RECORD is an output operation, so checks the output record for "duplicate-ness". As such, it checks only the fields included in the output record. This is different from using DUPLICATE ONLY KEYS, which forces Suprtool to check all the sort keys, irrespective of whether they're included in the output record.

So if you want to check the entire input record for duplicate-ness, but only extract a subset of fields, you can force this by declaring the entire record as a sort key, then using DUP ONLY KEYS:

   get D-SALES
   define entirerecord,1,80
   sort entirerecord
   duplicate ONLY keys
   extract ....etc.

[Hans Hendriks]

Moving Big Data

Robelle is in the business of working with large amounts of data. Generally, the larger the data source, the better Suprtool performs, relative to other serial retrieval methods. Historically, Suprtool is used to extract a subset of data from a large data source, which is then passed to another application for further processing. Traditionally, that's been another application on the same host computer. But for several years now, Suprtool has had the ability to prepare extracted data for use by non-HP 3000/9000 systems.

For example, let's say that your company's decision-makers are familiar with Microsoft Excel. Rather than generating voluminous paper or on-line reports, here's a real easy way of keeping them supplied with fresh data, which they can query in their tool of choice, thus removing their query overhead from your host machine:

Use Suprtool and Suprlink to extract the data fields they require:

   !run suprtool.pub.robelle
   base store
   get m-customer          {get customer records}
   sort cust-account
   output custfile,link
   xeq
   get d-sales             {get sales records}
   sort cust-account
   item deliv-date,date,yyyymmdd
   item purch-date,date,yyyymmdd
   item product-price,decimal,2
   item sales-tax,decimal,2
   item sales-total,decimal,2
   out salefile,link
   xeq
   link input salefile     {link customer info to sales info}
   link link custfile
   link output custsale
   link xeq
   exit

You now have the data you require, in a self-describing file. Use STExport to format this into a comma-delimited ASCII file, reformatting date fields into a format that Excel will understand:

   !run stexport.pub.robelle
   input custsale
   date DDMMYYYY "/"
   heading fieldnames
   output csalecsv
   exit

And this is what the resulting file looks like:

   "CUST-ACCOUNT","DELIV-DATE","PRODUCT-NO","PRODUCT-PRICE","PURCH-DATE",....
   10003,16/10/1999,50511501,98.31,16/10/1999,1,13.76,112.07,"Richmond",.....
   10003,16/10/1999,50512501,145.62,16/10/1999,1,20.39,166.01,"Richmond",....
   10003,16/10/1999,50513001,192.20,16/10/1999,1,26.91,219.11,"Richmond",....
   10010,20/10/1999,50533001,69.92,20/10/1999,1,9.79,79.71,"Vancouver",......
   10016,20/10/1999,50521001,24.59,20/10/1999,3,10.33,84.10,"Edmonton",......
   10016,20/10/1999,50532001,139.85,20/10/1999,1,19.58,159.43,"Edmonton",....
   .....etc

Notice that date fields have been reformatted to DD/MM/YYYY format, and decimals have been inserted into the "money" fields.

This file can be transferred to a network server, or to individual PCs, or, if you have your HP 3000 or 9000 configured as a Samba server, just moved to the appropriate directory.
Doing simple File/Open in Excel, specifying file type as ".csv" loads the file directly.

[Hans Hendriks]

Regular Expressions Do the Job

"Any sufficiently useful regular expression will resemble random keystrokes" - Dave Lo

We discovered some web pages that contained incorrect HTML. There were no quotes around anchors references, which is technically wrong. For example, <a href=file.html> instead of <a href="file.html">.

A single Qedit change command, using regular expressions, fixed this problem:

   search for  : <a (.*)=([^>]*)>
   replace with: <a \1="\2">

Explanation of each part of the regular expressions:

Portion	explanation	matches
`<a`	Start of anchor tag	`<a`
`(`	Remember 1st () expression which is
`.*`	a sequence of any characters	`href`
`)`
`=`	... until an equal sign	`=`
`(`	Remember 2nd () expression which is
`[^>]*`	a sequence of any characters except for a >	`file.html`
`)`
`>`	... until the closing tag	`>`
`\1`	1st () expression	`href`
`\2`	2nd () expression	`file.html`

Like I said, regular expressions may look incomprehensible, but they are a very powerful feature of Qedit.

[Dave Lo]

Another Happy User

Early in January, we received the following message from Randy Medd in our Support mailbox:

Hans/Francois/Paul/et al: Since I only normally communicate with y'all when something goes wrong, I thought I'd pass on a success story. This may be trivial to you guys.... Our PBX system was updated on 12/8/99, and the format of some of the SMDR records changed. I needed to change the file, inserting a blank in each record (at column 28), but only on records where there originally was a non-blank character in column 28. In Qedit, I used:
      c28' ' ' '(28/28 nom)1/1405
and I was done. Not exactly the King's, (uh, Queen's) English, but it did the trick, first time. Not to mention our having to patch Unison's (Tivole/ROC) Backpack for MPE/V. The built-in SYSDUMP code was performing a future-date dump for 12/30/99, which wasn't going to work quite correctly after last weekend. Simply texting-in the file, and doing
      cq"12/30/99"12/31/26"@
did the trick.

Hey, it's nice to get some good news too, sometimes. :^)

[Hans Hendriks]

Contributors to This Issue

In alphabetic order:

Neil Armstrong, Product Manager
Paul Gobes, Technical Support Manager
David Greer, President
Hans Hendriks, Technical Support Engineer
Dave Lo, Technical Support Engineer
Ken Robertson, IT Manager