What's Up, DOCumentation? 2001 # 1
January 2001
From:
- Robert M. Green, President
- Mike Shumko, Editor pro tem
To: Users of Robelle Software
Re: News of the HP e3000 and Robelle, 2001 #1
What You Will Find in This Newsletter:
- Suprtool's Powerful New String Functions
- Cleaning Up E-mail Addresses in Ecometry, by Chris Bartram
- WOW! 17 X Performance Improvement with Suprtool!
- Schedule: February Is Action Packed
- An HMO Tries the "e" in "e3000", by Eben Yong
- Robelle-L Discussion Group Has Moved, Now on Web
- Using Qedit For Windows to Test Suprtool Jobs
- Robelle Website Now Changes Daily
- Is Your Disc Being Eaten By MPE?
Suprtool's Powerful New String Functions
With Suprtool's extract already being so fast, and new machines being more powerful all the time, our next focus is to reduce the number of passes that you need to do to get your data into the format you want. So in pre-release version 4.3.03, Suprtool has some powerful new string handling capabilities that users have been looking forward to for some time. These allow you to trim blanks around string values, concatenate strings, and upshift or downshift values, all in a single task.
A simple example is the formatting of an Address line out of three separate fields, with one Suprtool command and in one pass.
The sample file with address information:
>form testaddr
File: TESTADDR.GROUP.ACCT (SD Version B.00.00)
Entry: Offset
NAME X30 1
STREET X20 31
CITY X20 51
PROV-STATE X2 71
COUNTRY X20 73
Limit: 10000 EOF: 15 Entry Length: 92 Blocking: 44
You can now combine fields and constants with the spaces trimmed with one single extract command, as follows:
>input testaddr >define address3,1,40 >extract name,street >extract address3 = $trim(city) + "," + prov-state + " " + $trim(country) >numrecs 1 >list >xeq
The result is a neatly formatted address line ready for printing on labels.
>IN TESTADDR.NEIL.GREEN (0) >OUT $NULL (0) NAME = Neil Armstrong STREET = 123 54th St West ADDRESS3 = New York,NY U.S.A
Note that if the output had been sent to a self-describing file, the field "address3" would have contained the result of the expression. This technique can be used in a range of applications, including date reformatting, firstname and lastname concatenation (with intervening blank) for generating form letters, etc, all in a single task.
The above example shows the use of the $trim function which removes both leading and trailing spaces from a field. The $ltrim and $rtrim are also available; these functions remove leading and trailing spaces respectively.
Two other string functions have been added, for upshifting or downshifting strings:
- $upper {upshift}
- $lower {downshift}
Say for example you wanted to find all the customers in New York City, but your data has combinations of New York, NEW YORK and New YOrk. You can find all of these records by doing the following:
>if $upper(city) = "NEW YORK"
These new functions can be used in the If and the Extract commands, and add a whole new dimension of functionality to Suprtool, by allowing you to do much more work in a single pass.
Please contact Robelle at 1.888.robelle or e-mail support@robelle.com if you'd like us to send you a pre-release version of Suprtool 4.3.03 with these new "string" functions. These new functions will also be included in the next general release of Suprtool, version 4.4, which will be sent out to all supported customers in Spring 2001.
[Neil Armstrong, Robelle Senior Programmer]
Cleaning Up E-mail Addresses in Ecometry
by Chris Bartram, U.S. Mint
Thought I'd pass along this script that scans the e-mail addresses in
CUST-XREF, looking for junk (like zeros, NONE, NA, etc) and getting rid
of them. It then does a second pass that checks the remaining addresses
for valid syntax (using a pattern match in Suprtool) - i.e.
something @ something . something ... not perfect, but it
will find most badly formed or mistyped addresses.
This script e-mails the list of CUST-EDP records to a contact list; easy enough to just print it or not as you want (replace the !xeq sendhtml line with a PRINT command or something). The lists are left on the system as the files 'NULLMAIL' and 'BADEMAIL' in ASCII format.
The script is written for the Ecometry application, but the techniques would apply, with revisions, to other applications as well.
!JOB fixemail,jobs.sgaii
!continue
!purge nullmail > $NULL
!SUPR
base macord,1,passwd
get cust-xref
define email,xref-no[3],47
define emtype,xref-no[1],2
if emtype='EM','EU' and (email='NONE ','N/A','NA ','0 ','00 ',&
'000 ','0000 ' OR email=='00000@')
extract cust-edp
extract email
sort cust-edp
duplicate none keys
output nullmail,ascii
delete
xeq
exit
!IF FINFO("NULLMAIL","EOF")>0 THEN
! sendhtml.xeq.threek ^macsl2.data.sys,,,NULLMAIL,&
! 'Improperly entered Email Addresses - deleted'
!ENDIF
!continue
!purge bademail > $NULL
!SUPR
base macord,1,passwd
get cust-xref
define email,xref-no[3],47
define emtype,xref-no[1],2
if emtype='EM','EU' and email><'?@&@?@.?@'
extract cust-edp
extract email
output bademail,ascii
xeq
exit
!IF FINFO("BADEMAIL","EOF")>0 THEN
! sendhtml.xeq.threek ^macsl2.data.sys,,,BADEMAIL,&
! 'Badly formatted Email Addresses - need to be corrected'
!ENDIF
!EOJ
Note: Doing a DUPLICATE NONE KEYS and a DELETE in the same pass, as shown above, does work, but may not be best practice. Robelle generally suggests that users don't archive and delete in the same pass, because it's not recoverable (i.e., if the job is aborted, records are deleted but you don't have the archive copy). Better practice is to archive in pass one, and once you have the secure copy, go back and delete from the dataset in a separate pass.
I was toying with figuring out how to check the domain names in the e-mail addresses against the Internet to see if they are valid. That is described in part 2 of this article.
[Chris.Bartram@usmint.treas.gov]
17 X Performance Improvement with Suprtool!
Recently, Robert Allen Green, of Long Reach, Inc., Houston, evaluated our Suprtool product on MPE, and was impressed with the results...
Robert took one of their existing monthly reports, "Activity to closed Work Orders", which uses Query to access 2 datasets in a database, selecting on date and order-status, and replaced it with a Suprtool job.
The Query job was run monthly, and took 2 hrs and 18 minutes to run. It also didn't include component costs, so users would have to manually lookup the part costs to compute the standard cost of each order. In his Suprtool replacement, Robert included tasks to link in the component costs, so that the final report includes the computed cost per order. And, to top it all off, Suprtool creates an ASCII output file that can be imported directly into an Excel spreadsheet by the user.
Total time to execute the Suprtool jobstream? 8 minutes. That's a 17-times performance improvement!
Way to go, Robert!
[Hans Hendriks, Robelle Technical Support]
February Is Action Packed
There are four major e3000 related events in February 2001:
- SigSoftVend, a meeting of vendors with HP
- Solutions Symposium: training for new and intermediate users
- SIG/3000, a meeting of technical special interest groups such as SIG/IMAGE
- Performance Training Seminar in Florida
Robelle plans to attend all four. Here are the details of two of them:
Performance Training Seminar in Florida
This seminar hosted by FloRUG on February 13-16, 2001 offers four full days to rub shoulders with the top e3000 performance experts in a more intimate setting than the big conferences. Robelle's resident Suprtool performance authority, Neil Armstrong, will be speaking Thursday afternoon from 1:30 to 3:00. Neil describes his talk, entitled
"I/O Performance: Living with 400 I/Os Per Second""The slowest link in the performance chain will always be physical mechanisms, i.e. disk and tape drives. Where CPU transactions are measured in millions or billions of instructions per second and logical memory I/O's measured in tens of thousands per second, physical I/O's are still measured in dozens, or perhaps hundreds. I'll explain software and database strategies for improving I/O performance, and also detail my explorations of HP's new PCI backbone and what it suggests for increased I/O performance."
Robelle will also have a booth at the show where Neil will be available to answer your performance questions. And if you have Suprtool, bring any Suprtool questions along as well. We look forward to seeing you there.
Visit the FloRUG web site to see the program schedule, abstracts of the sessions, hotel form, and the seminar registration information.
Increase Your Skills at the Solutions Symposium
Mark February 7-10, 2001 on your calendar for the second HP e3000 Solutions Symposium, an event that offers 4 days of extensive technical training for e3000 System Managers, Programmers, Webmasters and Application Developers. Paul Gobes, Technical Support Manager of Robelle, attended the Symposium last year and sums it up as "great talks by sharp people". This year Paul will be giving three tutorials:
IMAGE PROGRAMMINGThursday 8th, 1-3pm
Audience: Application Developers
Topics Covered: IMAGE Fundamentals, Creating Databases, Intrinsics, Programming Considerations, New Features of Image, Tools and Utilities.
Abstract: IMAGE is the rich, functional database management system that is at the core of almost every application developed for the HP e3000. Programmers with any level of application development experience will benefit from this insider's look at the theory and practice of programming IMAGE.
DATABASE PERFORMANCE ANALYSIS USING HOWMESSYWednesday 7th, 11-12am
Audience: System Administrators and Developers
Topics Covered: Migrating Secondaries, Interpreting HowMessy Reports, Masters and Detail Issues, DBGeneral Reports, Repacking Detail sets.
Abstract: IMAGE databases are inherently messy. Their tidy exteriors sometimes conceal nasty surprises that can hamper performance and reduce the speed of retrievals and updates. But armed with the right tools and know-how you can quickly bring order to the chaos. The HowMessy program from Robelle can report on the internal inefficiencies of your IMAGE database, to help you determine why performance is not what it should be. Then you can use Adager or DBGeneral to apply the changes that will reorganize the data along the most efficient lines.
SHARING HP e3000 DATA WITH THE WORLDFriday 9th, 3-4pm
Audience: Application Developers, Webmasters, and System Administrators
Topics Covered: Direct Access, Publishing on the Web, Mapping Drives, Exporting and Converting Data, Transport Methods.
Abstract: No HP e3000 is an island. The data that lives on your e3000 needs to be shared with applications that reside on other machines, platforms, and networks. The "E"-revolution means that you need to make your e3000 data available to the rest of the world. Learn how you can be the hero of your workgroup, department or company.
An HMO Tries the "e" in "e3000"
by Eben Yong, Health Plan of San Mateo
The Health Plan of San Mateo (HPSM) first began working with the "e" portion of the e3000 approximately 6 months ago.
It began with my desire to simplify some of the IS Operational aspects of our environment. As a California Medi-Cal HMO with a duty to report consistent and accurate data to the Department of Health Services of California, HPSM has fairly large number of different jobs that are set up to report various slices of claims data based on fiscal year parameters. These jobs are all run on a monthly basis to reflect changes in our claims data. Since it is our standard practice to retain seven years of historical data on our systems, each job essentially runs up to seven times each month (based on the request of our CFO). With approximately 50 jobs (and rising), up to 350 job streams may be requested during an end-of-month cycle!
Since all of these reports were using JOBBER as the input screen, all parameter input was not validated at stream time, and it caused problems when even one parameter was incorrectly entered.
Enter STREAMX. STREAMX allowed fully validated stream-time parameters and even reusable STREAMX modules. Jobs were rewritten to use STREAMX in place of JOBBER, and STREAMX modules were created to further simplify parameter entry. [Note: STREAMX is part of Security/3000 from Vesoft.]
Enter Apache. We installed the Apache webserver on our e3000.
Enter Perl. With invaluable assistance from Robelle, HPSM received the kickstart necessary to launch e3000 administration using Perl and Apache. We successfully created reusable Perl modules that allowed for extremely easy integration within our existing environment.
The end result of all this activity was about 20 (or so) different web pages that allowed the majority of the end-of-month jobs to be streamed with just a few mouse-clicks, and with the jobs having fully accurate parameters. We have one web page that has the ability to STREAM up to 49 jobs. This takes just a few seconds. Doing this manually would require between one and two minutes per job and introduce the potential for inaccurately entered parameters.
How has Robelle's Qedit for Windows assisted in all this?
It has been, in a word, a beautiful linchpin. This is because Qedit for Windows speeds up development time by allowing users to edit both POSIX, MPE and PC files within an integrated GUI environment. In the above scenario, the STREAMX jobs were MPE-based, while the PERL scripts were POSIX-based.
In August 2000, we implemented ReflectionWeb. In September, we implemented a full Virtual Private Network at the Health Plan. Both methods of remote access allow for authenticated, secure and encrypted access! With ReflectionWeb, we can access the e3000 using Internet Explorer (and Java); but with our VPN, we can now also access all of our protected systems (behind our Firewall and DMZ) that also have web interfaces. Qedit for Windows works onsite and remotely (through our VPN) and allows for much quicker viewing and troubleshooting of reports and $stdlists.
Again, all remote access is secure, authenticated and encrypted. This is very important for HMO's because of the new HIPAA regulations regarding security and privacy.
We are always looking for ways to further automate things, and we have done quite a bit using Visual Basic and Reflection1. We're now looking at Qedit's Scripting Language to see how it might fit into our framework. No development thus far, but it is in our future. Qedit for Windows truly simplifies our work and provides a great GUI editing environment for files from any source--MPE, POSIX or PC-LAN.
[eben_yong@hpsm.org]
Forum.Robelle.com
We have enhanced our Robelle-L discussion list with the ability to search and browse past postings via your web browser. This forum is a friendly place where users of Qedit and Suprtool can post tips and questions. For example, see the suggestion by Ole Nord on a simple way to delete related records from many datasets.
To webify the mailing list, we had to move it from our
Nitro.robelle.com server to another server. Please check
your e-mail address books: any references to NITRO should be removed.
If your address book has
robelle-l@nitro.robelle.com then it needs to be
changed to robelle-l@robelle.com
[Bob Green, Robelle President]
Using Qedit For Windows to Test Suprtool Jobs
I maintain a number of jobstreams which check the integrity of our production databases. They do this by executing Suprtool tasks which check for valid data values. These jobstreams have grown over time and may now contain over 1000 lines of intermixed MPE commands and Suprtool tasks. They also can take many minutes to complete and it is not something which should be tested during the day. So maintaining and testing them had become a big problem for me.
I found a technique that lets me change and test the jobs on the fly. The solution is to use the HOST command facility within Qedit for Windows.
I open my host jobstream in Qedit For Windows. It consists of code portions like this:
!comment !comment If a customer has SA outstanding !comment in d-products must have update flag in master !comment get m-customer if sa-outstanding = "SA" sort account-no extract account-no,company-name output mcust,link,temp xeq get d-products ...
To test this piece of code without streaming the whole jobstream I can simply add the lines
run suprtool.pub.robelle base kco.db
after the last !comment, and replace the
xeq with an exit.
Then I select all the lines between the "run suprtool" and the "exit", use the right mouse button and choose "run selection as host command".
This causes the host command dialog box to pop up, and my commands are executed with the results showing in the box.
When I have finished debugging this section of the jobstream I need to remember to remove the "run suprtool", "base" and "exit" lines and save my file. But I didn't have to run the entire job stream, and work late to schedule the run!
[Robyn Rennie, Robelle IS Programmer Analyst]
Robelle Website Now Changes Daily
The Robelle web site is now updated daily with items of interest to HP e3000 users. Besides tips about Robelle's Qedit and Suprtool software, you will find advice about major HP e3000 applications such as Ecometry, pointers about the Internet, and news about upcoming versions of MPE.
Articles are written by Robelle customers and HP e3000 experts, and are edited and published by Robelle president Bob Green. He made the people of the little Caribbean island of Anguilla famous with his daily news updates at www.news.ai. Submit your HP e3000 stories and tips to bgreen@robelle.com and let him bring you a little fame too.
Visit www.robelle.com often -- make it your home page so you don't miss any of the news.
If you had been following the new www.robelle.com site regularly, you would have already read everything that is in this newsletter.
[Mike Shumko, Robelle Senior Manager]
Is Your Disc Being Eaten By MPE?
While doing some consulting work offsite, I overheard the system manager discussing the possibility of having to get more disc space for the system volume set. They were looking at an upgrade from 5.5 to 6.0 shortly and were concerned whether they had enough room to do the upgrade. Disc space on the system volume set was getting critical.
I asked if they purged their log files on a regular basis. They
replied that they did this regularly. I then asked if that included the
NM log files? The look I received in response was my answer. They were
not aware of the log files that all network processes write to. I told
them to sign on as manager.sys and do a listf nmlg####,2
and see how much disc space the log files took up. A few screenfuls
later they realized that the these log files took up nearly 1 gig of
space, on the system volumeset.
We added the following commands to the end of their backup job, so they would never run into this situation again.
!continue !purge log####.pub.sys !continue !purge nmlg####.pub.sys
[Neil Armstrong, Robelle Senior Programmer]
Newsletter Distribution and Format Choices
Robelle Solutions Technology Inc. provides the What's Up, DOCumentation? newsletter as a service to our customers, who may also subscribe to this publication via e-mail by sending a message to admin@robelle.com. This newsletter is available as HTML formatted e-mail, plain-text, or PDF.
Comments about this issue, as well as ideas for future issues, including news, tips, and suggestions for articles, can be sent to support@robelle.com.
Newsletter Copyright 2001 Robelle Solutions Technology Inc.
Suprtool and Qedit are trademarks of Robelle Solutions Technology Inc.