Bringing Robelle KB To The Web

Introduction

To aid customer support and provide call and problem tracking, Robelle tech-support has been using an in-house knowledge-base (KB) system for over a decade.

When a call is logged into the TurboIMAGE database, it is filter-broadcast to Robelle technical staff (aka "techies") by email. Comments and suggestions can then be added to the database by anyone who receives the broadcast, which are then re-broadcast. An e3000 application searches the database via a combination of keywords and free-form text to aid in customer support - whether to help jog the memory for solutions, or to track outstanding trouble-calls and enhancement requests.

Although KB works very well internally, there were no hooks in the system to allow external access by our customers.

In our emerging 21st century web-centric society, our computer-savvy customers fully expect to have information available to them at a mouse-click's notice. From this perspective, Robelle KB has a problem: its current interfaces are terminal-based.

Our first thought was to write a web-based thin-client that accessed the HP e3000 database directly. However, the issue of our customers' privacy came up. A frequent occurrence within the call text was "Another contact is so-and-so at 555-1212". Our database is huge! Sanitizing the data manually would be a tremendous job, and costly.

For our final solution we wanted reliability, compatibility, easy maintenance, and of course customer accessibility. We decided to export each individual entry as an automatically sanitized web page, allowing the customer to easily search and view the entries. The tools that we used were Suprtool to extract the data, Qedit to format it nicely, and FTP to transfer the files from the HP 3000 to our HP-UX web server. (Suprtool does provide some basic HTML output through STEXPORT, but I wanted finer control over the pages.) This is all tied together by MPE Command Interpreter programming.

Each KB entry web page would look like the following example:

Robelle Technical Database Reference KB14712 Product: Qedit for Windows 4.7 Subject: want host commands Status: Closed Date Created: 1999.02.01  08:32 Date Modified: 2001.05.01  09:29 Short Description:    want host commands Originator: DAVE_LO@ROBELLE.COM Comments from Richard's HP-UX users regarding Qedit for Windows (trial): What I really miss about remote host editing is the ability to pop up a window and send some quick commands to the host system. Yea I can toggle between Qedit and Reflection, but Windows doesn’t have a quick convenient way to switch programs, and I still have to login twice. Append: (E:E) NICKY GUNTHER This one is important to them 22 Mar99 11:39 AM Host commands from within the client is one of three KBs that are the most important to Say It With Hares. Rich understands that we are working on this for 1999. Append: (E:C) HANS HENDRIKS Implemented 01 May01 9:29 AM Host commands have been available for some time now...

Data Base Structure

The native database on an HPe3000 is TurboIMAGE. It is a network database, and the simplified structure of the KB database is shown below. Records are stored in datasets, analogous to tables in SQL.

The M-CALLS set contains information unique to each entry, and is called a Master set. The Detail Set, D-CALL-TEXT, contains doubly-linked chains of compressed text in 512-byte blocks. Although each block is a fixed length, there may be as few as two characters in the block!

The nature of IMAGE databases is that the sets contain fixed-length records. In order to save space, I came up with the scheme of ending lines with a carriage return and packing them together contiguously. This is great for storage, but not that easy to unpack and export. Because a lot of the text is formatted by the techie in a particular way on purpose, I could not just blindly wrap the text. I would have to trust that the line breaks were in the correct spot.

Creating The Output Data...One Step At A Time

Everytime we tackle a seemingly difficult task using Suprtool, we tend to surprise the programmers who wrote it. This task was no different.

The algorithm I chose to create the web pages is

Here's the abbreviated main loop of the jobstream...

!while finfo("hdr","EOF") > 0
! run suprtool.pub.robhome;parm=4;info="use kbs31450"
! stsetvar
! run qedit.pub.robhome;parm=4;info="use setva2.cmd"
! stsetva2
! echo !<html!> > qsect1
! echo !<title!>KB !callnbr: !KBTitle !</title!> 
  >> qsect1
! echo !<h2!>Robelle Technical Database Reference 
  KB!callnbr!</h2!> >> qsect1 
! run qedit.pub.robhome;parm=4;info="use qwebuse"
! ftpit.cmd.mis kbtogo, kb!callnbr.html, /users/WWW/kbs,
  daffy.robelle.com
!endwhile

A few things went wrong with this algorithm. For example, we needed to create 'Use' files for Suprtool and Qedit before the execution of the loop. If we do not, MPE will cause an EOF on the STDIN during the second pass of the loop; as the input stream will have been exhausted. We also have to convert special characters that are emebedded in the text.

Algorithm Implementation

Two of the off-the-shelf tools used in the processes are Qedit and Suprtool, both from Robelle. These are versatile "swiss-army" tools that allow you to extract and manipulate your data in virtually unlimited ways. The key is, of course, knowing how to use these tools effectively.

If you're unfamiliar with Qedit or Suprtool, you can

• Check out a quick primer on Qedit and the commands that I've used here
• Check out a quick primer on Suprtool and the commands that I've used here

So....off we go.

Step 1: Extract the call numbers. This is the easiest part. :-)

We only select KB entries with a special "WebPost" keyword and which have been modified within the last 2 days. The philosophy is that technical support will mark those calls which are suitable for publication. Some calls are too boring for publication ("Please send a pre-release tape to customer X."). And some contain customer private information from dumps and log files.

We deliberately do not extract the customer-information from the call dataset into the file, since we want the web listings to be anonymous. This step is accomplished with Suprtool.

!run suprtool.pub.robelle base kb,5,secretpassword get m-calls define MoveToWeb,keyword-block,7 define short-desc1,short-desc(1) define short-desc2,short-desc(2) item date-modified,date,yyyymmdd if $upper(MoveToWeb) = "WEBPOST" and date-modified > $today(-2) ext call-nbr, status-FLAG, call-taker, date-created, time-created ext date-modified, time-modified, product-code,subject ext short-desc1, short-desc2, keyword-block out refs2mov,link,temp xeq

Step 2: Get the records from refs2mov, and format them into headers. Each header contains unique information about the entry. We'll format it into html table rows for later processing. Recall that each header line only takes up one line in the file, which allows us to keep a synchronization with the call numbers file.

in refs2mov define dc-yyyy,date-created,4 define dc-mm,date-created[5],2 define dc-dd,date-created[7],2 define tc-hh,time-created,2 define tc-mm,time-created[3],2 define dm-yyyy,date-modified,4 define dm-mm,date-modified[5],2 define dm-dd,date-modified[7],2 define tm-hh,time-modified,2 define tm-mm,time-modified[3],2 comment Build a table for the main details of the header for each entry. Tables look nice. comment We extract vertical bars instead of &, as this is our *own* internal representation. ext "|lt;tr|gt;|lt;td|gt;Product: |lt;/td|gt;|lt;td|gt;", product-code,"|lt;/td|gt;|lt;/tr|gt;" ext "|lt;td|gt;Subject:|lt;/td|gt;|lt;td|gt;", subject,"|lt;/td|gt;|lt;/tr|gt;" ext "|lt;td|gt;Status:|lt;/td|gt;|lt;td|gt;%%", status-flag, "|lt;/td|gt;|lt;/tr|gt;" ext "|lt;td|gt;Date Created:|lt;/td|gt;|lt;td|gt;", dc-yyyy,".",dc-mm,".", dc-dd,"  ",tc-hh,":" ext tc-mm,"|lt;/td|gt;|lt;/tr|gt;"-mm,"|lt;/td|gt;|lt;/tr|gt;" ext "|lt;td|gt;Date Modified:|lt;/td|gt;|lt;td|gt;",dm-yyyy,".",dm-mm,".", dm-dd,"  ",tm-hh,":" ext tm-mm,"|lt;/td|gt;|lt;/tr|gt;" ext "|lt;td valign='top'|gt;Short Description:   |lt;/td|gt;|lt;td|gt;",short-desc1 ext short-desc2,"|lt;/td|gt;|lt;/tr|gt;" ext "|lt;/table|gt;|lt;pre|gt;|lt;hr|gt;" ext "|lt;font color=blue|gt;Originator: ",call-taker,"|lt;/font|gt;|lt;br|gt;" out hdr,temp,ascii xeq

Now you might be thinking....why go to all that trouble to insert |lt; and |gt; when I could be outputting < and > ?
The reason is that the data extracted from the database could very well contain < and >, which would create html errors when viewing the page. In the sanitizing step, we will change all occurrences of < and > to &lt; and &gt; then change |lt; to < and |gt; to > to complete the html formatting.

Step 3: Process the first record of the file.
We're going to name each web page as KB<referencenumber>.html. This will provide a simple method of maintaining files on our web site.
While we're at it, we'll set the title to a variable so that we can insert it into the web page easily in our formatting step.

Step 4: Get the Text of the Entry
This step is going to be repeated over and over again until the call numbers file is empty. An interesting quirk of MPE and re-directing STDIN from a job is that you can't use it in a loop. Although the code below on the left looks sane, it will NOT work!

Bad!	Good!
setvar count 1 While count < 10     run suprtool.pub.robelle     in foo     list standard     exit     setvar count count + 1 Wend	setvar count 1 echo in foo > usem echo list standard >> usem echo exit >> usem While count < 10     run suprtool.pub.robelle;info="use usem";parm=4     setvar count count + 1 Wend

In the "Bad" example, the second time through the loop we will get an EOF error on the STDIN of the job. To get around this, we need to put our Suprtool commands inside a "use" file. Since all of the STDIN is inside a file, Suprtool re-opens it every time and the problem goes away.

Here is the "Use" file that we will need.

!comment **** get the first rec from the call numbers file !echo purge ref2mov,temp > kbs31450 !echo setjcw cierror 0 >> kbs31450 !echo in refs2mov >> kbs31450 !echo numrecs 1 >> kbs31450 !echo out ref2mov,link,temp >> kbs31450 !echo xeq >> kbs31450	By using the numercs 1 command, we are only allowing one record in the output file. We'll use this as a key to retreive our text and set some variables. It will also give us a 'current entry' to work with inside the loop.
!comment **** setup the two Use files for setting the variables !echo purge stsetvar,temp >> kbs31450 !echo purge stsetva2,temp >> kbs31450 !echo setjcw cierror 0 >> kbs31450 !echo in ref2mov >> kbs31450 !echo ext "setvar callnbr ", call-nbr >> kbs31450 !echo out stsetvar,temp,ascii >> kbs31450 !echo xeq >> kbs31450	This creates the first of two small command files used for setting variables. We will use the Callnbr variable to name the web page. The Stsetva2 variable will become part of the <title> construct. setvar callnbr <callnbr>
!echo >> kbs31450 !echo in ref2mov >> kbs31450 !echo ext short-desc1, short-desc2 >> kbs31450 !echo out stsetva2,temp,ascii >> kbs31450 !echo xeq >> kbs31450	Here is the contents of the second command file: setvar stsetva2 <short description>
!comment **** Get all of the text records for that entry !echo base kb,5,secretpassword>> kbs31450 !echo table refs,call-nbr,file ref2mov >> kbs31450 !echo chain d-call-text,call-nbr=refs >> kbs31450 !echo ext text-block >> kbs31450 !echo out textdata >> kbs31450 !echo purge textdata >> kbs31450 !echo xeq >> kbs31450 !echo exit >> kbs31450	Using the refs file created above, we extract all of the text for the current entry. Recall that all of the individual lines are concatenated with carriage-returns as separators.

Step 5: Use Qedit to get the appropriate header and glue everything together.
This step is truly ugly. Similar to the problem with Suprtool, we have to stuff all of our commands into a "Use" file.

We format the header and the extracted text into a single file. At this point, we have the opportunity to cleanup and sanitize the text:

• We get the header.
• We change all of our internal codes into something that people will understand. For example, LQ is our internal code for Qedit, ST for Suprtool, etc.
• We change all occurences of < and > in the textdata file to &lt; and &gt;.
• For our desired HTML formatting tags, we put exclamations in front of the > and < so that the HP CI doesn't mistake them for I/O re-direction commands.
• We change our previous coding of |lt; and |gt; in header fields to be < and > for the final web page.
• We remove unneccessary blank lines: 13 is the decimal value for Carriage Return (cr). We change two adjacent 13s into a single 13. Since our data contains cr's as separators, and MS Windows needs crlf (carriage-return line-feed), we change all cr's to crlf. 10 is the decimal value for line-feed.
• We add some highlighting for particular status codes of the entry.
• For each person's comment to the entry, a date and time stamp are formatted to make it look nice.
• HP Terminal Enhancement escape sequences (which are in the data!) are modified so that they will look right.
• We remove email addresses from the text to avoid our customers receiving spam email from web-roaming spam robots.

Here is the code that does all the steps above:

!comment
!comment main qedit use file.
!comment
!
!echo set lang data > qwebuse
!echo set len 1000 >> qwebuse
!echo aq]=emptyln >> qwebuse
!echo dq all >> qwebuse
!echo aq ]=hdr [ >> qwebuse
!echo cq "<"&lt;"@ >> qwebuse
!echo cq ">"&gt;"@ >> qwebuse
!echo cq "|lt;"!<" >> qwebuse
!echo cq "|gt;"!>" >> qwebuse
!echo lq all >> qwebuse
!echo keep hdrchg,temp,yes >> qwebuse
!echo dq all >> qwebuse
!echo aq]=textdata >> qwebuse
!echo cq ">"!>"@ >> qwebuse
!echo cq "<"!<"@ >> qwebuse
!echo cq "Email: [ -~]*" (regexp) "" @ >> qwebuse
!echo l $c $o >> qfowebuse
!echo keep textdata,yes >> qwebuse
!echo dq all >> qwebuse
!echo set len 1000 >> qwebuse
!echo aq]= qsect1 >> qwebuse
!echo a]=hdrchg >> qwebuse
!echo cq "             "" * >> qwebuse
!echo cq "             "" * >> qwebuse
!echo cq "             "" * >> qwebuse
!echo cq "             "" * >> qwebuse
!echo cq "%%S"!<font color=blue!>Signed-Off!</font!>"@ >> qwebuse
!echo cq "%%B"!<font color=red!>Bug!</font!>"@ >> qwebuse
!echo cq "%%E"!<font color=blue!>Enhancement Request!</font!>"@ >> qwebuse
!echo cq "%%C"!<font color=blue!>Closed!</font!>"@ >> qwebuse
!echo cq "%%M"!<font color='#FF8C00'!>Mystery!</font!>"@ >> qwebuse
!echo cq "%%T"!<font color=green!>Tape sent to customer!</font!>"@ >> qwebuse
!echo cq "%%P"!<font color=green!>Pre-Release Available!</font!>"@ >> qwebuse
!echo cq "%%O"!<font color=blue!>Open!</font!>"@ >> qwebuse
!echo aq]=textdata >> qwebuse
!echo set decimal on >> qwebuse
!echo cq '13'13 '13 @ >> qwebuse
!echo cq '13 '13'10 @ >> qwebuse
!echo cq '27 "~"@ >> qwebuse
!echo cq "~&dH"!<hr!>"@ >> qwebuse
!echo aq]=qsectf >> qwebuse
!echo cq "LQ "Qedit " @ >> qwebuse
!echo cq "ST "Suprtool " @ >> qwebuse
!echo cq "QW "Qedit for Windows "@ >> qwebuse
!echo cq "* Note: "!<font color=blue!>Append: "@ >> qwebuse
!echo cq " AM *" AM !</font!>"@ >> qwebuse
!echo cq " PM *" PM !</font!>"@ >> qwebuse
!echo cq "* "'13'10 ""@ >> qwebuse
!echo set keep var on >> qwebuse
!echo keep kbtogo,yes >> qwebuse
!echo t hdr,labels >> qwebuse
!echo d [ >> qwebuse
!echo k,yes >> qwebuse
!echo t refs2mov,labels >> qwebuse
!echo d [ >> qwebuse
!echo k,yes >> qwebuse
!echo exit >> qwebuse

There are a few tricks in the above code...

cq '13'13 '13@ cq '13 '13'10@	'13 is the decimal value for Carriage Return (cr). Changes two carriage returns into one. Eliminates some blank lines. Since our data contains cr's as separators, and windows needs crlf (carriage-return line-feed), change all cr's to crlf. '10 is the decimal value for line-feed.
cq " AM *" AM !</font!>"@	We put exclamations in front of the > and < so that the HP CI doesn't mistake them for I/O re-direction commands.
cq "%%S"!<font color=blue!>...	Although there are many status code changes, only one of them will actually take effect. It's sort of like falling through an if...end constuct.
cq "<"&lt;"@ cq ">"&gt;"@	Change all occurrences of < and > to &lt; and &gt; This ensures that the data in the entry won't cause any html tag problems.
cq "|lt;"!<" @ cq "|gt;"!>" @	Change our coding of |lt; and |gt; to be < and >. We prefix the < and > with an ! so that MPE won't be confusing our data with I/O redirection.
set lang=data	We do this so that we can have at least 1000 characters per line. Set lang=text only allows 256 characters.
set keep var on	This ensures that we don't keep all 1000 characters per line! The recordsize is variable.
aq]=emptyln	This forces the switch from any defaults that may have been set regarding file type.

Step 6: Save the HTML as a variable-length file.

Most importantly, the file is kept as a bytestream, or variable-length file. This helps minimize the size of the webpages. If the files were fixed length, each line of the file would require 1000 bytes. So...a 40 line page would be 40k...just in text! Some KB entries are hundreds of lines long. With little effort on our part, using variable-length files makes the web-surfer's load-time much less.

Step 7: Copy the file to the web using FTP.

ftpit.cmd kbtogo, kb!callnbr.html, /users/WWW/kbs, daffy.robelle.com

The contents of the ftpit command file is

parm fromfile="?",tofile="",todir="",tocpu=""
if "!fromfile"="?" then
echo fromfile="?",tofile="",todir="",tocpu=""
endif
smcapset
echo user prodsend > ftpin
echo !PRODPASS >> ftpin
echo cd !todir >> ftpin
echo put !fromfile !tofile >> ftpin
echo site chmod 666 !tofile >> ftpin
ftp !tocpu <ftpin >ftpout

One important note to remember is to do a chmod 666 of the file. This allows non-owners the right to view the file. Depending upon how your FTP logon and web server directory are set up, you may need different chmod permissions.

Step 8 and 9. Delete the first entry from the headers and call-numbers files, then loop until done.

!echo t hdr,labels >> qwebuse
!echo d [ >> qwebuse
!echo k,yes >> qwebuse
!echo t refs2mov,labels >> qwebuse
!echo d [ >> qwebuse
!echo k,yes >> qwebuse

Much of the code written in the above steps was to set up the loop for execution. The loop itself is fairly simple and straightforward.

!while finfo("hdr","EOF") > 0
! run suprtool.pub.robhome;parm=4;info="use kbs31450"
! comment
! comment First html section to add in QW script
! comment
! stsetvar
! run qedit.pub.robhome;parm=4;info="use setva2.cmd"
! stsetva2
! echo !<html!> > qsect1
! echo !<title!>KB !callnbr: !KBTitle !</title!> >> qsect1
! echo !<link rel="stylesheet" href="/style.css" type="text/css"!> >> qsect1
! echo !<body bgcolor="#ffffff" background="/gif/background.gif"!> >> qsect1
! echo >> qsect1
! echo !<!-- std header for root web directory --!> >> qsect1
! echo !<!--#include virtual="/header.txt" --!> >> qsect1
! echo >> qsect1
! echo !<h2!>Robelle Technical Database Reference KB!callnbr!</h2!> >> qsect1
! echo !<table!> >> qsect1
! run qedit.pub.robhome;parm=4;info="use qwebuse"
! ftpit.cmd.mis kbtogo, kb!callnbr.html, /users/WWW/kbs, daffy.robelle.com
! tell ken,mgr.mis Moved http://www.robelle.com/kbs/kb!callnbr.html !hptimef
! print ftpout
!endwhile

Conclusion

As you can see, the method in this project was to use scripts to combine a few dependable tools that manipulate data and text. The tools that we used were Suprtool to extract the data, Qedit to format it nicely, and FTP to transfer the files from the HP 3000 to our HP-UX web server. This is all tied together by MPE Command Interpreter programming. No actual COBOL or C programming required. The conversion isn't very fast, but it runs late at night and doesn't have that great a volume to process each day. For more implementation details, read the expanded version of this article at www.robelle.com/library/papers/kbweb

Qedit Primer

Qedit has been one of the mainstay editors on the HPe3000 for decades. It understands how to handle the different types of files on the HP, as well as some from other platforms. You can use it as a text-entry tool, as source-code creation and maintenance, or in batch as a text/data-processing tool.

Qedit has many text-manipulation 'line-mode' style commands. You can also use almost every MPE CI command within the Qedit shell.

Here are the ones that I've used in the KBsToWeb process.

Suprtool Primer

Suprtool has been around almost as long as Qedit has. It is one of the foremost data extraction tools available for the HPe3000 and HP-UX computers. Using low-level tightly-controlled code, it's main use is to extract data as quickly as possible. Usually some criteria is used in extracting the data, such as customer numbers, date ranges, etc. You can automate the update of data in databases using Suprtool, or use it to create and format reports for the web.

Suprtool also has some text manipulation tools, but they are not as strong as Qedit's. Together these two products create an impressive tool for data extraction and maipulation.

Suprtool has its own command set. Here are the ones that I've used in the KBsToWeb process.

May 25, 2001