Porting Qedit from MPE to HP-UX By David J. Greer Abstract Robelle recently ported its Qedit full-screen editor from the MPE environment to the HP-UX environment. David Greer, manager of software R&D at Robelle, headed up the team doing the migration. This paper describes the problems they faced and the resulting solutions. Qedit is written in SPL and was ported to HP-UX using the SPLash! compiler from SRN. The paper covers both the gory details of how Robelle does cross-development using SPLash! and the theoretical problem of how to make UNIX look like MPE. Most of the Qedit source code is identical for the two systems, but the MPE file system differs radically and is more complex than the UNIX file system. Robelle Consulting Ltd. Unit 201, 15399-102A Ave. Surrey, B.C. Canada V3R 7K1 Phone: (604) 582-1700 Fax: (604) 582-1799 Email: david_greer@robelle.com Copyright Robelle Consulting Ltd. 1994 Permission is granted to reprint this document (but not for profit), provided that copyright notice is given. Porting Qedit from MPE to HP-UX By David J. Greer Introduction Our Qedit full-screen text editor was first written in 1977 and has been continually enhanced since then. We recently ported Qedit from MPE to HP-UX. This paper discusses our research strategy and the technical details of how we actually managed to port approximately 100,000 lines of code that had been written over the last fifteen years. Original UNIX Goals For a few years, our customers (especially in Europe) had started doing development on UNIX machines. We didn't know anything about UNIX, but we decided that we had better start learning. Our goal was to learn as much as we could about UNIX by seeing how much of our source code and programs we could convert to HP's version of UNIX: HP-UX. Why HP-UX We decided on HP-UX for a couple of reasons: 1. It was made by HP, so it was likely to be more familiar to us than another vendor's UNIX (and at least the Response Center telephone number would be the same). Time and again, this proved to be true and greatly helped our development effort. 2. Qedit is written almost entirely in SPL/SPLash!. Because of our compiler technology, we wanted to focus on the HPPA RISC architecture. For this reason, we restricted our research to HP's 700 and 800 series of computers (they have HPPA RISC CPUs). 3. We did not want to rewrite all of the source code into C (the usual language of choice for UNIX programs). We felt that rewriting Qedit into C would be too large an R&D project for Robelle. Automated tools for doing the conversion were reported to be ineffective. Major R&D Phases Our HP-UX research was done in different phases. We had a short-term goal for each phase and worked towards completing this goal in a reasonable time frame. The major phases were as follows: 1. Basic Research. In this phase we borrowed an HP-UX machine and investigated HP-UX object-code and run-time libraries. 2. MPE and Robelle Libraries. Robelle's software development is based on a concept of layers that build on each other. This phase involved getting our own machine, writing necessary MPE replacement routines, and porting our fundamental subroutine libraries. 3. Rebuild Qedit. Once the basic libraries were running we needed to look at Qedit. In this phase, we reengineered Qedit so that it has little knowledge about which computer platform it is running on. 4. Polish and Release Qedit/UX. After the first version of Qedit was working on HP-UX, we needed to do a series of enhancements to make Qedit work in a more UNIX-like manner. We also needed to complete all the details that turn an R&D project into a product. Personnel Three people worked on porting Qedit to HP-UX. Dave Lo Dave was the general programmer's helper. He was assigned specific research tasks (e.g., how to get the value of an environment variable in SPLash!). Robert Green Bob is the original author and chief architect of Qedit. His main job was to keep Dave's and David's spirits up and to reengineer Qedit so that it didn't depend on MPE any more. David Greer I was the overall project leader. I ported most of our subroutine libraries, learned everything I could about programming interfaces to HP-UX, and implemented all of the Qedit/UX operating system interface. Development Diaries At Robelle, we keep a diary with summaries of each day's development effort. We kept a development diary for HP-UX -- much of the material in this paper is drawn from it. It takes work to keep these diaries up-to-date, but they pay off in the long-term by saving useful information. Time and Effort From first inception to delivering a production product took about 20 months. The work effort is estimated as 19 person-months. We feel that this is incredibly fast for the amount of work that needed to be done. We could never have achieved this if we had rewritten all of our software into a new language as well as for a new operating system. We estimate that a complete rewrite of Qedit into C would have taken at least twice the time (four years) and three times the work effort (at least 60 person-months). A rewrite goes against our long-held belief in doing things in small incremental steps, rather than striking out for the big win. Another disadvantage of rewriting Qedit into C was that we likely could not have used the C version on Classic 3000s, due to poor C compiler support on the Classic 3000. However, we always try to make available as many Qedit enhancements as possible to all of our HP 3000 customers. This is one reason why we have one set of source code for all versions of Qedit. Software Development At Robelle To understand how we ported Qedit from MPE to HP-UX, it is important to understand how we develop software at Robelle. Most of our development, including Qedit, is done in SPL and SPLash!. SPL was the original Systems Programming Language for the HP 3000. The SPL compiler only produces object-code for Classic HP 3000s (you can also run this code in compatibility-mode on MPE/iX machines). SPLash! is a compiler from Software Research Northwest (206-463-3030) that translates SPL source code into HPPA RISC object-code. We use SPLash! on MPE/iX to produce the native-mode version of Qedit. There is no SPLash! compiler for HP-UX. SPL has no run-time library (which is similar to how the C language works). In Pascal, you would write something to the screen as shown below: writeln('Hello, World!'); In SPL, there is no language construct that will print a string on your screen. Instead, you move the string to a buffer and then call a library routine to print the string: move outbuf := "Hello, World!"; print(outbuf,-13,0); ! print is a library routine The print routine displays the thirteen characters on the screen. Print is a library routine that comes with MPE. Subroutine Libraries Because SPL does not rely on a language-specific run-time library, we thought it might be possible to port the SPLash! object-code to HP-UX running on the HPPA RISC architecture. The only thing left would be to replace all of the subroutine libraries on which Qedit depends. Since Qedit depends on hundreds of routines, that was still a lot of potential work. Qedit is built upon a series of subroutine libraries. Each set of libraries depends on other sets of libraries. The libraries look something like this: Qedit Qedit Library Calculator, Qhelp, and Other Libraries Low-level Robelle Library MPE and Compiler Libraries View of Robelle Libraries While these libraries build on each other, libraries at the upper levels can call libraries at any lower level. For example, Qedit makes many calls directly to the MPE library (e.g., readx, print, fopen, fread, ...). Similarly, the calculator makes calls to both the Robelle low-level library and the MPE library. Development Summary Because we use SPL and SPLash!, we needed to develop ideas that would let us port object-code to HP-UX. In order to port Qedit from MPE to HP-UX, we would first need to port many of our own libraries and investigate how many MPE library routines were available on HP-UX. Phase One: Basic Research We wanted to start our research by doing two things: 1. Learn more about HP-UX (e.g., how to log on!) 2. Prove whether or not the object-code formats on MPE and HP-UX were the same. Get A Machine You can't make an omelet without breaking eggs, and it's hard to learn about HP-UX without having access to a machine. Because we had no idea how successful our initial research would be, we didn't want to purchase one. We asked HP if we could rent an HP-UX machine for a week, but HP doesn't have short-term rentals. Since we've had good relations with HP/Canada, they arranged for us to have one of their demo machines for a month. It's a good thing we had the machine that length of time, because it took us about three weeks to get it up and running. We could tell HP-UX was made by HP. When we first brought the machine up, one of the first things it did was ask us if our console was a 2392 or a 700/92. We can't imagine any non-HP machine asking this question. Networking We learned a lot by just installing the borrowed HP-UX machine and getting it connected to our network. On our HP 3000 machines, we had HP's NS software and used Thinlan to connect all of our machines together. We lost a lot of time because the demo HP-UX machine had a jumper switch on the network interface card turned off. It took us two weeks to figure out that it was a hardware problem and not a configuration problem. By default, HP-UX is not able to communicate with MPE machines. On MPE, we are used to having front-end software that handles configuration (NMMGR being the one for configuring network software in MPE). In UNIX, it's common for configuration to be done by changing scripts (although some common configuration tasks can be done with HP's SAM tool). Scripts are similar to MPE/iX command files. One of the files that is used to configure networking is called /etc/netlinkrc. By default, it contains a line like this: /etc/lanconfig lan0 ether For HP-UX to communicate with MPE machines in the Thinlan/NS environment, this line needs to be: /etc/lanconfig lan0 ether ieee Notice the addition of "ieee" to the end of the line. We also needed to the following: 1. Fill in the DOMAIN and ORGANIZATION in /etc/netlinkrc. 2. Add machine names and IP addresses to /etc/hosts. 3. Reboot our machine for these changes to take effect. Since our machine had the NS software for HP-UX, we could now use vt3k to log on to our MPE machines from HP-UX (vt3k provides virtual terminal access from HP-UX to MPE). After using NMMGR to add our new HP-UX machine to our MPE configuration, we could also use dscopy to copy files from MPE to HP-UX and vice versa. Object-Code Format Our next major goal was to test our theory that object-code files were the same on MPE/iX and HP-UX. To show this was the case we planned to compile a SPLash! program on MPE, copy the object-code to HP-UX, link the object-code, and run the resulting program. One of the first programs we tried was written by Randy Medd of Telamon (510-987-7700). This program calls the HP-UX routines puts (which writes a string to stdlist) and exit (which terminates the program). It looked like this: $native, nocc, unix begin equate line'feed = 10; byte array buf(0:14); double procedure puts(buf); virtual byte array buf; option native, nocc, external; procedure exit(int); value int; double int; option native, nocc, external; move buf := ("Hello, World!",line'feed,0); puts(buf); exit(0); end. Compiling and Linking We compiled the above program using SPLash! and copied the resulting object-code to HP-UX with these commands: :splash testsrc,testobj :dscopy testobj to /users/david/test.o:daffy[david:pass] The next part took about a week of work to figure out: the exact link command to use in HP-UX in order to have our SPLash! program run. The HP-UX command for linking is called ld. To link our example program, we used this ld command: ld /lib/crt0.o test.o /lib/libc.a This linked the three files /lib/crt0.o, test.o, and /lib/libc.a. The first file is /lib/crt zero dot oh (don't mix up the zero and the oh), which is the C run-time startup library. The test.o file is our SPLash! object-code that we generated on MPE. The /lib/libc.a file is one of the HP-UX run-time libraries. Like most HP-UX command names and commands, everything must be in lower case. The general form of the ld command is: ld /lib/crt0.o objfiles... /lib/libc.a ^ ^ C library C run-time startup Running Our Program By default, the ld command creates a program file called a.out. To run this program, we only had to type its name (like using implied run on MPE/iX). We tried running the resulting program and got: a.out {run our sample program} Hello, World! {result of running a.out} This was exactly what we expected. The buffer that we initialized with "Hello, World!" was printed out on stdlist and the program stopped. We had now answered one of our most basic questions -- we could compile programs on MPE, copy the object-code to HP-UX, link the object-code into a program and run it on HP-UX. Run-Time Libraries Once our basic research question was answered, it was time to start exploring more of HP-UX. We had no idea how many of the MPE routines that we needed were available on HP-UX. We used two HP-UX tools to search for routines: man and nm. Man Pages The man pages are similar to MPE's on-line help. For example, if you want to see if there is a description of a routine called binary, you do the following: man binary Once done, you will see this: No manual entry for binary. Another way to search the man pages is to have the man utility search a keyword index from all the man pages for a string (see below for how to create the man page index file): man -k binary {-k requests a keyword search} . . {many entries printed} . There were many routines with a description that included the word "binary", but none were similar to the MPE binary routine. The man page index is in the file /usr/lib/whatis. You can rebuild this file by doing: catman -w You must be logged on as super user to run catman. It takes catman a long time to create the index, so we suggest that you run it in the background or in a batch job. Searching Object-Code Just because a routine is not documented doesn't mean that it does not exist. Many MPE compiler-library routines are available in HP-UX, but not documented. The tool to use for finding symbolic names in object-code is nm. The nm utility is similar to the MPE/iX Linkedit Listobj command. The nm output is similar enough to the Linkedit listobj output to make MPE users feel at home (another advantage of using HP-UX). We combined nm with two other HP-UX tools to search entire directories of files. For example, to search the directory /usr/lib/ for any files with a symbolic entry for binary we did the following: cd /usr/lib nm -r /usr/lib/* | grep binary | more If you do this on HP-UX, you get a large number of error messages (not everything in /usr/lib is an object-code file) and the following line is found: libnsfmt.a:binary |1073789772|extern|data |$SHORTDATA$ Unfortunately, this is "data". What we needed was "code" or "entry". Libraries on HP-UX are also stored in the directory /lib, so we checked those libraries too: cd /lib nm -r * | grep binary | more We did this search and still did not find any routine called binary. So it appears that there is no binary routine for HP-UX. Since this routine is used everywhere in our tools, we ended up writing our own "binary" routine in C (note that the HP-UX routine atoi is not the same). Floating-Point Numbers In MPE, we use the compiler-library routines hpinext and hpextin to convert floating-point numbers from human-readable form to the internal IEEE format and vice versa. These routines have many options and handle different sizes of floating-point numbers. Rewriting these routines would be a lot of work. Qedit does not use floating-point numbers very often, except in the calculator. In order to implement the calculator, we would need the functionality of the hpinext and hpextin routines. A search of the man pages revealed no documentation for either routine, although there were other routines to handle floating-point conversions. All of our code assumes that hpinext and hpextin are available. So we searched for these routines using nm: cd /usr/lib nm -r * | grep hpinext | more The result of this search was as follows: libcl.a:hpinext | 13836|extern|entry |$CODE$ libcl.sl:hpinext | 421568|extern|code |$CODE$ libcl.sl:hpinext | 421524|extern|entry | libf.a:hpinext | 13836|extern|entry |$CODE$ libf.sl:hpinext | 421568|extern|code |$CODE$ libf.sl:hpinext | 421524|extern|entry | Success! We had found the hpinext routine. We also found the hpextin routine in the same libraries (using the same technique). These routines were located in either the libcl.a or libf.a library. Once we started converting the calculator to work on HP-UX, we had to change our ld command to include one of these libraries: ld /lib/crt0.o test.o /usr/lib/libcl.a /lib/libc.a ^ New run-time library This was one example of how familiar HP-UX is to MPE programmers. It is highly unlikely that any other version of UNIX has the hpinext or the hpextin routines. Even if they did, they are unlikely to have the same parameters and work the same way as the HP version. Phase One Summary We succeeded in our basic research goals. We obtained access to an HP-UX machine (due to the kind loan from HP) and we proved that SPLash! code compiled on MPE could be linked and executed on HP-UX. The time that elapsed for this phase was about three months (February to April). The work time: Dave Lo One person-month. Bob Green Half person-month. David GreerOne person-month. Phase Two: MPE and Robelle Libraries The goals for phase two were: 1. Obtain our own HP-UX machine. 2. Learn more about HP-UX. 3. Implement the necessary MPE routines on which we rely. 4. Start porting the Robelle library to HP-UX. Obtaining an HP-UX Machine After we had returned our demo machine to HP, we started work to obtain our own HP-UX machine. We did some investigation and decided to rent a Series 705 workstation from HP, complete with the necessary networking (NS) and development (C) tools (don't attempt C development with the "free" C compiler that comes with HP-UX). Since Bob and I work at home most of the time, we really planned on using the workstation as a server. Because HP-UX is a multi-user operating system, we decided that it would work just fine as a development server. Even today, our Series 705 is used as a workstation by one user in the office and as our primary development machine by those of us doing HP-UX program development. Implementing MPE Routines on HP-UX In phase one, we had already started implementing some common MPE routines (binary, dbinary, ascii, and dascii were now done). We continued with other routines such as print, readx, ccode, and hpsetccode. Once these routines were completed we could start porting our most fundamental library source modules. Porting the Robelle Libraries With a basic set of MPE routines now available, several of our source modules started working on HP-UX. However, we would often try to port a new Robelle library only to find that we needed to implement additional MPE routines. So the process went back-and-forth between porting Robelle source modules and writing new MPE replacement routines. In total, we implemented about twenty different MPE routines. MPE File System Routines Things went well, until we had to port our first source module that needed to access files. Using sample code written by Stan Sieler of Allegro Consultants, Inc. (415-369-2303), we investigated how difficult it would be to implement the MPE fopen, fread, fwrite, and fclose routines. The HP-UX file system is simple compared to the MPE file system. Files are collections of bytes. There is no implied structure to any file (although you may deduce the structure by examining the filename extension or by looking at some of the data in the file). In MPE, we have fixed-length, variable-length, ascii versus binary, record-length, blocking-factor, and many other file attributes. Our software depends on a number of assertions that are provided by MPE's file system routines. For example, if you open a new file with a filecode of 111 and a record size of 256 words, write some records to it, close it, open the file again, then ask about its structure, you expect to get a filecode of 111 and a record size of 256 words. In HP-UX, there is no place to store this information (without using an auxiliary file). It was at this point that we made a pivotal decision about how we would port our software. We would not emulate the MPE file system routines. Instead, we would reengineer our products to invoke an interface layer that would provide file system functionality. We would isolate the interface layer for each module that needed file system access into a separate source file (in object-oriented terminology this is called encapsulation). Recognizing Qedit Files On MPE, Qedit files are easily recognized as those with a filecode of 111 and a record length of 256 words. But how were we going to recognize Qedit files on HP-UX? HP-UX doesn't have filecodes or record lengths. Bob and I designed a routine that would examine the first 1024 bytes of an HP-UX file. By looking at various relationships in this file, we hoped that we could determine if the file was a Qedit file or not. We managed to implement such a routine. To test its effectiveness, we used dscopy to copy one Qedit file from MPE to HP-UX. We wrote a test program that opened every file on our Series 705 workstation and checked to see if it was a Qedit file. Our first implementations found many HP-UX files that the routine thought were Qedit files when they were not. But our final implementation found the one and only Qedit file that we had copied from MPE. Porting Qhelp to HP-UX Qhelp is the software that provides all of Robelle's on-line help facilities. Qhelp was nicely isolated into one source module. It uses a Qedit file to store all of the help text and the help keywords. We decided that Qhelp would be our first HP-UX module to use an interface layer to provide file system access on both MPE and HP-UX. File System Layer The MPE version of Qhelp invoked the MPE file system directly. It looked basically like this: Qhelp MPE File System Library Original Qhelp We reengineered Qhelp to call an interface layer to provide all file system functionality. This interface layer would open a file, let us know whether it was a Qedit file or not, read the file, and close the file. Once Qhelp called the interface layer, it would no longer "know" whether it was running on MPE or on HP-UX. The new Qhelp looked like: Qhelp Interface Layer OS File System Library Revised Qhelp Structure Implementing the Qhelp Interface Layer In our revised Qhelp diagram, we added an interface layer that is not operating system specific. The job of the interface layer is to hide the details of the underlying operating system. Each interface layer has two implementations: one for MPE and one for HP-UX. The MPE interface was written in SPL/SPLash! and the HP-UX interface was written in C. A lot of research went into learning how to map parameters in SPL to parameters in C. We also had to learn all of the HP-UX file system calls, how they worked, how to detect and report errors and so on. All of this took considerable time, but once we got Qhelp working we knew that future rewriting of other modules would be faster. Phase Two Summary During this phase we made a lot of progress. Approximately 20,000 lines of existing SPL source code were ported to HP-UX (e.g., the calculator and Qhelp modules) and about 1,500 new lines of C code were written. This provided a strong foundation to begin the port of Qedit to HP-UX. The time that elapsed for this phase was about six months (August to January). The work time: Dave Lo Two person-months. Bob Green Half person-month. David GreerThree person-months. Phase Three: Rebuild Qedit By this point, things were looking pretty good. We had a number of useful MPE routines written, a lot of the Robelle library was running on HP-UX, and we had one module (Qhelp) that did file I/O using an interface layer. It was now time for the big show: porting the Qedit source code to HP-UX. This was a much bigger project than what we had completed so far, since Qedit itself consists of about 75,000 lines of code. The First Attempt We started by compiling all of Qedit's source modules using SPLash! and copying the resulting object-code to HP-UX. Our first attempt to link Qedit using the ld command had many errors about missing procedures (e.g., all of the MPE file system routines). You can run an HP-UX program with missing externals, but the program aborts if any missing routine is called. To work around this problem, we wrote one source module that had all of the MPE routines that we had not rewritten for HP-UX. Each dummy routine returned a failure status. A portion of this file looked like this: integer procedure fopen; begin fopen := 0; hpsetccode(ccl); end'proc; <> procedure flock ;begin hpsetccode(ccl); end'proc; procedure fpoint ;begin hpsetccode(ccl); end'proc; procedure fread ;begin hpsetccode(ccl); end'proc; procedure freaddir ;begin hpsetccode(ccl); end'proc; procedure freadlabel ;begin hpsetccode(ccl); end'proc; Redo Module Qedit supports the Do, Redo, and Listredo commands. After our first attempt to link Qedit on HP-UX, we realized that we needed the redo module to work. The redo module also requires a "new" file (these are files that on MPE you cannot see with Listf or Listftemp). HP-UX does not have new or temporary files, but there is a solution. For the invisible scratch files needed by a program, you use the following algorithm on HP-UX: get a temporary filename (use tmpnam or tempnam) open the temporary filename unlink the filename from the file system If your program stops for any reason, the scratch file will disappear and the space the file occupied will be returned to the HP-UX free space list. With the redo module working, we were now able to link our first version of Qedit/UX. However, this version couldn't do much. It accepted commands, but most failed immediately. For example, the command: /open file would fail because the open command was still calling fopen. Therefore, the next step was to completely reengineer Qedit so that no direct calls were made to MPE routines. Like Qhelp, we began to reengineer Qedit to call an interface layer. Implementing the Qedit Interface Layer Bob and I took about one month to completely reengineer Qedit so that it called an interface layer for all operating system services. Bob changed Qedit and wrote the MPE version of the interface. I wrote the equivalent interface for HP-UX in C. A lot of our time involved Bob asking me if it was possible to do certain things in HP-UX (e.g., how to obtain exclusive access to an opened file in HP-UX). By now, I had ported enough source code to have a pretty good idea of what was possible and what was not. Writing the interface layer was not our only problem. There were many other parts of Qedit that had to be changed. Upshifting Filenames Inside Qedit, filenames were always upshifted, often in several places. On MPE this didn't matter, since filenames are not case-sensitive. On HP-UX, this was now important since filenames are case-sensitive. Bob had to go through Qedit to find every single location where filenames were upshifted and remove the upshifting code. We also had to cope with a different filename syntax (e.g., ../init.c is a valid HP-UX filename). While we made these changes for HP-UX, we are now taking advantage of these same changes for POSIX on MPE. In MPE/iX 5.0, POSIX filenames have the same syntax as on HP-UX (they are mixed-case and can contain directory names). So our efforts have paid off twice. The 32-Bit Alignment Problem The HP-UX C compiler prefers to have 32-bit variables aligned on 32-bit boundaries. In SPL and SPLash!, it is common to have 32-bit variables aligned on 16-bit boundaries. The interface layer for Qedit was much more complicated than any of the other interfaces that we had written. Because the interface was more sophisticated, large data structures had to be passed between SPLash! and C routines. We had to carefully implement these structures so that all 32-bit variables were aligned on 32-bit boundaries. We also had to insure that these structures themselves started on a 32-bit boundary. You can force SPLash! to allocate arrays on 32-bit boundaries by making the arrays virtual. This often has unexpected side-effects (e.g., you cannot pass a virtual integer array to an option splash procedure that is only expecting an integer array). In Qedit, we ended up doing the following. We would allocate one more word of storage than was necessary and then would have code that looked like this: if @ext'record mod 2 <> 0 then @ext'record := @ext'record(1); ! assumes integer array This code fragment adjusts the start of the ext'record structure if it does not start on a 32-bit boundary. We also had to be very careful of 32-bit variables that were passed by reference (these are often status parameters). It was often very difficult to detect a 32-bit alignment problem: due to the nature of SPL and SPLash!, the problems can come and go. The End-Of-File Problem Many parts of Qedit need to know how many records are in a file. For example, to print the line numbers for a command like: /list somefile last-10/ requires that Qedit know how many records are in somefile. MPE remembers how many records are in a file while HP-UX does not. Bob realized that rewriting Qedit so that it did not need to know how many records were in a file would take longer than everything we had done so far. This meant that in the HP-UX interface layer we have to read a file from beginning to end in order to count the number of lines in it. We do this every time that we open an external file in Qedit/UX. It's not very efficient, but it was the only way to provide this information to Qedit. Implementing Full-Screen By this point we had many of Qedit's line-mode commands working on HP-UX. The big question left was what to do about Qedit's full-screen mode. On MPE, Qedit uses block-mode to implement full-screen. On HP-UX, we didn't know what we were going to do about full-screen. One Friday night after a long day, my two children came into my office (at my home). They wanted to play the computer game "The Playroom". I soon had one child on each knee (they were 4 and 2 at the time), happily playing on my computer. I was surrounded by the HP-UX reference manuals, which I had been using all day. As my children were playing, I started flipping through one of the manuals. I wasn't really concentrating, but suddenly I noticed an entry for blmode. The opening description read: "This terminal library interface allows support of block mode transfers with HP terminals." By accident, I had found a set of routines that implemented block-mode. Bob liked the HP-UX block-mode interface enough that he reengineered the Qedit block-mode interface on MPE to look like the one on HP-UX. In a few days time, we had full-screen working on HP-UX. Phase Three Summary Qedit was now running on HP-UX. The process we used helped ensure that future Qedit enhancements would appear in both versions, since only one set of source code is used for the MPE and HP-UX versions of Qedit. Most of Qedit no longer knew which platform it was running on. The time that elapsed for this phase was about three months (February to April). The work time: Dave Lo One person-month. Bob Green Two person-months. David GreerTwo person-months. Phase Four: Polishing Qedit/UX With the basic research complete, we now had to polish Qedit/UX to turn it into a product. Up to this point I had to write all of the C code on HP-UX using the vi editor. I was certainly pleased when we had a working version of Qedit/UX that I could start using. I wasn't sorry to see the end of vi. Porting the Test Suite For years, we have developed test suites for all of our products. The Qedit test suite has a few hundred tests organized into sixty-three batch jobs. We knew that in order for Qedit/UX to be reliable, we would have to have a good portion of our test suite working on HP-UX. There was no automated way to convert our tests from MPE to HP-UX. We had to go through each test manually and convert MPE commands to their equivalent HP-UX commands. We also had to implement several of our test tools on HP-UX (e.g., the program that compares two Qedit files one line at a time looking for differences). Temporary Files On MPE, Qedit uses temporary files for a scratch file and for "hold" files. The hold filename on MPE was chosen so that you could use the name in commands. For example, /open file1 /hold 15/20 {save lines 15/20 in file "hold"} /open file2 /add 75=hold {add lines from the "hold" file} Initial versions of Qedit/UX created files in the current directory called QEDITSCR, HOLD, and HOLD0 (note the upper case filenames). The Add command in the previous example would not work, since there was no file called "hold": it was "HOLD". We modified Qedit/UX to create the qeditscr, hold, and hold0 files in /usr/tmp. HP-UX system managers expect temporary files to be created in the /usr/tmp directory (our nightly backup script removes all files from /usr/tmp more than twenty days old). We also had to modify Qedit/UX so that when it held lines or when you wanted to look at a file called "hold", it substituted the temporary filename. For example, /open file1 /hold 15/20 {save lines in "qholdBAAa23359"} /open file2 /add 75=hold {add lines from "qholdBAAa23359"} On HP-UX there really are no temporary files. HP-UX programs just create permanent files in a directory considered to be temporary. The different-looking filename is a result of the HP-UX routine tempnam which creates a file with a unique name in /usr/tmp. More HP-UX Features Qedit/UX was modified to execute shell commands (a shell is like the MPE/iX CI.PUB.SYS program). One of the most common HP-UX commands is ls (like the MPE Listf command). This conflicted with the Qedit command LSort. In Qedit/UX, we made ls mean list files and LSO mean sort lines (although the LSOrt command is not yet implemented). HP-UX users can have different shells. We changed Qedit/UX so that when it executes shell commands it does so with the user's preferred login shell. We could not obtain command line arguments in our SPLash! programs. Fortunately, Stan Sieler worked on the problem and modified SPLash! to make the argument count and a pointer to the argument list available to SPLash! programs. HP-UX programs expect options to be specified by a leading dash. We changed Qedit/UX to accept command line arguments such as this: qedit -s -cvi file1 This example asks Qedit/UX to edit a single file (-s), go directly into visual-mode (-cvi), and to edit file1. This also made it possible to set the HP-UX EDITOR variable, so that any HP-UX program that invoked an editor would invoke Qedit/UX on behalf of the user. Tab Characters One problem that we have not solved adequately is how to handle tab characters. In MPE, tab characters are used as a short-hand way of moving the cursor across the screen. The tab character is always translated into spaces and never stored in files. On HP-UX, the tab character is also used to move across the screen, but it is also used as data. Some programs (e.g., make and sendmail) require that certain lines have tab characters. In Qedit/UX, we automatically expand tabs to spaces when we text a file. This is not what HP-UX users expect, especially if you are editing a Makefile or the sendmail configuration file (important data gets removed by Qedit/UX). Qedit/UX has a feature to prevent tabs from turning into spaces, but it is not the default. If you leave tabs as data and go into full-screen, you will lose the tab characters that were in the lines shown on the screen. There are various technical problems that make it difficult for Qedit to treat tabs as data. How to handle tab characters is similar to our number-of-records problem in the file system interface. It's a fundamental assumption that is different on HP-UX from MPE. We continue to work on ideas in Qedit/UX to solve this problem. Releasing a Product Besides all of the technical and programming problems, there were many other issues to deal with. We had to create a whole new set of software to make tapes. I spent two full weeks rewriting the Qedit documentation so that we could have both a Qedit/UX user manual and on-line help. There were data sheets to write, advertising, training (every technical person at Robelle has taken the week-long HP-UX introductory course), and pre-release customers to work with. It was all worth it when we first demonstrated Qedit/UX at the San Francisco Interex User Group Meeting. We had many excited customers and just a week later we started shipping Qedit/UX. Phase Four Summary Our final phase took a working R&D program and turned it into a new product. Many changes were made to integrate Qedit/UX into HP-UX. The supporting documentation and technical support were created to provide a high quality product for our customers. The time that elapsed for this phase was about five months (May to September). The work time: Dave Lo Two person-months. Bob Green Two person-months. David GreerTwo person-months. The Future of Qedit/UX We have proven that the initial concept of cross-platform development using SPLash! was feasible. The object-code format on MPE/iX and HP-UX are the same. It is possible to compile SPLash! code on MPE, copy it to HP-UX, and then link and run the resulting program on HP-UX. In total, we ported over 100,000 lines of code. We could never have rewritten Qedit in the same amount of time. Full-Screen Our use of the blmode routines enabled us to get full-screen mode up and running quickly. Now that Qedit/UX has been released for several months, it is clear that even on HP-UX many users have decided to use vt terminals or vt terminal emulators. Because full-screen uses HP block-mode, it requires an HP terminal and will not work with vt terminals. One major research goal for 1994 is to reengineer full-screen to use a terminal-independent method of communicating with the user. Shell Commands On MPE, users are used to living inside Qedit all day. You cannot do this in Qedit/UX, because many common shell features are missing (e.g., setting environment variables). Some of these are due to limitations of UNIX, but we expect to implement many more shell features in Qedit/UX. It took us more than ten years to fully integrate MPE command features into Qedit/MPE, so it isn't a big surprise that it will take us a year or two to add all the necessary shell features to Qedit/UX. MPE Flavor Qedit was originally written for MPE. There are many features of Qedit/UX that retain an MPE flavor. If we are to expand the potential market of Qedit to users who have never seen MPE, we will have to remove most of this MPE feel and introduce more of an HP-UX feel. This is a challenge, since we still want Qedit/UX to look familiar to former Qedit/MPE users. The End Qedit/UX has been both a technical and a marketing success. We now have one set of source code that can be used to generate two different products: Qedit/MPE and Qedit/UX. If you have a lot of SPL/SPLash! code, perhaps our solution will work for you as well.