XEDIT history - blinkkin/blinkkin.github.com GitHub Wiki
Copyright (c) 2010, Cheyenne Wills
original message was obtained from [email protected] archives
Let me take a stab at this.
A lot of how editors "work" are based on the environments that they originated from, and there is a huge amount of historical baggage being pulled along.
So.. first a history lesson (yes -- this is long... but hopefully informational)
The history of XEDIT (working backwards in the CMS environment) XEDIT <- EDGAR (kind of) <- EDIT <- (maybe some others) <- punched cards
The 360 I/O architecture was record based. Meaning that data was read from and written to devices a record at a time instead of a byte at a time (in Unix speak, all 360 I/O devices are block devices). For example, lets take a punched card. A punched card consisted of 80 characters. One card corresponded to one "record". It took one I/O operation to read or write this one card.
Programming in the punched card days consisted of using a keypunch (for example an IBM 029), where when you had a typing error, you needed to replace the entire card. "Newer" ones had a small 80 character buffer that allowed some very primitive typing corrections (IBM 129 -- allowing the use of the backspace to correct a simple keystroke errors).
So programming with a deck of cards, you could:
- replace a card with a different one
- insert new cards following a card
- delete cards
- and if you had a spiffy enough keypunch machine, you could read a card into a buffer, and change some of it's content.
Contrast this with early DEC systems that used paper tape instead of cards. Paper tape was a continuous stream of characters, read one at a time. TECO (Tape Editor and COrrector) was the historical origin of EMACS (Editor MACroS). And vi's heritage was ed <- qed.
Step forward a little. Early terminals where not display terminals, but some form of teletype. Again in the IBM world these were block devices (IBM 3215 console). The terminal would buffer data until the operator would transmit the data via certain keys that would signal the computer that the device had a buffer (record) of data ready. CP and CMS to this day still uses this model for it's command line mode, and CP's TERMINAL settings still reflect the special characters used for "line editing" (CP Q TERM, TERM LINEND #, LINEDEL @, etc.) (same as Unix still uses a /dev/tty model for it's command line interface).
The CMS EDIT editor was targeted for the teletype environment. The data metaphor was still a deck of cards, where each line (record) corresponded to a single card. So a CMS data file was "viewed" as a deck of cards, and a single line of the file was a single card. The only "cursor" position was "what is the current card" (i.e. the current line position -we will call this position the "current line").
Every operation with EDIT was via a command line "command".
The interaction between the teletype (console) and the editor was entirely via an entire line of data. Any keystroke editing was handled in order by:
- the device itself
- CP via the terminal line editing characters
So EDIT only "saw" a completed line as it's input.
So with the CMS EDIT editor, you could:
- Adjust the current line position within the "deck"
- +/- relative to the current line
- absolute position within the "deck"
- the line that matches a simple string search
- replace a single "card" of data with a "card"
- insert a new "cards" following a specific position in the "deck"
- delete a specific "card"
- change part of the content of a single line by using simple string substitution (i.e. change /a/b/ -- > change "a" to "b" ).
The basic editor "logic" would look something like the following: Using some pseudo code
dcl curline integer
dcl data[numberoflinesinfile] string(80)
dcl buffer string(80)
dcl command string(variable)
dcl params string(variable)
do forever
buffer = readfromconsole() # Returns a single line
command,parms = parsebuffer(buffer) # 1st token, rest of string
if command = 'quit' then exit()
if numeric(command) then curline = curline + command
else if command = 'replace' then do
data[curline] = params
end else if ....
end
So moving forward a little more.
3270's (yeah!!)
The 3270 was a display terminal, gone was the paper for the teletype. You could display a whopping 24 rows of 80 columns. Each 3270 maintained a "large" buffer that represented what was displayed on the screen along with "field" attributes (along with some other stuff). This buffer was "addressable" (indexed) by row and column. The interaction between the 3270 and the system (and application) was via a structured data buffer. This structured data buffer consisted of 3270 "commands" as well as data. With the 3270 "commands", the system could:
- clear the screen
- Set a Buffer Address (row/column) for where the next "3270 buffer" operation was to take place
- Start a field with attributes (i.e. "protected", hidden, high lighting, etc.)
- data to be displayed
So a typical data buffer sent to a 3270 would look something like:
<wcc><SBArc><SF><FA>blahblahblah<sbaRC><IC><SF><FA>blahblahblah
where:
<WCC> would do things like reset the keyboard,clear the buffer (screen), ring the alarm, etc.
<SBArc> would SetBufferAddress to a row and column
<SF> would start a new field
<FA> would be the field attributes (such as numeric input only, highlight, etc.)
<IC> would "move" the 3270 cursor to this location
blahblahblah is the data to be displayed.
The 3270 itself was responsible for a bunch of "local" functions, such as "reading" the keyboard and displaying the key pressed at the current cursor position, local buffer editing (i.e. overtyping vs insert mode, deleting characters within a field, etc.), moving the cursor around on the screen, all under the restrictions of the defined fields and their attributes. For example if the user moved the cursor to a location on the screen that was defined as "protected" the 3270 was responsible to "lock" the keyboard and provide some audible feedback, or only allow numeric values to be entered.
Once the user had performed whatever operations they wanted, they would then press certain keyboard keys that would signal the 3270 to tell the system that a buffer of data was ready. The system would then request that the 3270 transmit a structured data buffer back to the system. This structured data buffer would contain:
- the AID (Attention IDentifier) - the key that caused the 3270 to say it had data ready (ENTER, CLEAR, one of the Function keys, and a few others).
- the cursor position
- buffer address (row/column) followed by the data within that field
- next buffer address (row/column) followed by the data
- ...
The system could request either the entire buffer, or just those fields that where modified.
Under VM, CMS still operator as if it was talking to a 3215 (linemode), and CMS did not understand what a "3270" was. CP handled all the 3270 interaction, the display was divided into:
- a protected output area (top 22 rows of the screen)
- input area (a field starting on row 23 and going for 132 characters) - cursor placed at the start of this field.
- a status area in the lower right part of the screen (row 24, for 16 characters). (those counting chars will see that 132+16 does not equal 160, the field attributes actually take up "screen" space).
CP would read the input, and provide it to CMS making it look as though it came from a 3215. CMS would simply write single lines to the "console" and CP would determine what part of the 3270 buffer the output should be displayed in. If the current output line would exceed the display area, CP would change the status to "MORE...." and would wait a few minutes for the user to clear the screen, or it would timeout and clear the screen and continue displaying.
So.. CMS is still using just single line input and single line output for all it's user interaction. Next CP defined an interface allowing for some limited interaction with the 3270. CP would still "manage" the buffer, but would allow an application perform some 3270 operations via the DIAG interface (the DIAG instruction was CP's way of providing an application programming interface back to CP services i.e. a hypervisor call). The initial services provided by this interface were fairly primitive. The application could:
- clear the screen
- say which line on the screen the output should go on.
CMS EDIT goes full screen (kind of)
The first pass of updating the EDIT editor to utilize the 3270 allowed the editor to simply display 21 lines of data. The current line was placed in the middle row of the screen. You had the 132 character command/input area. The top row contained the editor status line, which listed the name of the file, and what current line number (the index value into the array of lines). The output area was protected, meaning that you could not move the cursor to a line on the screen and "overtype" what was there. You still had to perform your all of your editing tasks by using just the command line with one really neat new feature. You could "yank" a copy of the current line into the command/input area, perform any "editing" changes and "put it back".
Why the middle of the screen for the current line? (my theory)... The initial 3270s did not have highlighting. But there were two little lights along the left side of the screen to indicate power (and something else). These two little lights where about halfway down the screen. Ah ha!! a visual clue to the middle of the screen. So.. the current line was placed there. You could see the current line in "context" with the other data. A common "hack" during this time was to take a piece of clear tape and place it on the screen right where the current line was (along with a line drawn in pen, or whiteout). When 3270's started to support character highlighting, CP updated the interface to allow the application to set the attribute, and EDIT would highlight the current line.
Part 1 brought us up to the CMS EDIT command getting a "full screen" interface.
So.. at this point a programmer in the VM environment had a basic editor that was still pretty much based on the card metaphor. While the data was displayed on a display terminal, all editing was performed using just the command/input area. It was possible to write very primitive "macros" that could automate some editing tasks.
At this point, it is probably a good idea to introduce "macros", and a little side journey to CMS's scripting facilities. At the time of EDIT, there was only one CMS scripting language, "EXEC", about the closest thing one can equate the classic "EXEC" language is MS-DOS's .BAT language. The EXEC language by todays standards would be extremely primitive, with 8 byte "tokens", and very rudimentary logic flow, etc. However, CMS itself provided a very interesting API, the CMS console STACK. In fact the CMS console stack is still present, and used to this day.
The CMS console STACK is simply a buffer that the CMS command processor, as well as the API that CMS applications used to read from the console. If this buffer was empty, then CMS would simply wait for input from the console. An application can push lines on to the stack in either FIFO or LIFO order, then whatever was reading from the console would consume these lines and not even know that they came from a program.
The way that this was used within EDIT, was that one would write a CMS EXEC that performed whatever logic it wanted then "pushed" a series of EDIT commands onto the stack then exit. EDIT would then read these stacked lines and perform whatever operations were requested. EDIT had a command to invoke CMS commands directly, which included the EXEC processor.
So given the following file:
--[DOIT EXEC A]--
&CONTROL OFF
&PUSH C /A/B/
&PUSH LOCATE /XYZZY/
&PUSH INPUT HELLO
&PUSH 5
&PUSH TOP
--[EOF]--
From an EDIT session, one could enter the following command:
EXEC DOIT
and the EXEC would be executed, and then the EDIT would perform the following commands:
TOP <-- top of the file
5 <-- go down 5 lines
INPUT HELLO <-- insert a new line: HELLO
LOCATE /XYZZY/ <-- locate the next line that contains the string XYZZY
C /A/B/ <-- and change the first 'A' character to 'B' in that line.
I remember having an EDIT macro that would take a short string and generate a "banner" of block letters from that string, for example the string "AB" would produce:
AA BBBBBB
AA AA BB BB
AA AA BB BB
AAAAAAA BBBBB
AA AA BB BB
AA AA BB BB
AA AA BBBBBB
It was a "wonder" watching the screen as EDIT built each letter at a time from the list of stacked lines.
The CMS command processor eventually had EXEC2, a much improved version of the CMS EXEC language, gone where some restrictions on the length of data (no more 8 byte token limits), much improved string handling, but most importantly, CMS introduced the SUBCOM facility. With this facility, any application could register with CMS a callback interface that the EXEC2 processor could feed commands back to. EXEC2 was soon joined by REXX, and the whole CMS scripting facility just took off.
While the 3270 was still fairly "new" a new editor became available as an extra product, the Display Editing System for CMS, EDGAR. About the same time CP was also enhanced to provide a better interface to allow guest systems to access more capabilities of the 3270.
EDGAR took advantage of the 3270 terminal. Many of the display features that EDGAR had would look very familiar to an XEDIT user. It had the status area along the top, command line, followed by the data area. Each line within the data area was prefixed by a "prefix" area where some commands could be entered to move, copy, or delete the associated lines. EDGAR also provided an API by which a CMS command (including EXECs) could "call back" into EDGAR and invoke a single EDGAR subcommand.
The above DOIT EXEC could be rewritten for EDGAR as:
--[doitedgar exec]--
&CONTROL OFF
ECOMMAND TOP
ECOMMAND 5
ECOMMAND INPUT HELLO
ECOMMAND LOCATE /XYZZY/
ECOMMAND C /A/B/
--[eof]--
</sidebar>
EDGAR picked up where EDIT left off, in order to maintain some familiarity with the past, EDGAR kept pretty much the same command set as EDIT for the most part. Files were still treated as a "deck" of cards, there still was the concept of the current line, and since the 3270 itself handled all cursor movement, and "local" editing, EDGAR only really concerned itself with "records" and the entire file.
However EDGAR did not have a long history of usage because XEDIT came along just a couple of years later, providing just about all the features of EDGAR but so much more. There is one feature of EDGAR that I miss, that XEDIT never duplicated (it was how the field mark key was handled) oh well..
XEDIT
When VM went from VM/370 to VM/SP (VM Systems Product), one of the new features was this new fangled built in editor, XEDIT. XEDIT was shipped as part of CMS (not as an additional addon as EDGAR was). Provided most of the same features as EDGAR, and then some.
Again XEDIT picked up where EDIT left off, it still maintained roughly the same command set, data was still treated as a "deck" of cards, you "moved" through the file by adjusting what line was the current line.
One of the new features of XEDIT was the concept of the column pointer. This feature took some of the concepts how one moved through the file (up a line, down a line, locate a specific line, delete a line, etc.) and applied them to working from a column within one line.
With the "Column" commands, you could, move a current column pointer around within a line. You could:
- move the current column pointer around
- clocate /some string/
- clocate :5 <- absolute column 5
- clocate -3 <- relative to the current column
- delete characters (cdelete)
- insert characters (cinsert)
- replace characters (creplace)
- overlay characters (cover)
XEDIT also provided a vastly improved macro facility via CMS and the SUBCOM interface. By using this facility, editor macros (written at first in the new EXEC2, and later in REXX), could interact directly with the editor. It was now possible to easily extract information from the editing session and control it in a programmatic fashion.
So.. next Part 3 - XEDIT
In part 1, I described some of the early history of editing in CMS, as well as the core concepts of how a 3270 worked, in part 2, I described the introduction of full screen editing with EDGAR and the introduction of XEDIT. In part 3, I will describe some of the core concepts of an XEDIT session, and try to relate back to THE, vi and emacs at a rudimentary level.
So with the introduction of XEDIT to CMS, the VM programmer now had a full scale editing system. It maintained a familiar command set from prior editors (EDIT and EDGAR). Coupled with CMS's SUBCOM facility, a user could develop extremely powerful programs that could interact directly with the editor.
In fact, the base command set and features of XEDIT grew, XEDIT itself became a useful display management facility. Several "non-editing" CMS commands where built around using XEDIT as a "graphical user interface", FILELIST, RDRLIST, NAMES, to name a few.
XEDIT core concepts.
(disclaimer... it's been about 20 years since I dug into the internals of XEDIT, so I forget exactly how XEDIT actually stores some of this data, wither each line is in a linked list, or some by blocks of data and pointers into those blocks, etc. or what level some of the attributes are maintained at.. so please forgive a foggy memory).
In learning XEDIT, there are several core concepts.
- The screen, display and views
- File representation
- The ring of files
- Commands
- Targets
Screen, display and views
VM/CMS can only exist as a single session, meaning that you log into a VM virtual machine, boot CMS, and all your interaction with CMS is via that one login session. CMS does not support multiple concurrent sessions (meaning that you cannot have multiple 3270 sessions into the same CMS virtual machine. (In Linux concept, no X11, and just a single console session - no ctrl-alt-f2, etc.)
So lets start with that single 3270 session, with a 24x80 screen size (later 3270 models provided larger screens).
XEDIT divides the 3270 screen into:
- a status line
- a command line
- file area
- message line
- (optionally a scale line)
- (optionally a tab line)
- (optionally a prefix area on either the right or left side of the file area)
- optional user define reserved lines
XEDIT can subdivide the screen into multiple "screens" The "screen" command defines the number of sub-screens that are created and their orientation (split vertically, or horizontally, etc). Each subscreen has all the areas listed above (e.g. each has a command line, a file area, etc.). Each subscreen is independent of the other subscreens (for example you could turn off the prefix area in just one subscreen, a command entered in one subscreen only acts within that subscreen, etc.). For the rest of the discussion, I will assume just one screen unless otherwise noted.
On this screen, certain 3270 operations can take place. The 3270 TAB key can be used to move from one 3270 field to the next. Starting with the cursor in say the command line area, pressing the TAB key repeatedly the cursor would move from the command line area to the prefix area, to the file area, to the prefix area in the next row, to the file area on that row, and eventually cycling back to the command line.
An important concept to be aware of is that each row in the file area is a separate 3270 field. A 3270 fields are the smallest unit of data that is transmitted from the 3270 to the system on a I/O read to the device.
Within an XEDIT macro, it is possible to simulate some 3270 operations using the "Screen Operation Simulation" (SOS) commands. So it's possible from an XEDIT macro to move the cursor to a particular row and column on the screen, or to "tab" the cursor to the next field, etc.
A screen is used to display a view of a file. Associated with this display of a file are several values:
- current line pointer
- current column pointer
- the display scope
- the VIEW which consists of a right or left "offset" of the file
- the row on the screen on which to display the current line
Scrolling in the screen basically means adjusting the value of the current line pointer and displaying the data accordingly.
File representation.
When a file is read in from disk, each record (line) is placed into an internal data structure that allows each line to be individually indexed.
Associated with each file is:
- the list of lines
- a count of the number of alterations to that file
Associated with each line is:
- a scope value
- a list of "point" names
The ring of files.
XEDIT can handle multiple files. The files are maintained in a circular list (the ringlist). There is an association between a particular file and a display (which contains the current line pointer, etc.). It is possible to cycle through the files individually on the screen
Commands
XEDIT commands can generally be divided into:
- global configuration (defining key settings, synonyms, etc)
- screen layout (e.g. screen, location of cmd line, etc)
- display layout (views, zones, display scope, etc)
- "scrolling" (locate, find, etc)
- acting within a single file (loading a file, saving, fileid, etc.)
- acting within a single line (column commands, change, replace, delete, etc.)
- macro support commands, these are specific commands used usually within macros to interact programatically with XEDIT, or bound to keys, for example the 'extract' command, the SOS commands, etc.
Some commands can be repeated over several lines, or can have a scope that includes several lines, for example:
change /a/b/ 5
which would change a to b on the next 5 lines
For the most part, most editing tasks can be handled using the built in commands (see below on comments about macro depth)
Targets
Targets are a very important concept within XEDIT. Many commands will utilize a target. A target can be:
- an absolute number ( :x )
- a relative number ( +/-x )
- a named line ( .name )
- matches a delimited string
- a logical combination of the above (e.g. locate /a/ | .z )
Targets are normally lines, but column commands that use targets works within one line and "adjusts" or references the current column pointer.
Delimited Strings. XEDIT doesn't use escaped character strings, instead it uses the first character of the string as the delimiter, thus:
locate ,/,
will scroll to the next line that contains the "/" character
To give an example of how all this is pulled together, look at the change command syntax:
Change {delimiter}str1{delimiter}str2{delimiter} {target} {n} {m}
so..
c/xyzzy/plover/ /foo/ 3 2
Means -- change the string xyzzy to plover from the current line down to the line that contains the string foo, change only 3 occurrences of xyzzy within each of those lines and start with the 2nd occurrence of xyzzy within each line.
Comment about macro depth
The following is a personal view of writing editor macros (or scripts). I personally try to avoid creating or using macros that rely on a series of other macros that increase the distance from the base command set. For the most part, I will usually only go to a depth of 2 (maybe 3) "layers" away from the base XEDIT commands. (e.g. mac1 is built using mac2, which only uses base XEDIT commands). My reasoning behind this is partially understanding and partially maintainability. I've found over the years that the XEDIT command set is extremely rich and I can perform most editing tasks using just the base commands.
Finally trying to tie this all back to THE, vi and emacs.
Outside of the IBM mainframe world, most data is stored as a stream of bytes, where any structure associated with the data is only defined by the application. While some devices (e.g. disks) do work with blocks of data, this is usually hidden from the application.
Lines of data are simply a series of characters with an "agreed to line separator character"
User interaction outside of the IBM mainframe world is also usually at a character at a time.
Comparing XEDIT (and other mainframe based editors) to vi/emacs (and other non-mainframe editors) means understanding the differences between record based I/O processing and stream based I/O processing.
Record based: Only entire records (blocks) of data can be transfered between the device and the system, and the system and application will utilize this "record". Record based terminals or consoles (e.g. 3270) can perform rudimentary data editing within their data buffer before transmitting the data to the system.
Stream based: A single byte of data can be transfered between the device and the system (for character based devices). The system and application will sometimes define a structure to the data, but mostly the data is just treated as a stream of bytes. Terminals or consoles do not usually perform any local data editing, but push that task up to the application (or the OS).
Vi (and emacs) rely on an "editing mode", and single character or chorded character command sets. For example (picking on vi) look at what the following keystrokes do:
jjjlllxxxixxx{esc}:wq
move the cursor down 3 lines, right 3 chars, delete 3 chars, change to insert mode, insert the characters "xxx", exit insert mode, enter command mode, write the data to the current file and quit.
Each one of those keystrokes was processed directly by vi. Vi needs to manage all the different modes and states.
With a record based terminal, all the cursor movement, character deleting, and switching between insert vs overtype mode are handled directly by the terminal. Instead of entering "command" mode, a special data field is used as an area to enter commands, and this data field is transmitted to the application (along with any other modified data fields), where the application simply needs to determine the mapping between the field and the specific lines within the file, and process any commands as entered in the "command" field.
Finally pulling in the "The Hessling Editor". Emulating XEDIT in a stream based system, means emulating record based concepts. Terminal input needs to be handled as a separate layer from the EDITing layer. The terminal layer needs to emulate the local editing functions as provided by a 3270. If the goal is to just emulate XEDIT, this should be sufficient (well there is implementing all the commands, etc.)
But there are area that the XEDIT emulation can be enhanced. Mouse processing, binding function to any key, etc.
There is one more comparison that I would like to make, and that is between EMACS and XEDIT.
EMACS is an extremely customizable editor. While there is a core set of EMACS commands, much of the power of EMACS is the creation of modes and extending the editing "environment" by adding in new functions and binding those to key sequences, as one continues to use EMACS they tailor the environment more and more to their own liking.
With XEDIT, while there is some customization, for the most part users tend to utilize the base command set, or a very small variation to those base commands. Most of the user customization deals with what is bound to the function keys, or just the layout of the screen.
So.... there's my take...