Richard L. Potyok
I.P. Sharp Associates
2 First Canadian Place, Suite 1900
Toronto, Ontario M5X 1E3
CANADA
The Network Shared Variable (NSV) and Network Shared Variable Processor (NSVP) provide a user signed on to a local SHARP APL system with the ability to communicate with a remote SHARP APL system over a communications interface by sharing a variable with a task on that remote system. More specifically, NSVP extends the shared variable protocol to allow communications between tasks on different computer systems.
Together, they represent dynamic innovations in APL technology that provide greater computing flexibility and power to the end user and the data centre. NSVP encourages users to maximize the potentials of distributed applications as it helps organizations optimize their data centre resources. NSVP also reinforces the image of APL at the forefront of system software technology.
The NSV, while transparent to the user after coupling, enables users to access data from other systems using just one set of commands, while it minimizes the human and hardware resources required to maintain the data centre.
In several ongoing tests of the NSV prototype, the product has proven its value to a variety of organizations around the world with substantially different needs.
(c)1988 ACM 0-89791-253-5/88/0002/0264 $1.50 264 Richard L. Potyok
Fig.1Interaction of Programs, Shared Variables and the Shared Variable Processor
The concept of the Shared Variable (SV) evolved in the early 1970s as a mechanism to allow greater access to APL [1]. The shared variable was developed to provide a method by which an APL program could access another APL program or data files. Shared variables also allow users to access other facilities which are not part of the APL system, such as output spooling systems, graphics packages or programs written in other languages.
A Shared Variable Processor (SVP) facilitates communication between partners using shared variables. The same SVP is utilized by all SV users on a given machine. It is not a component of either partner, but an entity between them. A number of APL primitives are used to establish, control and retract SVs that are invoked by the partners, but processed by the SVP. The SVP also maintains various tables to keep track of the transactions between pairs of partners.
Fig.1
Interaction of Programs, Shared Variables
and the Shared Variable Processor
An auxiliary processor (AP) is a program (usually written in a language other than APL) that runs as a unique task with which another SHARP APL task may communicate via a SV. It enables users to access data or services not otherwise available to a SHARP APL task.
As of October 1987, I.P. Sharp offers twenty-one APs to provide users with a great deal of enhanced functionality [2]. One such AP is SSQL, an interface between SHARP APL and DB2, IBM's relational database management system. SSQL enables a SHARP APL task to execute statements in IBM'S Structured Query Language and thereby manipulate DB2 tables.
A major restriction in the use of shared variables has been a requirement that both partners be on the same system. It is this restriction that is addressed by NSVP.
NSVP, developed and refined over the past four years by I.P. Sharp Associates' APL System Development Department, in Toronto, extends the shared variable protocol to computer systems which are connected by a communications link. This link can be IPSANET, I.P. Sharp's private packet-switched network, or VTAM.
The channel is assumed to be logical duplex with a several second delay and low bandwidth (less than 2000 cps), although the actual implementation might be half duplex. In either case, the exact properties of the channel do not affect the facilities offered.
A pilot project produced a version of NSVP written in SHARP APL in 1984 [3]. The success of this iteration encouraged us to rewrite NSVP in assembler to increase its speed, efficiency and ability to handle more data.
In simple terms, NSVP runs a pair of auxiliary processors on different computer systems. Fig. 2 illustrates that when NSVP is installed on each system, a task on one system can share a variable through the pair of NSVP tasks to a task or auxiliary processor on the other system. When the NSV is fully coupled on both systems, the NSVP can be utilized to pass data in a manner similar to that used to share variables on the same system.
For security reasons, some restrictions can be placed on the offers to share the NSV, since this remote connection creates greater security exposure than a plain vanilla SVP. For example, a user can never be absolutely certain of the identity of a user with whom he is sharing data via NSVP.
Fig. 2
Distributed Processing Between Computer Systems
Using NSVP
A major advantage of NSVP over other communications technologies is the relative simplicity of the user interface. Users only need to know (the shared variable protocol to communicate with the NSVP on a system. Each program offers a control variable, with which it passes information to the NSVP, and a data variable to pass values through the NSVPs to another computer system.
The commands that pass through the control variable include remote offering and retraction of variables. The remote offer command contains the remote user IDs as well as the local and remote variable names. Both user programs must make a remote offer to couple to establish communications.
The SHARP APL coupling instruction is shown in Fig. 3. The number 13 in this example is the processor number for the NSVP.
After the variables are coupled, data can be transmitted between the systems.
Fig. 3
NSVP Offer Commands
Once the user program has set the variable, the local NSVP detects the set via the SVP and moves the object into a work area, where it is transformed into NSVP internal format After attempting to compress the transformed data, it is transmitted. The transmission process is illustrated in Fig. 4.
Fig. 4
Network Shared Variable Processor
Data Flow
The receiving processor reconstructs the object and sets the remote variable. When the NSVP on that system detects that the value has been used, it transmits an acknowledgement to the transmitting system.
There are two general ways to represent SHARP APL objects in an NSVP-NSVP interface. One method employs the internal form in which objects are passed to and from a SVP. The other method translates the internal SHARP APL form into a format which is somewhat independent of the APL implementation on the two systems.
The internal form offers the advantages of speed and simplicity. It is faster because it requires only a straightforward conversion to a byte serial format.
The object is partitioned into discrete pieces, without conversion or compression, that are enclosed in control data for transmission. The lack of data compression keeps this system simple.
There are several limitations to this method of representing SHARP APL objects, however. Both systems must be operating the same version of system software and the internal representation of the objects must have the same meaning to both partners. A very wide bandwidth is required, since neither compression nor conversion are used. Also, for security reasons, one organization must have control over both systems.
If these requirements cannot be met, some degree of conversion is required to translate the SHARP APL objects into a format that is more independent of the two systems. In this case, the objects must be converted into a NSVP internal format which is based on, but different from the APL internal format. The internal format also involves data compression to achieve more efficiency within a smaller bandwidth. This approach has been followed in the current implementation.
The origins of this scheme are partly historic. The compression protocol was introduced to IPSANET in 1978 as part of bisynch support. It permits the transmission of intermixed single byte controls (48 values) with data (256 values).
The mapping of APL objects onto the 1978 bisynch protocol was done in 1983 as part of a file transfer scheme. This abandoned file transfer project provided a basis for the original NSVP [4].
Conversion to a form which is less opaque than (the internal representation does not always achieve compatibility between different systems. For example, an object, such as a complex number, which is allowed in the sending domain may not be permitted in the receiving domain. There can also be problems handling real numbers if the exponent or mantissa limit for the sending system is larger than that of the receiving system. For future implementations involving disparate hardware systems, we will have to consider further degrees of data conversion.
In SHARP APL the value type known as a 'package' also aggravates compatibility. A package may include functions in an internal format and these functions can be locked. This creates a new specification issue. If the function is locked because the author did not wish it to be moved to another APL system without consent, for example, the NSVP should be reluctant to transport the locked function.
here is a proposed transfer format standard for migration of workspaces between all versions of APL [5]. This was not used in the current version of NSVP however, since this version has been more concerned with speed than with compatibility. As the number of applications increases, this issue will have to be re-evaluated.
The SVP interface sets and uses APL objects as SBENTRYS (a SHARP APL internal format). This is not the format in which the data is transmitted through the physical network.
The conversion from the SVP input form to the transmission format is a two stage process (see Fig. 4, Network Shared Variable Processor Data Flow). The first stage linearizes the object. Using the case of a character matrix, the transformation would be a few bytes of control data followed by the ravel of the data value, a few additional control bytes (including the data object type), and finally the shape vector and more control information.
The second stage attempts to compress this data stream into the representation in which the data is actually transmitted.
For the receiving end, the incoming data is appended to the end of the input buffer and deciphered from the beginning of the buffer. The data portions that were compressed for transmission must be reconstructed. In the case of numeric data, the representation is also changed (i.e., from display to internal representation). While the data is retrieved, the APL object is also being reformatted. When an object is completed, the appropriate variable is set.
The communications link restricts the size of a transmission block. The receiving NSVP reads blocks and catenates them until the input buffer contains a complete object. This object is removed from the buffer and converted back into the SVP format.
Fig. 3 illustrated that SVP and NSVP commands are required to couple a NSV. Similarly, two commands are required to retract a NSV.
A soft retract attempts to mimic SVP behaviour. A ▯SVR issued by program 1 creates an offer to program 1 and causes the coupling level observed by program 2 to drop to 1 (i.e., offer extended, but not coupled). This is accomplished by sending a message from NSVP 1 to NSVP 2 and making appropriate adjustments in the NSVP tables.
After a retract is issued by user 1, there are two other user actions which can change the coupling level. User 1 may reoffer the variable to restore the coupling level to two (i.e., fully coupled). If NSVP 1 detects this reoffering by user 1, a message is sent to NSVP 2 to force a reoffer by NSVP 2. Alternatively, user 2 might issue a retract command before user 1 reoffers a variable. In this case, a retract message is sent to NSVP1.
When a NSVP has both sent and received retract messages, it knows that the coupling level has dropped to 0 (i.e., no offer to couple). The NSVP table entry for that variable can be deleted.
If the second end performs a retract before the first end re-offers, this is called a 'hard' retract The connection cannot be re-established by a simple SVP offer, since NSVP has no memory of the shared variable after a 'hard' retract. Re-establishing an offer is the same as the initial establishment.
If one end wants to terminate a share completely, a simple SVP retract does not suffice. However, there is a NSVP command by which a single partner could cause a hard retract.
It is the usual process for APs and user tasks to scan for offers extended by the partner. When they detect an acceptable offer, they make a matching offer to complete the share. Part of the information supplied by the scan is the IDs of the offering task.
This is not simple for the NSV situation, however. While the SVP scan does return the local NSVP ID, the important IDs are those of the remote user. A unique NSVP scan command can be invoked to return the IDs of the remote user and all local and remote shared variable names.
NSVP virtually ignores the Access Control Vector (ACV). No attempt is made to pass an ACV across the communications link. NSVP is indifferent to the ACV value set by a local user. NSVP can never be blocked by an access control interlock since the local NSVP will not attempt to use the local variable unless the remote user has used the last value transmitted and the local user has set a new value.
Consider the following. A user sets a variable. The local NSVP detects this set, uses the value, transforms it into a NSVP internal format, and finally transmits the resulting character stream to the remote NSVP.
On receipt of the data stream, the object is reconstructed. Then, just prior to setting the variable for the remote user, NSVP checks the intended variable and determines that there is a pending set by the remote user. We call this situation a simultaneous set condition.
NSVP's solution is to throw away one of the data objects by oversetting it with the other value. A 'yield parameter' associated with each NSV dictates which end will be overset. This parameter is specified as part of the share offer command. There are two possible solutions to simultaneous sets.
If the remote end must yield, its variable is overset and its status is set with a value to indicate that an overset has occurred. When the remote NSVP detects the use of the value by the remote user, it transmits an indication to the local NSVP that the remote system has used it. This results in the local status being set with a successful completion value.
If the local end must yield and the remote NSVP detects a pending value set just prior to setting the variable, the remote NSVP rejects the value received from the other system. An indication of this rejection is transmitted to the local NSVP, which then proceeds with normal processing of the object set by the remote user. When the local NSVP receives the rejection indicator, it is transformed into a value to set the local status variable. The overset value follows.
Fig. 5
NSVP Used to Access Application
via Auxiliary Processor
Here are three examples that demonstrate how NSVP has been applied to real business situations.
A large European bank running its own computer system requires data that is stored on the I.P. Sharp public databases. NSVP provides access to the remote data without increasing the bank's own computer storage requirements. This helps to increase the operating efficiency of the data centre and relieves the support staff from the responsibility to store and maintain large quantities of data.
NSVP also provides a major international energy company involved in worldwide trading with a 'hot' standby back-up of their real-time data feed. Since data is transferred to the back-up site as it is created, this provides traders with peace of mind and confidence that they can continue work without interruption.
If the main system fails at any time, the organization can rapidly transfer to the back-up system and continue operation, knowing that the most recent information is immediately available.
Finally, Fig. 5 illustrates how NSVP can be used to provide organizations running applications written in different languages on separate mainframes to access information using a SHARP APL auxiliary processor such as SSQL.
In this case, NSVP provides the organization with enhanced flexibility to manage their resources in the most efficient manner. For example, data might be stored on one machine, and applications on another. This total management control provides users and the computer centre with greater power by enabling and encouraging them to capitalize on the flexibility made possible by distributed applications.
NSVP has evolved from the traditional SVP as a method by which a user signed on to a local SHARP APL system can communicate with another system over a common interface to share an application or data. This is accomplished by sharing a variable with tasks running on the two systems.
It provides users with increased functionality as it improves the efficiency of the data centre.
The current implementation in assembler language provides a straightforward method to exchange information which is largely transparent to the end user. When coupled, via the SHARP APL coupling instructions shown in Pig. 3, data can be passed between the systems in the internal SVP format, or in a format which is somewhat independent of the two systems. When information sharing between the systems has ended, the applications invoke retraction commands to drop the IPSANET or VTAM communications link.
NSVP represents a major advance in SHARP APL system software which enhances the ability of organizations with dispersed operations or unique information needs to address those needs affordably. It helps to optimize an organization's human and hardware resources.
As a technology, it demonstrates the versatility of APL to meet the expanding computing needs of the approaching decade by building on the basic elements of APL. It helps to reinforce the image of APL as a vital component on the leading edge of system development technology for the 21st century.
I would like to thank Roger Moore for his invaluable advice and assistance in the development of NSVP and for his special assistance and comments on this paper. I would also like to express my appreciation to W. Craig Heron for his help to prepare this submission.
[1] R.H. Lathwell, "System Formulation and APL Shared Variables", IBM Journal of Research and Development, Vol. 17, No. 4, July, 1974.
[2] SHARP APL Auxiliary Processors Manual, I.P. Sharp Associates, March 1987, I.P. Sharp Publication Code 0452 8703 E19.
[3] I.P. Sharp Associates internal documentation.
[4] Bisynch Users' Manual, Version 1.2, I.P. Sharp Associates, 1982, LP. Sharp Publication Code 0332.
[5] "Language Standard for Programming Language" APL", ISO/DP 8485, February 1986.
APL Quote Quad; 18,3 271 Richard L.Potyok