Auxiliary Processors
The Auxiliary Processor (AP) mechanism
It is sometimes necessary to write parts of an APL-based application in a low-level procedural language such as C, for example to speed up a critical routine, to re-use an existing library of subroutines, or to access some part of the system like a filing system for which no APL- or object-based interface is provided. Such a piece of code is known as an Auxiliary Processor. It is accessed through the Shared Variable interface described here.
(See also ⎕NA which may provide a more convenient and flexible alternative.)
Dynamic Binding
Auxiliary Processors in APLX are held as separate disk files and are loaded by APLX when required using a mechanism known as Dynamic Binding. This is a mechanism used by many modern programs which call standard library routines. Instead of the library routines being permanently linked to the program when it is created, the operating-system loader delays performing the linking until the program is executed.
As an example, suppose you write a program in C which calls the C library function printf(). On systems which don't support dynamic binding, you would compile the main program and then link it to printf() to create an executable file. Using dynamic binding, the routine printf() is not automatically linked; instead when you run the program the loader binds it to the current version of the printf() library routine. A similar approach is used in Windows for calling the operating system or other libraries. The routines which are called are held in Dynamic Link Libraries (DLLs).
APLX Auxiliary Processors are treated in a similar way. They are dynamically bound to APL during execution. The only difference is that binding doesn't occur when APL is first executed; it is further delayed until APLX is explicitly instructed to go and load the routine from disk and bind it. The processor is loaded into user memory outside the APL workspace, and so has no impact on the available workspace size.
To load an Auxiliary Processor written in another language, it must be written to be a shared library, and must be compiled to the appropriate format (for example, under AIX it must be in IBM's XCOFF format, and in Windows it must be a valid DLL). Most modern compilers will be able to produce shared libraries in this way. Examples in this chapter are written in C.
Loading and Unloading Auxiliary Processors
Binding of Auxiliary Processors is carried out through APL Shared Variable interface. To load a module from disk and bind it, you can use the APL shared variable offer function ⎕SVO to share a variable with Auxiliary Processor 3 (AP3), and then assign to it the name of the module to load:
3 ⎕SVO 'PROC' Share a variable called PROC
PROC←'myprocessor' Load the AP called myprocessor
If the specified processor name includes a full path description, an attempt is made to load from the specified directory. If the name does not include a path element, APLX uses a library path to locate the module. The exact behavior varies according to the host system:
- AIX: The library search path for external modules can be set in the APLX resource database by specifying the resource 'ap_libpath'.
- Linux: The search path can be set in the environment variable LD_LIBRARY_PATH; if it is not found there, Linux looks in the list of libraries specified in /etc/ld.so.cache, and finally in /usr/lib, followed by /lib.
- Windows: Windows looks in the directories specified in the PATH environment variable. If you omit the file extension in the processor name, Windows will assume the default file extension ".dll"
- MacOS: Under MacOS, APLX supports two different file formats for auxiliary processors. These are Code Fragment Manager 'shared libraries', and the 'bundles' introduced in MacOS X. Shared libraries are files of type 'shlb' containing a 'cfrg' resource which identifies the shared library. APLX looks for these first in the 'processors' folder of the APLX installation, and then in the 'bin' folder of the APLX installation (i.e. the folder where the APLX executable resides). You cannot use a full path for a shared library. Alternatively, if you use a MacOS X 'bundle', it must be referred to by a full path.
As a convenience, instead of the two-step share procedure using AP3 outlined above, you can directly load an auxiliary processor using any processor number N over 100. The name of the processor to load will be apN. For example, to load a processor called "ap124":
124 ⎕SVO 'CTL' Share a variable called CTL and load ap124
We recommend that processors loaded directly using this method should be placed in the 'processors' directory of the APLX installation, although you can also place them in the host-specific search path described above. The name of the file containing the Auxiliary Processor would be 'ap124' under AIX or Linux, or 'ap124.dll' under Windows. Under MacOS, it could be either a shared library called 'ap124' (of type 'shlb' with a 'cfrg' resource identifying the library as 'ap124'), or alternatively a MacOS X bundle of name 'ap124.bundle'.
To unload a bound processor and free up the memory it uses, simply erase the shared variable (or retract the share using ⎕SVR).
Note that all auxiliary processors are automatically unloaded when the user executes a )CLEAR or an )OFF.
APLX Auxiliary Processor Calling Conventions
APLX accesses the auxiliary processor through a shared variable; you can assign data to the processor and get back results using normal APL variable assignment and referencing syntax:
PROC←DATA
R←PROC
The auxiliary processor has only one entry point which is called from APLX. The entry point must be called processor, and must be an exported routine name. It is called in four different circumstances:
- As soon as the processor is loaded using
⎕SVO
- When data is assigned to it (shared variable specified)
- When data is read from it (shared variable referenced)
- Just before it is unloaded
The syntax of the main routine of the processor, expressed in C notation, is:
int processor (
int op, // Operation code AP_xxx
WS_Entry *data, // Pointer to APL data specified, or where
// to put result
APLINT wsfree, // Free workspace (in bytes) on AP referenced
void *scratch, // Pointer to 256-byte scratch area
// for use by the AP
void *wsbase, // Pointer to base of workspace
WS_Entry *last_var, // On reference, pointer to header of last var
// specified, or NULL if it was temp
APLINT tie, // An internal tie number uniquely
// identifying shared variable
ExportedProcs *exports // A structure containing addresses of
// routines which the AP can call
);
Note: APLINT is a signed 32-bit integer for 32-bit versions of APLX, and a signed 64-bit integer for APLX64.
The op parameter is the reason for the call, and is one of:
AP_LOADED
AP_SPECIFIED
AP_REFERENCED
AP_UNLOADED
The include file aplx.h contains the appropriate definitions.
Note for users upgrading from APL.68000 Level II: This function prototype has changed somewhat, as there are now some extra parameters. APs written using the SHAREC mechanism of APL Level II for MacOS will need to be modified and re-compiled. APs written for APL Level II for the RS/6000 can still be used unchanged, as the old mechanism is still supported under AIX.
Auxiliary Processor data
Be aware that you can have the same auxiliary processor bound to more than one shared variable at one time. It is important to understand that the same copy of the processor is bound in each case, and that the processor is not finally unloaded from memory until all of the shared variables which use it have been erased. If the processor is designed to be used in more than one simultaneous instance, the programmer must be careful to associate any static data retained by the processor with the tie number by which it is being accessed, or to use the 256-byte scratch area to hold data associated with the instance. You can, of course, allocate more memory for use by the AP, as long as you free it again when the AP is unloaded.
Return value
The explicit result of the routine is an APL error code. 0 means no error. Error codes are defined in the aplx.h include file.
If the result returned is ¯4), then the system assumes that the processor has already displayed the error message. The system will therefore display only the line in error and the caret.
Internal Structure of an Auxiliary Processor
The internal structure of the auxiliary processor can be anything which makes sense to the programmer. The processor can call other library routines (which are dynamically bound when the processor is loaded), allocate and deallocate memory, fork other processes, etc. The processor can also take over signal handling, but it should restore APL's signal handlers before returning.
Typically, the processor runs a routine when it is loaded to allocate any resources it needs. The read and write data routines do the real work, and the unload routine is used to deallocate resources just before the processor is unloaded from memory.
In C, the main routine might be a simple switch statement:
#include "aplx.h" /* Include the APLX header equates */
int processor (int op, WS_Entry *data, APLINT ws_free, void *scratch,
void *wsbase, WS_Entry *last_var,
APLINT tie, ExportedProcs *exports)
{
int result;
switch (op) {
case AP_LOADED:
result = load_routine (tie);
break;
case AP_UNLOADED:
result = unload_routine (tie);
break;
case AP_SPECIFIED:
result = specify_routine (data, tie);
break;
case AP_REFERENCED:
result = reference_routine (data, wsfree, last_var, tie);
break;
default:
return APL_SYSERR; /* Can't ever happen */
}
return result; /* Explicit result is APL error number */
}
The question of when to do the real work - on the specify routine or the reference routine - will depend on the application.
For example, a processor which calculates the mean and standard deviation of a set of numbers might do the computation once when the data is specified, and store the results in static data to be picked up on the reference routine. This is the simplest construction.
However, a routine which performs a transformation on a matrix has a choice: (a) store the matrix data on the specify routine and do the transformation when the processor is referenced, or (b) perform the transformation when the data is specified and store the result.
In the latter case, where the original matrix is stored in an APL variable, a pointer to it is passed to the reference routine in the argument last_var, making it possible to omit the data storage in (a) above. If the original data specified was not a variable but constant data, it no longer exists when the reference routine is called, and a NULL pointer is passed.
Format of an APLX workspace entry
When data is written to the processor the pointer to the workspace entry header is passed as one of the parameters to the called routine. Similarly, when data is read a pointer to the result area where the processor must build the result header is passed. We recommend that you use the WS_Entry structure defined on aplx.h to access the fields. (Note that all data is held in the natural byte-order for the machine in question. Thus, for little-endian processors like the x86, the actual bytes in memory within each 2- 4- or 8-byte field are held backwards). The structure of the header is as follows:
The first four bytes are used internally by APL and are not important to the person writing the processor, except that the field must be accounted for when returning a result. The next four bytes (or 8 bytes in APLX64) are a length field (we_length), which is the total length of the variable in bytes. (APL automatically rounds this up to be a multiple of 4 or 8). The next byte of the header is a one-byte field (we_type) describing the variable type:
DT_CHR 0x05 Binary data
DT_INT 0x07 Integer data
DT_FLT 0x09 Float data (IEEE 64-bit format)
DT_CHR 0x0B Character data
DT_OBJ 0x0E Object- or class-reference
DT_NST 0x0F Nested or mixed data
Equates for the data types DT_xxx and for the WS_Entry data structure are defined in aplx.h.
The next byte is an unused padding byte, followed by a two-byte field (we_rank) containing the rank of the data times 4, with zero indicating that the data is a scalar. Following the rank field are multiple four (or eight) byte rho entries. After the rho entries, the data begins.
Data Formats
Binary data is stored as individual bits in left justified 32-bit or 64-bit fields, depending on the version of APLX. In a 32-bit APL, the binary vector
1 0 1 1 0 1 1 1 1 0 1 would be stored as follows in a 32 bit block (shown in Hexadecimal):
00000000 00000014 05000004 0000000B B7A00000 (Reserved) (Length) (Type-Rank) (Rho) (Data)
Integer data is stored in 32-bit (or 64-bit) signed blocks. The 32-bit workspace entry for the integer vector
10 ¯2 would look like:
00000000 00000018 07000004 00000002 0000000A FFFFFFFE (Reserved) Length) (Type-Rank) (Rho) (Data)
Float data is stored in the 64 bit IEEE floating Point Standard:
Bit 63 - Sign of Mantissa, (1=negative)
Bits 62-52 - Exponent biased by decimal 1024
Bits 51- 0 - Mantissa, binary, normalized, with the MSB
not stored and assumed 1
The 32-bit workspace entry for the floating scalar 256.5 would be:
00000000 00000014 09000000 40700800 00000000 (Reserved) (Length) (Type-Rank) (Data)
Character data is stored as 1 character per byte, with possibly unused pad characters to keep the character count a multiple of 4. The 32-bit workspace entry for the character vector HIT would look like:
00000000 00000014 0B000004 00000003 68697400 (Reserved) (Length) (Type-Rank) (Rho) (Data)
Object- or class-references (data type DT_OBJ) are a special case. The data which follows is an integer representing the index into the table of objects which APLX keeps in the workspace, or 0 for a NULL object. Auxiliary processors cannot make any use of this data, so you should report an error if an object reference is passed to an AP. Equally, do not pass back back data of type DT_OBJ (other than NULL objects) to APLX from an AP.
Mixed or nested data is stored in a potentially recursive format. Following the header fields (Reserved, length, type-rank, rho) are the appropriate number of 4 (or 8) byte pointers (giving an offset from the start of the workspace entry) to individual sub-arrays, which are held within the overall data portion.
VAR←2⍴10 'ABCD' 00000000 0000003C 0F000004 00000002 00000018 00000028 (Reserved) (Length) (Type-rank) (Rho) (Pointers) 00000000 00000010 07000000 0000000A (Reserved) (Length) (Type-rank) (Data) 00000000 00000014 0B000004 00000004 61626364 (Reserved) (Length) (Type-rank) (Rho) (Data)
Programmers should note that if data is inherently capable of being represented as simple data it must be. Thus a simple shape 2 2 matrix must be held as a simple array.
Prototype
Empty nested arrays must have a prototype entry appended. The overall length must account for the length of the prototype. The prototype entry follows the main entry and must contain only 0s or space characters.
VAR←⊂2 2⍴⍳4 X←0↑VAR 00000000 00000028 0F000004 00000000 00000000 (Reserved) (Length) (Type-rank) (Rho) (Reserved) 00000018 05000008 00000002 00000002 00000000 (Length) (Type-rank) (Rho) (Rho) (Data)
(All of the above examples are shown in big-endian order, for 32-bit systems.)
Using APLX Exported Routines
A number of useful routines in the APLX interpreter are available to be called by the auxiliary processor. A structure containing pointers to these routines is passed as a parameter to the AP, in the
Exported routines include :
char fontin (char font_char); /* Translate character from font representation to ⎕AV*/
char fontout (char apl_char); /* Translate character from ⎕AV to font representation*/
int check_event_attention (void); /* Call APL attention check. Returns non-zero
if attention hit */
Note that if the processor retains control for a long period (more than a second) without returning to APL, it should call
An Example Auxiliary Processor
The following (supplied as
//---------------------------------------------------------------------------
// Sample AP for APLX
//---------------------------------------------------------------------------
// *** MAKE SURE YOU DEFINE LITTLE_ENDIAN IF COMPILING FOR WINDOWS OR OTHER
// *** x86 PLATFORM. THIS MUST BE DONE BEFORE INCLUDING aplx.h
//
// *** MAKE SURE YOU DEFINE APL64 IF COMPILING FOR APLX64
// *** ON A 64-BIT PLATFORM. THIS MUST BE DONE BEFORE INCLUDING aplx.h
//
// For AIX, compile with:
//
// cc -o ap103 -G -Iusr/lpp/aplx/processors -qcpluscmt sample_ap.c
//
// (The -G option tells the linker to create a shared library.)
//
// This should create an output file 'ap103'.
//
// For Linux, compile with:
//
// gcc -o ap103 -shared -Iusr/local/aplx/processors sample_ap.c
//
// This should create an output file 'ap103'.
//
// For Windows, you need to add a DLL entry point:
//
// int WINAPI DllEntryPoint(HINSTANCE hinst, unsigned long reason, void*)
// {
// return 1;
// }
//
// You also need to export the 'processor' function by declaring it
// as type: __declspec(dllexport)
//
// Compile the AP as a DLL, and ensure it is called ap103.dll
//
// For MacOS, build as a shared library or bundle, for example using the
// Metrowerks Code Warrior or Apple Project Builder.
//
// Once you have built the AP, place it in the 'processors' directory of
// the APLX installation, and test it in APLX with:
//
// 103 ⎕SVO 'DATA'
// 2 <-- If it doesn't return 2, AP not found
//
// DATA ← ⎕A Give it a character vector
//
// ⎕AF DATA See what it returns
// 27
//
//
// Leave this undefined for MacOS (PowerPC) and AIX.
// Define it for Windows, x86 Linux and Mac Intel.
// #define LITTLE_ENDIAN 1
// Leave this undefined for 32-bit versions of APLX.
// Define it for APLX64
// #define APL64 1
#include "aplx.h"
int processor (
int op, /* Operation code AP_xxx */
WS_Entry *data, /* Pointer to APL data specified,
or where to put result */
APLINT wsfree, /* Free WS (in bytes) on AP referenced */
void *scratch, /* Pointer to 256-byte scratch area for
use by the AP */
void *wsbase, /* Pointer to base of workspace */
WS_Entry *last_var, /* On reference, pointer to header of last
var specified, or NULL if it was temp. */
APLINT tie, /* An internal tie number uniquely
identifying shared var */
ExportedProcs *exports) /* Structure containting exported routines
you can call */
{
int result; /* Error code to return to APL */
char xor = '\0'; /* The XOR result */
APLINT i; /* Counter */
switch (op) {
case AP_LOADED:
result = APL_GOOD;
break;
case AP_UNLOADED:
result = APL_GOOD; /* Nothing to do */
break;
case AP_SPECIFIED:
if (data->we_type_rank == CHR_VEC) {
for (i = 0; i < data->we_vec_length; i++)
xor ^= data->we_vec_cdata[i];
((char *)scratch)[0] = xor; /* Save result in scratch area */
result = APL_GOOD;
} else {
result = APL_DOMERR;
}
break;
case AP_REFERENCED:
if (wsfree < 256)
return APL_WSFULL;
data->we_reserved = 0; /* Plug 0 in reserved field */
/* total length of result: */
data->we_len = offsetof(we_vec_idata, WS_Entry);
data->we_type_rank = CHR_SCL; /* type and rank of result */
data->we_scl_cdata = ((char *)scratch)[0];/* return saved result */
result = APL_GOOD;
break;
}
return result;
}