💾 Archived View for mirrors.apple2.org.za › active › ftp.apple.asimov.net › utility › nulib › nulib2… captured on 2023-01-29 at 06:54:25.
View Raw
More Information
-=-=-=-=-=-=-
[ Home <index.htm> ] [ NuLib Downloads <downloads/index.htm> ]
[ NuLib Library <library/index.htm> ] [ NuLib2 Manual ] [ NufxLib API
<nufxlibapi.htm> ] [ Bugs & Features <bugs.htm> ]
------------------------------------------------------------------------
NuLib2 v2.1.1 Manual - By Andy McFadden - Last revised
2006/02/18
Table of Contents
* Introduction <#introduction>
* Command Overview <#overview>
* Listing Archive Contents (-v, -t) <#list>
* Creating Archives and Adding Files (-a) <#createadd>
* Extracting Files (-x, -p) <#extract>
* Verifying Archive Integrity (-i) <#verify>
* Deleting Files (-d) <#delete>
* Disk Images <#diskimages>
* Binary II Archives <#binary2>
* Miscellaneous <#miscellaneous>
* Acknowledgments <#acknowledgments>
Introduction
NuLib2 is a command-line archive utility, along the lines of "PKZIP".
It allows you to perform many common operations on NuFX archives, such
as those created by the Apple II "ShrinkIt" utility, as well as Binary
II archives. Files with extensions "SHK", "SDK", "BXY", "BSE", "SEA",
"BNY", and "BQY" are handled.
You can add, delete, extract, test, and list files in a NuFX archive.
Compressed disk images can be extracted to files, and vice-versa, making
it a handy utility to have when using an Apple II emulator.
NuLib2 is the successor to NuLib, which did many of the same things.
NuLib2 is not meant to set a new standard or fight for supremacy on the
PC desktop; rather, it is intended to help people working with Apple IIs
and Apple II emulators. All compression algorithms specified in the
NuFX specification are fully supported.
NuLib2 has a number of features not found in NuLib:
* Supports filetype preservation (on systems with "long" filenames).
* Supports resource forks.
* Archive integrity test actually does something useful.
* ShrinkIt-style table of contents listing formats filenames correctly.
* Support for adding and viewing comments.
* Allows streaming input (i.e. read from stdin) for most "read-only"
operations.
* Can operate on large archives while using very little memory (with
streaming mode).
* Transparently handles .BXY, .SEA, and .BSE wrappers.
* Automatically recognizes and handles Binary II archives.
* EOL conversions (e.g. CRLF to CR) are more automatic.
* Deals with some Y2K issues.
* Comes with an auto-configuration script for UNIX-like platforms.
* Creates "version 3" records, and uses LZW/2. Most of
GS/ShrinkIt's quirks are emulated, so the archives created by
NuLib2 are nearly identical to those created by GSHK.
* Optionally supports gzip-style "deflate" compression and bzip2 BWT
(as an extension to the NuFX standard).
* Source code is cleaner and command-line options are simpler.
All features of the original NuLib are supported, except for a couple of
really obscure ones. NuLib2 is built on top of NufxLib, a NuFX archive
manipulation library.
Command Overview
Commands are specified like this:
| nulib2 -command[modifiers] archive [filename-list]|
There are seven commands: add files, list files (two variations),
extract files (two variations), delete files, and verify archive
integrity. There are ten modifiers, discussed in later sections.
If you run nulib2 without any arguments, you will be presented with an
identification banner and a command summary. The identification banner
describes the current version of the NuLib2 application, and the version
of NufxLib upon which it was built. The latest version of NuLib2 can
always be found at http://www.nulib.com/. If you want a more detailed
command summary, use:
|nulib2 -h|
If you wish to specify modifiers, you may lump them together with the
command, like this:
|nulib2 -xel archive.shk|
or specify each independently, like this:
|nulib2 -x -e -l archive.shk|
If you want to give your archive a name that starts with a hyphen, you
will have to specify it as a full or partial path, e.g.
"|./-archive.shk|". Otherwise, NuLib2 will think you are trying to
specify more modifiers. One exception to this is that you can specify
stdin as the archive for list and extract operations:
|nulib2 -v - < archive.shk|
Some commands require a list of filenames. These must be listed after
the archive name, e.g.:
|nulib2 -x archive.shk foo ack:splat bar|
The names given will be compared with records in a case-insensitive
fashion, so asking NuLib2 to extract "foo" or "FOO" would match on
"foo", "FOO", and "fOo".
The commands are explained next.
Listing Archive Contents (-v, -t)
The contents of NuFX archives are listed in a format similar to what
ProDOS 8 ShrinkIt 2.x displays. If you use the command:
|nulib2 v archive.shk|
you will see output similar to this:
archive.shk Created:22-Aug-90 15:33 Mod:22-Aug-90 15:33 Recs: 4
Name Type Auxtyp Archived Fmat Size Un-Length
-----------------------------------------------------------------------------
BUG.REPORTS TXT $0003 22-Aug-90 15:33 lz2 32% 3089
FINDER.DATA FND $0000 22-Aug-90 15:33 unc 100% 458
+HELLO BAS $0801 22-Aug-90 15:33 lz2 86% 605
GIF.SYS16 S16+ $0000 22-Aug-90 15:33 lz2 31% 39165
-----------------------------------------------------------------------------
Uncomp: 4152 Comp: 1988 %of orig: 47%
For each file, the display includes the filename, ProDOS file type,
ProDOS "aux" type, the date and time when the file was added to the
archive, the compression format, a compression ratio percentage, and the
length of the file before it was compressed. The first line has some
information about the archive, and the last line has a summary of file
sizes and compression performance.
A "+" in the leftmost column, in front of the filename, indicates that
the file is "locked". NuLib2 considers a file to be locked when it has
the ProDOS write, rename, and delete permissions disabled, but still has
read permission enabled. All other files are considered to be unlocked.
A "+" is shown next to the file type if the file is "extended", meaning
it has a resource fork. A "-" next to the file type indicates that it's
an empty file stored improperly due to a bug in GSHK.
The listing for a disk image is similar:
prodisk.shk Created:21-Apr-90 15:11 Mod:21-Apr-90 15:11 Recs: 1
Name Type Auxtyp Archived Fmat Size Un-Length
-----------------------------------------------------------------------------
TEST Disk 140k 21-Apr-90 15:11 lz1 37% 143360
-----------------------------------------------------------------------------
Uncomp: 143360 Comp: 53051 %of orig: 37%
The file type is displayed as "Disk", the "aux" type is the size of the
disk image in KBytes, and -- if the image is for a ProDOS disk -- the
filename is the disk volume name.
The "Fmat" column will be one of "lz1", "lz2", or "unc", indicating
ShrinkIt LZW/1, LZW/2, or uncompressed. Files added to archives by
NuLib2 always use LZW/2 or, if compression fails, are stored
uncompressed. LZW/1 is used by P8 ShrinkIt and the original NuLib.
The filename field will show the full name, including all subdirectories
leading up to the file. If there isn't enough room to display the
entire filename, it will be truncated on the left, replaced with two
dots. The following example has five files from the "gno:user:man:man2"
directory, one of which was too long to fit in the display:
Name Type Auxtyp Archived Fmat Size Un-Length
-----------------------------------------------------------------------------
gno:usr:man:man2:sigblock.2 GWP $8010 01-Dec-97 00:07 lz2 58% 2667
gno:usr:man:man2:signal.2 GWP $8010 01-Dec-97 00:07 lz2 56% 7036
gno:usr:man:man2:sigpause.2 GWP $8010 01-Dec-97 00:07 lz2 61% 2358
..usr:man:man2:sigsetmask.2 GWP $8010 01-Dec-97 00:07 lz2 56% 3046
gno:usr:man:man2:wait.2 GWP $8010 01-Dec-97 00:07 lz2 51% 6577
-----------------------------------------------------------------------------
You can also list the contents of an archive like this:
|nulib2 t archive.shk|
The 't' command generates a simple list of filenames:
BUG.REPORTS
FINDER.DATA
HELLO
GIF.SYS16
The filenames are never truncated or embellished, which makes this
command useful when you're searching for a specific file.
Creating Archives and Adding Files (-a)
You can add files to a new or existing archive with the "-a" command.
If the archive you specify does not exist, a new one will be created.
All files will be added to the end of the archive. If the name of the
file being added matches the name of a file already present in the
archive, you will be allowed to replace the existing file or skip adding
the new file:
|Replace BASIC.System? [y]es, [n]o, [A]ll, [N]one:|
If you respond with "y", the existing file will be deleted when the
archive is updated. If you say "n", the new file will be skipped.
Entering "A" or "N" will cause NuLib2 to automatically enter "y" or "n"
respectively on any future conflicts. (You may also hit 'q' here to
abort the operation and quit.)
Files in subdirectories will be added with whatever path separator is
appropriate for the current system. On a UNIX-like system that would be
'/', while under Win32 it's '\'. GS/OS follows the Mac OS convention
and uses ':'.
There are a number of modifiers that can be used with this command.
-u
Update files. If a matching file is found in the archive, it will
only be replaced if the file on disk is newer. Files that don't
already exist in the archive will be added. The "Replace...?"
dialog will not appear.
-f
Freshen files. If a matching file is found in the archive, it will
only be replaced if the file on disk is newer. Files that don't
already exist in the archive will *not* be added. The "Replace...?"
dialog will not appear.
-r
Recursively descend into subdirectories. The standard behavior for
NuLib2 is to ignore directories. When this flag is set, it will add
all of the files in a specified subdirectory. (Note: empty
directories will not be added.)
-j
Junk paths. If you add a file called "foo/bar/myfile", when this
flag is set it will be stored simply as "myfile".
-0
Don't compress. Files will be stored without compression. (Note
that's "dash zero", not the letter 'O'.)
-z
Use "deflate" compression, equivalent to "gzip -9". Both NufxLib
and NuLib2 must be built with zlib <http://www.zlib.org/> support,
or the flag will be disabled. Note that "deflated" files can only
be opened by applications based on NufxLib, so don't use this for
files that will be opened on an Apple II. If you specify "-zz",
compression equivalent to "bzip2 -8" will be used instead. (Support
for bzip2 compression is disabled by default, so it may not be
available.)
-c
Add one-line comments. NuLib2 will prompt you for a comment on
every new file. Hitting return ends the comment. (Hint: on most
UNIX systems, you can use Ctrl-V Ctrl-M to insert a carriage
return. NuLib2 automatically converts carriage returns in comments
to whatever is appropriate for the current system, so you can
"sneak" multi-line comments in this way.) Maximum length is 200
characters.
-k
Store files as disk images. All files will be processed as
ProDOS-ordered disk images. See Disk Images <#diskimages> for more
information.
-e
Preserve ProDOS file types. Files with preservation information
will be added with their file type and aux type intact, and resource
forks and disk images will be processed correctly. For this to
work, the files must have been extracted with the "-e" flag set. If
you specify "-ee", NuLib2 will attempt to guess the file types of
common files by their extensions (e.g. "file.txt" would be stored as
type $04 (TXT)). See the ProDOS Attribute Preservation
<library/nulib2-preserve.htm> document for more information.
While adding files, NuLib2 displays the name of the file as it will
appear in the archive, along with a completion percentage:
DONE compressing BASIC.System
DONE storing BOOT.GSOS
DONE compressing BOOTU3
DONE storing COMM/FINDER.DATA
DONE compressing COMM/ProTERM/PT3.SYSTEM
DONE compressing COMM/ProTERM/PT3
DONE compressing COMM/ProTERM/PT3.GLOBAL
DONE storing COMM/ProTERM/PT3.WELCOME
15% compressing COMM/ProTERM/PT3.CODE0
Files stored without compression will say "stored", while files
compressed with LZW/2 will say "compressing". NuLib2 tries to mimic
GS/ShrinkIt as closely as possible, so files shorter than 512 bytes are
never compressed, and files that don't get smaller are stored uncompressed.
Extracting Files (-x, -p)
Files can be extracted with the "-x" and "-p" commands. You may specify
a list of files to extract, or specify none and extract all files.
A simple example that extracts all files from an archive:
|nulib2 x archive.shk|
If you wanted to extract a file called "fubar" and a file called
"ack:splat", you would use:
|nulib2 x archive.shk fubar ack:splat|
The directory hierarchy is always preserved unless "-j" is set. In the
above example, the files would be extracted as "fubar" in the current
directory and "splat" in a subdirectory called "ack" (assuming that the
file was archived on a system where ':' separates directory names).
Using the "-r" flag (described below), you could extract all of the
files in the "ack" subdirectory, like this:
|nulib2 xr archive.shk ack:|
The "-r" flag does a prefix string match, meaning it just compares the
first part of the name in the archive against the name you specify. So
if you had entered:
|nulib2 xr archive.shk ack|
above, it would have extracted "acknowledge" and "acksent" as well as
"ack:foo" and "ack:bar". Specifying the filesystem separation character
(usually '/' or ':') allows you to grab just the directory you want.
If you try to extract a file with the same name as an existing file, you
will be prompted for instructions:
|Replace BASIC.System? [y]es, [n]o, [A]ll, [N]one, [r]ename:|
You can choose (from left to right) to overwrite the existing file, skip
the extraction of this file, overwrite all existing files, never
overwrite an existing file, or rename the current file. If you elect to
rename, you will be prompted for a new name for the file. (You may also
hit 'q' here to quit immediately.)
The modifiers usable with the "-x" command are:
-u
Update files. If a matching file is found on disk, it will only be
replaced if the file in the archive is newer. Files that don't
already exist on disk will be extracted. The "Replace...?" dialog
will not appear.
-f
Freshen files. If a matching file is found on disk, it will only be
replaced if the file in the archive is newer. Files that don't
already exist on disk will *not* be extracted. The "Replace...?"
dialog will not appear.
-r
Recursively describe the set of files to extract. This allows you
to extract an entire subdirectory tree from the archive. (Perhaps
someday NuLib2 will support extraction by regular expression, and
this modifier can go away.)
-j
Junk paths. The pathname is stripped off of the file. If you
extract "foo" and "ack:splat", you will end up with a file called
"foo" and a file called "splat" in the current directory.
-l
Convert EOL. When set, NuLib2 tries to identify which files in the
archive are text files and which have binary data. It then converts
the text file end-of-line markers to whatever is appropriate on the
current system. If you specify "-ll", NuLib2 will convert *all*
files except disk images. High-ASCII files, such as DOS text files
and Merlin 8 source files, are automatically stripped.
-c
Display comments while extracting. If a comment was stored with the
record, it will be displayed on the screen. End-of-line markers
(e.g. CR, LF, or CRLF) used in the comment are automatically converted.
-s
Stomp existing files. When set, NuLib2 will overwrite existing
files without prompting you for confirmation.
-e
Preserve ProDOS file types. Extracted files will have the ProDOS
file type and aux type appended to the name (e.g.
"foo.txt#040000"). Resource forks and disk images will be labeled.
If you specify "-ee", NuLib2 will append the file's extension to the
end of the file, so that systems like Win32 that rely on filename
extensions can recognize the file type (e.g. "foo.txt#040000.txt").
See the ProDOS Attribute Preservation <library/nulib2-preserve.htm>
document for more information.
While extracting files, NuLib2 displays progress information:
DONE expanding BASIC.System
DONE extracting BOOT.GSOS
DONE expanding BOOTU3
DONE extracting COMM/FINDER.DATA
DONE expanding COMM/ProTERM/PT3.SYSTEM
DONE expanding COMM/ProTERM/PT3
DONE expanding COMM/ProTERM/PT3.GLOBAL
DONE extracting COMM/ProTERM/PT3.WELCOME
90% expanding COMM/ProTERM/PT3.CODE0
The filenames shown are the names as they are being written to disk. For
example, if you extracted files with "-e" (preserve types) and "-l"
(convert end-of-line character) set:
|nulib2 -xle archive.shk|
You would see something like:
DONE expanding BASIC.System#ff2000
DONE extracting BOOT.GSOS#fc0801
DONE expanding BOOTU3#060800
DONE extracting COMM/FINDER.DATA#c90000
DONE expanding COMM/ProTERM/PT3.SYSTEM#ff2000
DONE expanding COMM/ProTERM/PT3#ff2000
DONE expanding + COMM/ProTERM/PT3.GLOBAL#040000
DONE extracting+ COMM/ProTERM/PT3.WELCOME#040000
90% expanding COMM/ProTERM/PT3.CODE0#060000
The "+" sign indicates that an EOL conversion was performed on the file.
The "-p" command extracts the files to a pipe. The contents of the
extracted files are written to stdout. The normal status messages are
suppressed.
Only the "-r" and "-l" modifiers can be used with "-p".
This command is useful for examining the contents of individual files.
For example, the command:
|nulib2 -pl archive.shk document.txt | more|
will dump the contents of "document.txt", with text automatically
converted, and pipe the results into "more". If you specify more than
one file, they will be sent to the output one after the other.
Verifying Archive Integrity (-i)
If you want to verify that an archive is undamaged, use this command.
NuLib2 does the same processing that it does when extracting files from
the archive, but no output is produced. Every CRC (Cyclic Redundancy
Check - a checksum computed when the data was added to the archive) is
tested.
The progress information shown when testing an archive looks like this:
DONE verifying BASIC.System
DONE verifying BOOT.GSOS
DONE verifying BOOTU3
DONE verifying COMM/FINDER.DATA
DONE verifying COMM/ProTERM/PT3.SYSTEM
DONE verifying COMM/ProTERM/PT3
DONE verifying COMM/ProTERM/PT3.GLOBAL
DONE verifying COMM/ProTERM/PT3.WELCOME
90% verifying COMM/ProTERM/PT3.CODE0
If a problem is found, you will see a message like this one:
Found a bad CRC in BASIC.System
Archive may be damaged, continue anyway? [y]es, [n]o:
If you choose 'y', NuLib2 will pick up where it left off, and continue
processing the archive. If you choose 'n', NuLib2 stops immediately.
If you don't specify a list of files:
|nulib2 -i archive.shk|
then NuLib2 will test every file in the archive. If you use a list:
|nulib2 -i archive.shk foo ack:splat|
it will only test the files you told it to.
You may use the "-r" modifier when specifying files to test, in the same
fashion as when extracting files.
Deleting Files (-d)
This command deletes entire records from an archive. A list of
filenames must be specified. If you manage to delete all of the files,
the archive itself will be removed.
An example of this command:
|nulib2 -d archive.shk foo ack:splat|
A progress report will be displayed during processing, e.g.
Deleting foo
Deleting bar
Deleting ack/splat
The "-r" modifier may be used to select entire subdirectories. See the
notes in the section on extraction <#extract>, above.
Disk Images
NuLib2 doesn't try to write floppy disk images onto a floppy the way
ShrinkIt does. Instead, it extracts disk images as ProDOS-ordered
files, suitable for use with your favorite Apple II emulator.
When adding disk images, NuLib2 assumes that the file is in ProDOS block
order. If you want the archive to work correctly with ShrinkIt on an
Apple II, don't add DOS-ordered disk image files.
The NufxLib library comes with a sample program called "imgconv" that
can convert between 2IMG (".2mg") images and NuFX (".shk") archives. It
is able to convert DOS-ordered .2mg files to ProDOS order before adding
them to the NuFX archive.
NuLib2 will only accept files that are a multiple of 512 bytes. If you
insist on trying to add odd-sized files as disk images, NuLib2 will
print a warning and add it as an ordinary file.
Binary II Archives
NuLib2 supports Binary II (.BNY, .BQY) archives in a more or less
transparent fashion. For all operations except adding and deleting
files you can specify a .BNY file in place of a .SHK and get more or
less the same results.
A few modifier flags aren't currently supported for Binary II archives.
You can't specify EOL conversion with "-l", freshen or update with "-f"
or "-u", or extract comments (there are no such things) with "-c". You
can, however, specify filenames using "-r" for partial matches, use "-e"
and "-ee" to extend filenames, and use "-j" to truncate paths.
The default behavior is to refuse to overwrite existing files. NuLib2
currently just stops if it encounters a file that already exists. You
can use the "-s" flag to tell NuLib2 to overwrite any existing files.
It will not, however, overwrite directories with files or files with
directories.
Files that were compressed with "Squeeze" compression (via BLU v2.27 or
SQ3) will be automatically un-squeezed. If the filenames end in ".QQ",
the extension will be stripped. Programs like ShrinkIt and BLU use the
filename embedded within the SQ format as the output name when
extracting, but NuLib2 ignores that because it tends to leave you with a
filename you weren't expecting.
In some cases you will need to explicitly tell NuLib2 that a file is a
Binary II archive with the "-b" flag. If you want to strip the Binary
II header off of a .BXY file, you will need to use "|nulib2 -xb
file.BXY|". Otherwise, NuLib2 defaults to extracting from the NuFX
archive embedded in the BXY file. You will also need to use the "-b"
modifier for a streaming archive (e.g. "|nulib2 -xb - < file.BQY|"),
because the stream can't be rewound after the test for NuFX fails.
Miscellaneous
Files that were somehow archived without a filename will be referred to
as "UNKNOWN". Rumor has it some older versions of ShrinkIt omitted the
file name for DOS 3.3 disk images.
On Win32 and UNIX-like systems, the ProDOS access permissions are not
preserved precisely. ProDOS has read, write, rename, and delete
permission, as well as "backup needed" and "invisible" flags. If NuLib2
believes the file is locked, it will disable write permission on the
file. When adding files, "locked" files will have read and "backup
needed" enabled and the other flags disabled, while "unlocked" flags
will have all the flags except "invisible" enabled. In other words, the
basic concepts of "locked" and "unlocked" are preserved, but the full
set of flags is not.
The NuFX specification forbids filenames that start with the filename
separator character, i.e. you can't put "/foo/bar" in an archive. If
you specify a full pathname, the leading '/' will be dropped.
On Win32 and UNIX-like systems, a leading "./" in the filename is
redundant and will be stripped. If ".." is used as a path component,
the pathname will be reduced to just the filename.
When making changes to an archive, a new archive is constructed in a
temporary file ("nulibtmpXXXXX") in the current directory. When
everything completes successfully, the temp file is renamed over the
original archive. The exception to this is that newly-created archives
are written in place. If NuLib2 crashes or is killed with signal while
modifying an archive, the temp file may be left lying around.
Filenames with invalid characters will be converted to something
palatable for the current system (e.g. "/" is set to "_" on UNIX-like
systems). If file type preservation is enabled, the character will be
preserved exactly (e.g. '/' becomes "%2f").
NuFX archives store three dates with every file: creation, modification,
and when it was archived. On systems that don't have creation dates,
the modification date will be substituted.
There are certain filenames you can't use on a Windows "FAT" filesystem,
such as "AUX" and "PRN". Neither Win98 nor Linux's vfat driver will
allow it. Standard utilities like WinZip fail with a mysterious error
message. As a workaround, the Win32 version of NuLib2 will consistently
prefix all MS-DOS device entries with '_', so "AUX" and "aux.foo.txt"
will be extracted as "_AUX" and "_aux.foo.txt". If filetype
preservation is enabled, the name used will be "%00AUX" (the %00 is
removed when the file is added with -e, thus preserving the original
name). This handling does not apply to versions built for other
systems, so attempting to extract a file named "AUX" onto a FAT
filesystem under Linux will cause NuLib2 to fail.
No attempt is made to extract and preserve comments, even in "preserve"
mode. This is probably a bug.
Archive files that start with junk -- such as a vestigal MacBinary
header or HTTP headers -- appear occasionally on FTP sites. NuLib2 will
search through the first 1024 bytes of the file to find the actual
archive start.
Silly benchmark of the day: creating a 14MB archive containing the
contents of my hard drive took about 40 minutes on an accelerated IIgs.
NuLib2 accomplished the same feat in about six seconds on a 500MHz
Pentium-III running Linux.
Acknowledgments
NuLib2 would not have been possible without all the lessons learned from
NuLib. Andy Nicholas, Kent Dickey, Frank Petroski, Robert B. Hess,
Bruce Kahn, and Devin Reade contributed code to NuLib's development.
My original plan for preserving ProDOS file types was, in retrospect,
mighty screwed up. My thanks to Bill North for setting me straight.
Eric Shepherd spent a bunch of time messing around with early versions
of NuLib2 on BeOS, and helped me get all the configuration stuff in order.
Devin Reade built it on several different platforms, and made a
repository for binary distributions.
------------------------------------------------------------------------
This document is Copyright � 2000-2006 by Andy McFadden
<http://www.fadden.com/>. All Rights Reserved.
The latest version can be found on the NuLib web site at
http://www.nulib.com/.