Note that this is an open case and the e-mail discussion is being distributed to addresses outside of Sun... I am submitting the fast-track case for Roland Mainz (OpenSolaris contributor) and April Chin (Sun engineer). This case seeks a patch binding. (The current plan is to integrate into ONNV, but patches to a Solaris 10 Update and Solaris 9 are possible if there is sufficient demand.) This case times out Wednesday, September 27, 2006. Note that if there is any significant discussion on this case, I will ask for a short discussion on this case after the ARC business discussion on September 27th. This will allow Roland and the Open Solaris community to participate in the discussion. Sincerely, Don 1. Introduction 1.1 Project/Component Working Name Korn Shell 93 Integration 1.2 Name of Document Author/Supplier: April Chin (april.chin@sun.com) 1.3 Date of this Document: 3 October, 2006 1.4 Name of Major Document Customer(s)/Consumer(s) 1.4.1. The community this proposal comes from ksh93-integration-discussion@opensolaris.org 1.4.2. The ARC(s) you expect to review your project: PSARC 1.5. Email Aliases: 1.5.1. Responsible Manager: tim.sparlin@sun.com 1.5.2. Responsible Engineers: roland.mainz@nrubsig.org april.chin@sun.com 1.5.3. Marketing Manager: jeff.mcmeekin@sun.com 1.5.4. Interest List: ksh93-integration-discussion@opensolaris.org 2. Project Summary 2.1 Project Description This project seeks approval to introduce the latest version of the open source ksh93, maintained by korn shell creator David Korn and others at AT&T, into the Solaris system as a new set of binaries. Supporting AT&T libraries will also be introduced as private shared libraries. 2.2 Risks and Assumptions: This project is based on an external open source product and may have dependencies on the development of or the inclusion of required fixes by the korn shell owners at AT&T. It is the goal of the project team to keep ksh93 in the Solaris operating system updated with AT&T's latest version of ksh93 and its supporting libraries. Thus it will be important to keep Solaris ksh93 source in sync, including contributing Solaris-specific changes back to the AT&T source. There are cases of Solaris-specific functionality, covered under the CDDL (Common Development and Distribution License), which should be merged upstream to AT&T source for Solaris-specific builds. AT&T allows contributions under the condition that only a copyright may be added to AT&T source. At this time it is unknown if a blanket licensing agreement will be reached which allows contribution of such source to AT&T without requiring the addition of the CDDL. Although this project will introduce ksh93 as binaries separate from the existing /usr/bin/ksh and /usr/xpg4/bin/sh (the standard shell), which are both based on ksh88, the eventual goal in one or more future projects will be to replace both /usr/bin/ksh and the standard-compliant /usr/xpg4/bin/sh with ksh93. Thus this project needs to be aware of potential compatibility problems. See COMPAT file for a list of known incompatibilities between ksh93 and Solaris ksh. This project will also introduce built-in commands in ksh93, which have a superset of the functionality of the corresponding Solaris /usr/bin commands (see Section 4.3 Description). These ksh93 built-in commands may lead to the future replacement of the source of some existing /usr/bin commands with the corresponding AT&T code used by the ksh93 built-ins. Some functionality available in the current Solaris /usr/bin/ksh or /usr/xpg4/bin/sh is either not fully verified or not fully implemented in ksh93, including standards conformance. Although the intention is for ksh93 to be POSIX compliant, it does not yet meet UNIX98 or UNIX03 standards conformance, that is, ksh93 does not pass the latest OpenGroup Shell and Utilities Verification Suite, VSC version 5.2.8. This issue will be revisited before ksh93 replaces /usr/xpg4/bin/sh. 3. Business Summary 3.1 Problem Area Since at least 1998 (CR 4113420), and especially after AT&T's ksh93 was open sourced, circa 2000, users have been requesting that Solaris systems include the latest version of the Korn shell, ksh93, as a replacement for or alternative to the existing /usr/bin/ksh and /usr/xpg4/bin/sh, which are both derived from ksh88i. ksh93s- is under the Common Public License which allows Sun to freely redistribute a derived verison of ksh93, provided Sun includes the license terms and any changes we make allow us to grant the copyright licenses in the CPL. The derived ksh88 we currently ship in /usr/bin/ksh and /usr/xpg4/bin/sh is encumbered by copyrights which do not allow us to ship the source through the Open Source Solaris product. 3.2 Market Requester In addition to Open Solaris users, there have been multiple Change Requests (CRs): 4113420 *ksh* request for ksh93 integration many CRs have been closed as duplicates of this CR. 4113420 has many Customer Call records, including those from Nokia, Lockheed Martin, Lexis Nexis, Bell South, and AT&T. 6332421 Solaris Kornshell outdated involves a customer trying to port their application from SUSE Linux to Solaris 10 and running into problems with existing ksh93 scripts. 3.3 Justification ksh88 is out dated and considered obsolete by many. The current version of the Solaris ksh is not even considered ksh88 compatible, since its behavior has diverged significantly from the original AT&T ksh88. ksh88 source is closed on the Open Solaris source tree, but ksh93 will be open source, allowing community members to contribute fixes, resulting in a better quality Solaris korn shell. Users with existing ksh93 scripts on other platforms are reluctant to migrate to the Solaris OS without ksh93 functionality (see CR 6332421, above). Users on Open Solaris discussion groups and other media, such as news:comp.unix.solaris, have requested that Sun replace /usr/bin/ksh with ksh93, to allow default login shells to take advantage of the many features of ksh93 and to allow interoperability with other operating systems (see 3.4). In addition to many new features (see 4.2), many bugs in the existing /usr/bin/ksh are fixed in ksh93, including: 4500743 *ksh* 'test -w' should exit with zero exit status 4703213 *ksh* ksh can't handle [[ "$s" ]] as manpage claims 4771050 *ksh* test -z "=" should not produce an error 4869545 *ksh* ksh should print error message for "trap - YABBA_DABBA_DOO" 6359008 *ksh* dumps core running command builtin with large argument list 6417347 *ksh* test -nt/-ot doesn't compare all of the timestamp 6435815 *ksh* sometimes chokes when piping output to a shell function 6442036 *ksh* ksh: "export -p" doesn't work as documented Presumably, a majority of the remaining currently-open ksh RFEs and defects would be resolved by the use of ksh93 in place of Solaris /usr/bin/ksh. 3.4 Competitive Analysis Sun is one of the only major vendors which does not include ksh93. /usr/dt/bin/dtksh on the Solaris OS is not a viable login shell and is based on ksh93d alpha, a very early, out-of-date version of ksh93. ksh93 is included with IBM AIX, HP-UX, Debian, Redhat/Fedora, MacOSX, Darwin, and SUSE Linux distributions. The absence of ksh93 on the Solaris OS has been a major complaint by users [5]. HP-UX 11i Version 2, released September 2004 still keeps /usr/bin/ksh as ksh88 and ksh93 as the binary dtksh. On Debian, ksh93 is available as /usr/bin/ksh93, and does not include the AT&T libraries libast and libshell [6]. IBM AIX 5L (released in 2004) introduced ksh93 as /usr/bin/ksh93 /usr/bin/ksh is still an enhanced POSIX-compliant version of ksh88. [7] 4. Technical Description 4.1. Bug/RFE Number 6437624 RFE: Add ksh93 (as /usr/bin/ksh93) and libshell.so to OS/Net 4.2 Features ksh93 has many enhancements over the Solaris ksh utility, which is based upon ksh88i, including: - key binding (trap on KEYBD, to intercept all interactive user input) - floating point arithmetic - associative arrays - complete ANSI-C printf formatting - name reference variables (variables referring to other variables) - new parameter expansion operators - dynamic loading of built-in commands, including the ability to add user-defined built-in commands - active variables, allowing users to trap on variable assignments and references by associating intercept functions - compound variables - increased performance, due to the addition of many built-in commands for AT&T versions of common UNIX commands - read with timeouts - command and variable completion, in addition to filename completion - Various pattern matching facilities, including normal shell pattern, regular expressions, extended regular expressions and user-defined expression systems 4.3. Description The latest version, ksh93r, is available from AT&T Resarch at http://www.research.att.com/~gsf/download and was released on 1/24/06, with an updated alpha version (ksh93s-) on 9/12/06. See http://www.opensolaris.org/os/project/ksh93-integration/announcements/ ksh93 is available from AT&T under Common Public License [3]. This proposal will introduce a new set of binaries for ksh93. For the ksh93 and rksh93 binaries, the stability levels are as follows: The scripting interface is Uncommitted. The environment variables, .paths feature, editing modes, and pathname bindings are Volatile. Interface Description --------- ----------- /usr/bin/ksh93 the korn shell /usr/bin/rksh93 restricted shell which will be hard links to /usr/lib/isaexec; isaexec will execute the corresponding 32-bit binary in /usr/bin/{sparcv7,i86} or the 64-bit binary/usr/bin/{sparcv9,amd64}, depending on the architecture. The isaexec links will allow the 64-bit version of ksh93 to be executed by default on 64-bit platforms. Interface Description --------- ----------- /usr/bin/sparcv7/ksh93 32-bit sparc korn shell /usr/bin/sparcv7/rksh93 hard link to /usr/bin/sparcv7/ksh93 /usr/bin/sparcv9/ksh93 64-bit sparc korn shell /usr/bin/sparcv9/rksh93 hard link to /usr/bin/sparcv9/ksh93 /usr/bin/i86/ksh93 32-bit x86 korn shell /usr/bin/i86/rksh93 hard link to /usr/bin/i86/ksh93 /usr/bin/amd64/ksh93 64-bit x86 korn shell /usr/bin/amd64/rksh93 hard link to /usr/bin/amd64/ksh93 Four new shared libraries from AT&T will be introduced. One name, libcmd, collides with an existing Solaris library. To resolve this collision, libcmd must die (PSARC/2006/561) is introduced. This case is dependent on PSARC/2006/561 and may not be delivered before it. Although these libraries are to be introduced as Project Private, they could have potential use by Solaris applications and external users if upgraded to public libraries in a future project. See Section 4.5 Interfaces. libshell - contains functions used for writing extensions to ksh93 or for potentially embedding shell command processing into other commands, such as dbx. Most of ksh93 is contained in this library; the ksh93 binary itself is primarily a set of calls to libshell interfaces. libast - AT&T's AST (Advanced Software Technology) library which contains major components used by ksh93, including implementations of interfaces for I/O (sfio), memory, strings, stacks, and queue and list management. It also contains an OS abstraction layer for portabilty, such as a stdio-compatible wrapper over sfio for the benefit of plug-in modules (which need to have sfio underneath for proper integration with ksh93). libdll - portable runtime plug-in interface; allows "builtin -f foo" to use high-level dll/so names which are mapped and searched using local conventions. libcmd - the AT&T library contains built-in commands implementations which the Solaris OS implements as binaries under /usr/bin, /usr/xpg4/bin, and /usr/xpg6/bin. A subset of these AT&T implementations will be introduced as built-in commands in ksh93 (see below). Since they are project private interfaces, the AT&T libraries and the compilation environment names will not be exposed. New Ksh93 Built-ins Below is the set of eight built-in commands in ksh93 which are not built-ins in Solaris /usr/bin/ksh and which will have "pathname binding" to /bin (and implicitly to /usr/bin). These built-ins will also have pathname binding to /usr/ast/bin, as described below. cat chown head mkdir rmdir tee uniq wc With pathname binding to /bin, if, for example, "cat" is executed in ksh93 without a pathname prefix, and a $PATH search first finds an executable cat file on /bin or /usr/bin, the built-in cat command is executed, not the Solaris /bin/cat binary. The advantages of using these built-ins are increased performance and additional functionality provided by the AT&T versions of these commands (see manpages in case materials). In addition, the below two upwardly-compatible ksh93 built-ins, which are not built-ins in Solaris /usr/bin/ksh, will always be invoked when called without a pathname prefix, instead of the corresponding Solaris binary in /usr/bin: sleep printf The above ten ksh93 built-ins are designed to conform to the POSIX standard. They are also expected to be compatible with the Solaris /usr/bin utilities. Incompatibilities between the above ksh93 built-ins and the Solaris /usr/bin versions shall be fixed or the built-ins shall be disabled by default or removed from ksh93. See builtin.bugs in the materials directory for a list of incompatible behaviors in ksh93 built-ins which are known bugs to be fixed in ksh93. Although the above built-ins will be documented in manpages (see case materials), a number of additional built-in commands, with pathname binding to /usr/ast/bin, will be made available as an undocumented feature of ksh93. Note that almost all ksh93 built-ins will print an internal manpage when invoked with the --man option. /usr/ast/bin/basename /usr/ast/bin/cat /usr/ast/bin/chgrp /usr/ast/bin/chmod /usr/ast/bin/chown /usr/ast/bin/cmp /usr/ast/bin/comm /usr/ast/bin/cp /usr/ast/bin/cut /usr/ast/bin/date /usr/ast/bin/dirname /usr/ast/bin/expr /usr/ast/bin/fds /usr/ast/bin/fmt /usr/ast/bin/fold /usr/ast/bin/head /usr/ast/bin/id /usr/ast/bin/join /usr/ast/bin/ln /usr/ast/bin/logname /usr/ast/bin/mkdir /usr/ast/bin/mkfifo /usr/ast/bin/mv /usr/ast/bin/paste /usr/ast/bin/pathchk /usr/ast/bin/rev /usr/ast/bin/rm /usr/ast/bin/rmdir /usr/ast/bin/stty /usr/ast/bin/tail /usr/ast/bin/tee /usr/ast/bin/tty /usr/ast/bin/uname /usr/ast/bin/uniq /usr/ast/bin/wc Since most of these undocumented built-ins are not currently compatible with the corresponding Solaris commands in /usr/bin, these built-ins will be bound to the new pathname /usr/ast/bin. Unlike the /bin pathname binding, the /usr/ast/bin pathname binding does not require that the /usr/ast/bin directory exists. These built-ins are being included to give ksh93 users easy access to the AT&T implementations of these commands by adding /usr/ast/bin in the front of their path, without affecting default Solaris commands behavior for other ksh93 users. The stability of these undocumented built-in commands is Project Private. 4.4. Interfaces: See section 4.3 above for ksh93 binary interfaces. The following new interfaces are introduced from AT&T: Interface Stability Description --------- --------- ----------- /usr/lib/libshell.so.1 Project Private versioned 32-bit library /usr/lib/amd64/libshell.so.1 Project Private versioned 64-bit x86 library /usr/lib/sparcv9/libshell.so.1 Project Private versioned 64-bit sparc library /usr/lib/libast.so.1 Project Private versioned 32-bit library /usr/lib/amd64/libast.so.1 Project Private versioned 64-bit x86 library /usr/lib/sparcv9/libast.so.1 Project Private versioned 64-bit sparc library /usr/lib/libdll.so.1 Project Private versioned 32-bit library /usr/lib/amd64/libdll.so.1 Project Private versioned 64-bit x86 library /usr/lib/sparcv9/libdll.so.1 Project Private versioned 64-bit sparc library New interfaces will be added to the existing library libcmd.so.1: Interface Stability Description --------- --------------------- ------------ b_basename Project private AT&T built-in for basename command b_cat Project private AT&T built-in for cat command b_chgrp Project private AT&T built-in for chgrp command b_chmod Project private AT&T built-in for chmod command b_chown Project private AT&T built-in for chown command b_cmp Project private AT&T built-in for cmp command b_comm Project private AT&T built-in for cmp command b_cp Project private AT&T built-in for cmp command b_cut Project private AT&T built-in for cmp command b_date Project private AT&T built-in for date command b_dirname Project private AT&T built-in for dirname command b_expr Project private AT&T built-in for expr command b_fds Project private AT&T built-in for fds command b_fmt Project private AT&T built-in for fmt command b_fold Project private AT&T built-in for fold command b_getconf Project private AT&T built-in for getconf command b_head Project private AT&T built-in for head command b_id Project private AT&T built-in for id command b_join Project private AT&T built-in for join command b_ln Project private AT&T built-in for ln command b_logname Project private AT&T built-in for logname command b_mkdir Project private AT&T built-in for mkdir command b_mkfifo Project private AT&T built-in for mkfifo command b_mv Project private AT&T built-in for mv command b_paste Project private AT&T built-in for paste command b_pathchk Project private AT&T built-in for pathchk command b_rev Project private AT&T built-in for rev command b_rmdir Project private AT&T built-in for rmdir command b_rm Project private AT&T built-in for rm command b_stty Project private AT&T built-in for ssty command b_tail Project private AT&T built-in for tail command b_tee Project private AT&T built-in for tee command b_tty Project private AT&T built-in for tty command b_uname Project private AT&T built-in for uname command b_uniq Project private AT&T built-in for uniq command b_wc Project private AT&T built-in for wc command New directories: Interface Stability Description --------- ---------- ---------------------------- /usr/demo/ksh Uncommitted ksh93 function definitions, test suite, miscellaneous /usr/include/ast Project private AT&T header files 5. Reference Documents 1. Home page for the KornShell Command and Programming Language http://www.kornshell.com 2. The Official AT&T release of ksh93 http://www.research.att.com/sw/download/ 3. Common Public License http://www.opensource.org/licenses/cpl Also, see cpl1.0 file in case materials. 4. The Open Solaris Korn Shell 93 Integration Project http://www.opensolaris.org/os/project/ksh93-integration/ 5. User comments on the absence of ksh93 in Solaris systems See user.comments file in case materials. 6. Debian ksh packages http://packages.debian.org/unstable/shells/ksh 7. IBM AIX 5L documentation in Section 2.9.1 ksh93 from 2004: http://www.redbooks.ibm.com/redbooks/SG245765/wwhelp/wwhimpl/java/html/wwhelp.htm 6. Resources and Schedule 6.1 Projected Availability A prototype of ksh93 and its libraries has been built on Open Solaris build 37. See http://www.opensolaris.org/os/project/ksh93-integration/ for downloads of tarballs of the latest OpenSolaris ksh93. 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack