From tpm@darwin Sat May 17 17:13:54 1997 Date: Sat, 17 May 1997 17:16:43 -0700 From: tpm@darwin (Tim Marsland) To: psarc@Eng Subject: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng Content-Length: 4362 Direct Execution of Java Programs on Solaris -------------------------------------------- Java programs are composed of collections of one or more class files. This can sometimes make it unwieldy to move applications or install them. PSARC/1996/330 introduced the javald(1) command to allow the numerous dependencies of a Java application to be captured in a (separate) executable (a shell script). However, this did not attempt to solve the distribution or installation problems. In JDK1.1, JavaSoft introduced JAR (Java ARchive) format [1], which is largely based on the ubiquitous ZIP file format [2]. This format allows class files and other data files to be collected together and stored as a single executable file, while still allowing them to be read and executed by the Java Virtual Machine (JVM): the contents of the JAR file appear to be prepended to the CLASSPATH of the JVM. JavaSoft is now introducing a platform-independent executable format based on this same JAR file format. Such executables are basically JAR files, but with a special 'extra field' marker; an application launcher is used to find the JVM and feed it the contents of the executable. The format of the executable has been agreed upon, after much review and discussion, with the appropriate JavaSoft engineers [3]. On Windows, the name of the launcher is inferred from the file extension e.g. '.jxe'. On Solaris, this project extends exec(2) to recognise this file type, and invokes the JVM application launcher from a known point in the filesystem. This will enable a '100% pure Java' application to be captured in a single file and allow it to run on Solaris (or on Windows) without any shell scripts or other form of platform-specific wrappers being created. The application launcher has been developed by JavaSoft. It is part of JDK1.1.2, and will naturally ship as part of the JVM consolidation on Solaris. The kernel module needed to bind the executable to the launcher will ship as part of the ON consolidation as an exec module (/usr/kernel/exec/javaexec) which is the subject of this PSARC case. The file(1) program will also be enhanced to recognize ZIP files and Java executable files. Interfaces Imported ------------------- - Specification of Java executable files, controlled by JavaSoft. The ZIP format is fully specified (see [2]); the kernel relies on the following tests before invoking the launcher: (a) That bytes zero through three of the file contain the bytes 'P', 'K', '\003', '\004' This is the magic number at the beginning of the "local file header" at the head of a JAR archive. Essentially, this identifies the executable as a ZIP file. (b) That bytes 28 and 29 contain a little-endian short word describing a length 'flen'. That bytes 30 and 31 contain a little-endian short word describing length 'xlen' that is at least 2. That within the xlen bytes of byte 31 + flen, a member of the list of extra field data headers contains '0xfe' followed by '0xca'. Test (b) corresponds to checking that the first 'local file header' of a ZIP archive has 'extra field data' which contains a header element which is the magic number '0xcafe' (little-endian). This is not a complete validation of the format of the executable. Failure to pass these tests result in exec(2) returning ENOEXEC. Successfully passing these tests, but with an otherwise invalid executable will cause exec to succeed, but the application launcher to complain. - The name of the JVM launcher for Java applications is: /usr/java/bin/jre which is invoked with the flag "-jar" plus the pathname of the executable. Committment level: Stable. Interfaces Exported ------------------- - The name of the launcher can be explicitly overridden using: set javaexec:jre = "/some/where/else" and given a different arg using: set javaexec:jre_arg = "-other" in /etc/system. The expectation is that these variables will principally be of use to those maintaining and updating the JVM and the jre, thus they will not be documented. Committment level: Evolving. References ---------- [1] Java Archive information. http://java.sun.com/products/jdk/1.1/docs/guide/jar/index.html [2] ZIP Archive Format. ftp://ftp.uu.net/pub/archiving/zip/doc [3] Conversations and email with David Connelly of JavaSoft. From tcm@kindra-79 Tue May 20 14:43:13 1997 Date: Tue, 20 May 1997 14:43:12 -0700 From: tcm@kindra-79 (Terrence Miller) To: psarc@Eng, tpm@darwin Subject: Re: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng Content-Length: 596 > The application launcher has been developed by JavaSoft. It is part of > JDK1.1.2, and will naturally ship as part of the JVM consolidation on > Solaris. > - The name of the JVM launcher for Java applications is: > > /usr/java/bin/jre The PSARC cases which approved bundling of the JVM (1996/330 and 1996387) did not include /usr/java/bin/jre. Since that program has a substantially different user interface than /usr/java/bin/java (not just a change in input file type), bundling it should be an explicit part of this or some other case and a specification should be provided. Terry From tpm@darwin Tue May 20 17:31:36 1997 Date: Tue, 20 May 1997 17:34:36 -0700 From: tpm@darwin (Tim Marsland) To: psarc@Eng, tpm@darwin, tcm@kindra-79 Subject: Re: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng Content-Length: 2193 | From tcm@kindra-79 Tue May 20 15:10 PDT 1997 | | > The application launcher has been developed by JavaSoft. It is part of | > JDK1.1.2, and will naturally ship as part of the JVM consolidation on | > Solaris. | | > - The name of the JVM launcher for Java applications is: | > | > /usr/java/bin/jre | | The PSARC cases which approved bundling of the JVM (1996/330 and 1996387) | did not include /usr/java/bin/jre. Since that program has a substantially | different user interface than /usr/java/bin/java (not just a change in | input file type), bundling it should be an explicit part of this or | some other case and a specification should be provided. The 1.1.2 JDK contains 'jre'. It's a JavaSoft thing. It uses the JNI invocation API to fire up the platforms JVM. Why would we ship a Solaris JVM that doesn't contain it? Isn't this "batteries not included"? If you like, we can go through the somewhat sterile exercise of approving that the JVM consolidation ships something conformant with the 1.1.2 contents. But I would far rather we (PSARC) were in the business of reviewing proposals to NOT ship things that JavaSoft ship, and not be piece-part approving JDK components. Thus I think it should require an ARC case to not ship the 'jre' program, rather than vice versa. We deliberately designed the javaexec/jar hooks so that JavaSoft own both the format of the executables created by 'jar', as well as the program that interprets the executable in order to run it (i.e. 'jre') so that they can extend it independently of us. The only thing that we're both committed to is the initial magic number, the '0xcafe' header in the extra field data of the first file in the archive, and the semantics of 'jre -jar' meaning "run this application using the platforms' JVM" Another idea is for us to add an extra layer of indirection so that the kernel invokes a different launcher program e.g. '/usr/lib/javal' and then that launcher can decide how to find and invoke the JRE and thus the JVM. This does seem like a bit too much overkill to me, but it may increase your comfort level. tim P.S. Joe Eykholt -is- aware of his login being used by JavaSoft in this way ;-) From tcm@kindra-79 Wed May 21 10:40:29 1997 Date: Wed, 21 May 1997 10:40:27 -0700 From: tcm@kindra-79 (Terrence Miller) To: psarc@Eng, tpm@darwin, tcm@kindra-79 Subject: Re: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng Content-Length: 716 > From tpm@darwin Tue May 20 17:31:38 1997 > The 1.1.2 JDK contains 'jre'. It's a JavaSoft thing. It uses the JNI > invocation API to fire up the platforms JVM. Why would we ship a > Solaris JVM that doesn't contain it? ...... Because it is a new feature added in the late stages of micro release with no formal architecture review and no documentation has been supplied to go with the Solaris release. We wouldn't let anyone else just drop a new binary into bundled Solaris, so why should JavaSoft be allowed to do it. As far as I have been able to tell, because of the late arrival of this feature, its first customer exposure will be FCS and there will have been ZERO product testing by SunSoft. Terry From robinson@jetsun Wed May 21 10:45:58 1997 Date: Wed, 21 May 1997 10:46:10 -0700 From: robinson@jetsun (David Robinson) To: psarc@Eng Subject: Re: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Length: 1096 > > The 1.1.2 JDK contains 'jre'. It's a JavaSoft thing. It uses the JNI > > invocation API to fire up the platforms JVM. Why would we ship a > > Solaris JVM that doesn't contain it? ...... > > Because it is a new feature added in the late stages of micro release > with no formal architecture review and no documentation has been supplied > to go with the Solaris release. > > We wouldn't let anyone else just drop a new binary into bundled Solaris, so > why should JavaSoft be allowed to do it. As far as I have been able to tell, > because of the late arrival of this feature, its first customer exposure > will be FCS and there will have been ZERO product testing by SunSoft. It is presumed that JavaSoft does some architectural review. What we are concerned with is if this fits into the Solaris architecture? From what I have read it does fit cleanly. It follows the existing architecture for exec'ing used by COFF or intpexec. You have a good case why the C/W-team might consider leaving this feature out of 2.6. But that is a business case, not an architectural issue. -David From tpm@darwin Wed May 21 11:17:02 1997 Date: Wed, 21 May 1997 11:20:04 -0700 From: tpm@darwin (Tim Marsland) To: psarc@Eng, tpm@darwin, tcm@kindra-79 Subject: Re: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng Content-Length: 2002 | From tcm@kindra-79 Wed May 21 10:37 PDT 1997 | | > From tpm@darwin Tue May 20 17:31:38 1997 | | > The 1.1.2 JDK contains 'jre'. It's a JavaSoft thing. It uses the JNI | > invocation API to fire up the platforms JVM. Why would we ship a | > Solaris JVM that doesn't contain it? ...... | | Because it is a new feature added in the late stages of micro release | with no formal architecture review and no documentation has been supplied | to go with the Solaris release. This is pure "batteries not included". We tell people "we have a JVM compatible with JDK1.1.2" - except that it isn't because we missed something out. So people ask "why did you miss that out"? I guess we can say "JavaSoft is very naughty and they ship new things in micro releases but we want to protect you from that". But since we do exactly the same thing as they do, and no customer really seems to be very upset by that, I don't think that's a very compelling argument. All we did is to annoy people by missing something out. Given your concerns about architectural review, we should insist that the current wad that the JVM consolidation is delivering to the 2.6 WOS i.e. "JDK1.1.2 minus jre" is architecturally reviewed so that the omission of the jre program from that consolidation can be considered explicitly. | We wouldn't let anyone else just drop a new binary into bundled Solaris, so | why should JavaSoft be allowed to do it. We do let JavaSoft drop arbitrary chunks into Solaris. They're called class libraries :-) We test them as best we can, but we have an explicit goal of tracking their product so that our customers get coherent snapshots of JavaSoft releases. | As far as I have been able to tell, | because of the late arrival of this feature, its first customer exposure | will be FCS and there will have been ZERO product testing by SunSoft. Clearly, this is a legitimate concern which should be addressed elsewhere, but it has nothing to do with the architecture of this change. tim From jek3@pacific-88 Wed May 21 12:22:52 1997 Date: Wed, 21 May 1997 12:23:03 -0700 From: jek3@pacific-88 (Joseph Kowalski) To: psarc@Eng, tpm@darwin Subject: Re: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng Content-Length: 5769 Sorry to be so close to meeting time with this, but I've got a few questions. Fortunately, I think most are pretty trivial. > From tpm@darwin Sat May 17 17:14:05 1997 ... > Direct Execution of Java Programs on Solaris > -------------------------------------------- > > Java programs are composed of collections of one or more class > files. This can sometimes make it unwieldy to move applications > or install them. PSARC/1996/330 introduced the javald(1) command to allow > the numerous dependencies of a Java application to be captured in > a (separate) executable (a shell script). However, this did not attempt > to solve the distribution or installation problems. > > In JDK1.1, JavaSoft introduced JAR (Java ARchive) format [1], which is > largely based on the ubiquitous ZIP file format [2]. I infer that [1] != [2] (hopefully [1] supersets [2]. > This format > allows class files and other data files to be collected together and > stored as a single executable file, while still allowing them to be > read and executed by the Java Virtual Machine (JVM): the contents of > the JAR file appear to be prepended to the CLASSPATH of the JVM. > > JavaSoft is now introducing a platform-independent executable format > based on this same JAR file format. Such executables are basically > JAR files, but with a special 'extra field' marker; an application > launcher is used to find the JVM and feed it the contents of the executable. > > The format of the executable has been agreed upon, after much review > and discussion, with the appropriate JavaSoft engineers [3]. > > On Windows, the name of the launcher is inferred from the file extension > e.g. '.jxe'. On Solaris, this project extends exec(2) to recognise > this file type, and invokes the JVM application launcher from a known > point in the filesystem. > > This will enable a '100% pure Java' application to be captured in a > single file and allow it to run on Solaris (or on Windows) without any > shell scripts or other form of platform-specific wrappers being created. > > The application launcher has been developed by JavaSoft. It is part of > JDK1.1.2, and will naturally ship as part of the JVM consolidation on > Solaris. > > The kernel module needed to bind the executable to the launcher will > ship as part of the ON consolidation as an exec module > (/usr/kernel/exec/javaexec) which is the subject of this PSARC case. > > The file(1) program will also be enhanced to recognize > ZIP files and Java executable files. (Assuming they aren't exactly the same as generic zip files (as specified by [2])) will this also recognize jars? If so, how are to two distinguished? > Interfaces Imported > ------------------- > > - Specification of Java executable files, controlled by JavaSoft. > > The ZIP format is fully specified (see [2]); the kernel relies > on the following tests before invoking the launcher: > > (a) That bytes zero through three of the file contain > the bytes 'P', 'K', '\003', '\004' > > This is the magic number at the beginning of the > "local file header" at the head of a JAR archive. > Essentially, this identifies the executable as a ZIP > file. > > (b) That bytes 28 and 29 contain a little-endian short > word describing a length 'flen'. > > That bytes 30 and 31 contain a little-endian short > word describing length 'xlen' that is at least 2. > > That within the xlen bytes of byte 31 + flen, a member > of the list of extra field data headers contains '0xfe' > followed by '0xca'. > > Test (b) corresponds to checking that the first 'local > file header' of a ZIP archive has 'extra field > data' which contains a header element which is the > magic number '0xcafe' (little-endian). Is this the answer to distinquishing a jar as a specific form of zip file? (In which case maybe zip supersets jar - I'm obviously a little confused here.) > This is not a complete validation of the format of the executable. > > Failure to pass these tests result in exec(2) returning > ENOEXEC. Successfully passing these tests, but with an > otherwise invalid executable will cause exec to succeed, but > the application launcher to complain. > > - The name of the JVM launcher for Java applications is: > > /usr/java/bin/jre > > which is invoked with the flag "-jar" plus the pathname of the > executable. > > Committment level: Stable. Do I read this as "/usr/java/bin/jre -jar" is Stable and nothing else? If its more, what other options does jre have? Does this declare the behaivor of "/usr/java/bin/jre" as stable? (I don't think there is a wrong answer here. I just find this a little ambiguous.) > Interfaces Exported > ------------------- > > - The name of the launcher can be explicitly overridden using: > > set javaexec:jre = "/some/where/else" > > and given a different arg using: > > set javaexec:jre_arg = "-other" > > in /etc/system. The expectation is that these variables will > principally be of use to those maintaining and updating the JVM > and the jre, thus they will not be documented. > > Committment level: Evolving. Sounds Unstable or something Private to me. Evolving is something we encourage enduser to *think* about using. I assume the motivation here is to deliver a new JVM (with a renamed launcher) and miminize the patch impact. If so, I'd go with unstable. > References > ---------- > > [1] Java Archive information. > http://java.sun.com/products/jdk/1.1/docs/guide/jar/index.html > > [2] ZIP Archive Format. ftp://ftp.uu.net/pub/archiving/zip/doc In the past, this stuff has tended to evolve. Do we need to bind to a rev. of this spec? If so, what rev? > [3] Conversations and email with David Connelly of JavaSoft. - jek3 From tpm@darwin Wed May 21 12:58:54 1997 Date: Wed, 21 May 1997 13:01:57 -0700 From: tpm@darwin (Tim Marsland) To: psarc@Eng, tpm@darwin, jek3@pacific-88 Subject: Re: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng Content-Length: 2002 | > In JDK1.1, JavaSoft introduced JAR (Java ARchive) format [1], which is | > largely based on the ubiquitous ZIP file format [2]. | | I infer that [1] != [2] (hopefully [1] supersets [2]. Nope, ZIP files are still a more general file format. A JAR file is a ZIP file with particular contents i.e. Java .class files. | > The file(1) program will also be enhanced to recognize | > ZIP files and Java executable files. | | (Assuming they aren't exactly the same as generic zip files (as specified | by [2])) will this also recognize jars? If so, how are to two distinguished? I'm not proposing to make 'file' recognize anything other than ZIP files and executables here. I can ask JavaSoft which parts of a JAR file they consider to be immutable, so that I can make that distinction if you think that's worthwhile. Though perhaps the recognition of JAR files should be a separate bug-fix -- since such ZIP files already exist e.g. the 'classes.zip' that ships with every JVM, there's a good reason to enhance 'file' that way today. | Is this the answer to distinquishing a jar as a specific form of zip | file? Not qwite. The 0xcafe tells you it's an executable JAR file. | Do I read this as "/usr/java/bin/jre -jar" is Stable and nothing else? Yes. It may be that other flags should also considered Stable, but that's not what this case is about. | Sounds Unstable or something Private to me. Evolving is something we | encourage enduser to *think* about using. | | I assume the motivation here is to deliver a new JVM (with a renamed | launcher) and miminize the patch impact. If so, I'd go with unstable. Unstable is fine with me. I don't want anyone to build their product on these. | > [2] ZIP Archive Format. ftp://ftp.uu.net/pub/archiving/zip/doc | | In the past, this stuff has tended to evolve. Do we need to bind to | a rev. of this spec? If so, what rev? I've dropped the relevant spec in the case materials subdirectory. This is the 970311 version of the spec. tim From tpm@darwin Thu May 22 00:04:31 1997 Date: Thu, 22 May 1997 00:07:37 -0700 From: tpm@darwin (Tim Marsland) To: psarc@Eng Subject: Re: PSARC/1997/123 - javaexec - fast-track Cc: tpm@Eng Content-Length: 2395 This case was formally approved at this afternoons ARC business meeting, with the caveat that the delivery of the project is conditional on the completion of the review of the contents of the JVM-on-Solaris consolidation that will capture the specification of the jre command. Now the fun begins. Turns out that the JVM consolidation is currently up-to-date with 1.1.2 except for the 'jre' part of it, -and- they're also taking other critical bug fixes from the JDK1.1.3 source base. It also turns out that the 'jre -jar' option is NOT available in the 1.1.2 delivery, though it will be available in early versions of the 1.1.3 JDK. Similarly, the ability to create Java executables will not be available until 1.1.3. So, I'm left in something of a quandary w.r.t. what to do here. First, I will submit the JVM-on-Solaris 2.6 ARC case as we discussed, so that we know what we're shipping there, including the jre specification. But I'd also like to get the support for Java executables into 2.6 so that we don't get into post-FCS kernel patches. But I think that the JavaSoft end of this does not feel nearly complete enough to be confident about any of it making 2.6. What would be ideal would be to repartition the problem and its solution while the jre picture is stabilizing. So, I'd like to propose to the ARC that this project become transformed into a two projects with two-part delivery. Part 1 (this case) Deliver the existing specification, except that we change the name of the Java application launcher to /usr/java/lib/jexec. The committment level and interfaces are otherwise unchanged. We deliberately do not deliver /usr/java/lib/jexec. Ensure that the javaexec module returns ENOEXEC if the application launcher (/usr/java/lib/jexec) cannot be found. With all this in place, attempts to execute Java executables will result in an ENOEXEC error return from exec(2) i.e. the system behaviour will be unchanged. But we will have a hook for subsequent delivery of the support for Java executables when we ship a /usr/java/lib/jexec as part of an update to the JVM consolidation. Part 2 (a future case) Deliver /usr/java/lib/jexec -either- as a symlink to ../bin/jre -or- as a separate executable that will find and exec jre. How's that sound? Unless I hear objections, what I propose to do is to let the timer run until the timeout at the end of the week. tim From jean.bandfield@mailhost.eng.sun.com Fri Jan 29 09:56:30 1999 Date: Fri, 29 Jan 1999 09:55:51 -0800 (PST) From: Joseph Kowalski To: psarc@lowpost.eng.sun.com Subject: psarc/1997/123 Cc: kto@lowpost.eng.sun.com Content-Length: 1220 I just noticed a "bug" in this case, which is the baseline for the automatic execution of jar files on Solaris. When Tim at the last minute changed the name of the target executable from /usr/java/bin/jre to /usr/java/lib/jexec (a good thing), he forgot to change the names of the associated Unstable kernel variables from javaexec:jre and javaexec:jre_arg to javaexec:jexec and javaexec:jexec_arg. The implementation has always had the jexec names and I'm posting this so that this inconsistancy can be captured in the mail log (there is no rewritten spec). In this case it is *far* better to change the spec (which was an opps anyway) rather than change the implementation. This is so trivial that I'm not reopening the case, or setting a timeout or anything like that. Just respond if anybody is unhappy with this. I've strunk the process as much as possible to fit. As a matter of fact, I've shrunk this so much, that I didn't even talk to Tim about this specifically (although I've talked to him a bunch about this feature). I hope I'm not annoying him by shrinking that step out of the process. Anyway, as I said, if anyone has a problem with this, just speak up and I'll do something more formal. - jek3