From sac-owner Tue Mar 23 21:36:27 2004 Received: from sunmail1brm.Central.Sun.COM (sunmail1brm.Central.Sun.COM [129.147.62.17]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id i2O5aQJJ008416 for ; Tue, 23 Mar 2004 21:36:27 -0800 (PST) Received: (from noaccess@localhost) by sunmail1brm.Central.Sun.COM (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) id i2O5V4M22250 for one-pager-not-2b-used-directly; Tue, 23 Mar 2004 22:31:04 -0700 (MST) Received: from sfbaymail2sca.sfbay.sun.com (sfbaymail2sca.SFBay.Sun.COM [129.145.155.42]) by sunmail1brm.Central.Sun.COM (8.11.7p1+Sun/8.11.7/ENSMAIL,v2.2) with ESMTP id i2O5V3q22235; Tue, 23 Mar 2004 22:31:03 -0700 (MST) Received: from phys-bos-2.sfbay.sun.com (phys-bos-2.SFBay.Sun.COM [129.146.14.24]) by sfbaymail2sca.sfbay.sun.com (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id i2O5V39s006880; Tue, 23 Mar 2004 21:31:03 -0800 (PST) Received: from x1-6-00-03-ba-45-00-58 (vpn-129-150-16-180.SFBay.Sun.COM [129.150.16.180]) by bos-mail1.sfbay.sun.com (Sun Java System Messaging Server 6.1 (built Feb 6 2004)) with SMTP id <0HV200H8VFBQE400@bos-mail1.sfbay.sun.com>; Tue, 23 Mar 2004 21:31:03 -0800 (PST) Date: Tue, 23 Mar 2004 21:30:49 -0800 (PST) From: govinda Subject: [/]Advanced DDI Interrupt Framework To: one-pager@sun.com Cc: ddi-intr-iteam@sun.com Reply-to: govinda Message-id: <0HV200H8WFBQE400@bos-mail1.sfbay.sun.com> MIME-version: 1.0 X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6_49 SunOS 5.10 sun4u sparc Content-type: TEXT/plain; charset=us-ascii Content-MD5: UIiI1gPfGV99VlamqFTIAw== Status: RO Content-Length: 8208 This information is Copyright 2004 Sun Microsystems, Inc. 1. Introduction 1.1. Project/Component Working Name: Advanced DDI Interrupt Framework 1.2. Name of Document Author/Supplier: Anish Gupta (anish.gupta@sun.com) Govinda Tatti (govinda.tatti@sun.com) 1.3. Date of This Document: 03/23/04 1.4. Name of Major Document Customer(s)/Consumer(s): 1.4.1. The PAC or CPT you expect to review your project: Solaris PAC 1.4.2. The ARC(s) you expect to review your project: PSARC 1.4.3. The Director/VP who is "Sponsoring" this project: Tony Barreca (tony.barreca@sun.com) 1.4.4. The name of your business unit: Core Software Technology (CST), VSP Solaris Software, OPG 1.5. Email Aliases: 1.5.1. Responsible Managers: Ivan Chung (pao.chung@sun.com) Lii Chen (lii.chen@sun.com) 1.5.2. Responsible Engineers: Anish Gupta (anish.gupta@sun.com) Govinda Tatti (govinda.tatti@sun.com) Johnny Cheung (johnny.cheung@sun.com) Wesley Shao (wesley.shao@sun.com) 1.5.3. Marketing Manager: Mark Thacker (mark.thacker@sun.com) 1.5.4. Interest List: ddi-intr@sun.com 2. Project Summary 2.1. Project Description: This project will provide an advanced DDI interrupt framework which will add the following features o Maintain source and binary compatibility for existing DDI compliant drivers. Also it accommodates re-implementation of legacy interfaces. o Support for new interrupt types, MSI (Message Signaled Interrupts), MSI-X (extended MSI) and INTx interrupts defined by PCI-Express 1.0a, PCI-X 2.0, Conventional PCI 2.3 and PCI 3.0 specifications. o Get/set capability, resource management and priority management interfaces to the new framework, making new I/O bus features available to drivers that need them. Some of these features will be available even to legacy interrupts. o Make the new framework generic enough to support other new interrupt types, where possible. o New DDI interrupt interfaces will be based on handle, which makes them more consistent with existing DDI interfaces. 2.2. Risks and Assumptions: o This framework will be tested without functional MSI/X (implies both MSI and MSI-X) capable hardware. o It is our assumption that this project will not convert any existing DDI-compliant device drivers to be MSI/X enabled. o It is our assumption that this project will not convert any existing DDI-compliant device drivers to be "new DDI interrupt framework" enabled. o This project will use a Sun supported X86 platform for proof of concept. Other X86 platforms will not be certified for lack of resources. o This project will not address any interrupt redistribution issues and these issues will be addressed by "intrd" and other related projects. 3. Business Summary 3.1. Problem Area: PCI Express devices uses in-band MSI/Xs to assert interrupts rather than using out-of-band interrupt pins. Server platforms like N2U, Eagle and future VSP X86 systems must have support for MSI/X in the DDI interrupt framework, so that device drivers can setup and handle these interrupt requests. 3.2. Market/Requester: Any PCI Express capable platform customer. Currently this includes VSP's N2U, X86 servers and ESP's Eagle server. 3.3. Business Justification: These changes are necessary to install and use Solaris on a N2U, Eagle, X86 or any PCI Express server. 3.4. Competitive Analysis: o Linux kernel 2.6 already supports MSI. Windows (Longhorn) will also support MSI. Solaris needs to stay competitive. o See N2U, Eagle Systems PRD. 3.5. Opportunity Window/Exposure: The Advanced DDI Interrupt Framework should be made part of Solaris 11. 3.6. How will you know when you are done?: Device drivers for MSI/X capable devices have the interfaces necessary to setup and use MSI/Xs. Complete delivery of DDI support for MSI/X interrupts. 4. Technical Description: Existing DDI interrupt interfaces are antiquated and need updating. New I/O buses with new interrupt programmable functionality require new support for drivers. Of course, binary and source compatibility for existing DDI-compliant drivers must be maintained. Legacy devices on legacy buses signal interrupts using an external interrupt pin. The INTx mechanism defined by the PCI Express 1.0a specification works similar to legacy interrupts, except that these are standard PCI Express messages instead of hardware signals. Legacy or INTx interrupts do not require any allocation of system resources. MSI/X interrupts are issued by devices as in-band messages. These messages are posted writes targeting a resource in the host bridge. The address and data value used in a posted write message is assigned by system software. The number of unique messages available in the host bridge hardware may be limited. Thus, MSI/X interrupts must be allocated and assigned before they can be used. A maximum of 32 MSIs are supported per function and a maximum of 2K MSI-Xs per function. A function could support INTx, MSI or MSI-X but only one of these could actually be used by the driver at any given point. In the proposed Advanced DDI Interrupt Framework, new DDI interfaces are available that allow a driver to determine the set of interrupts supported by the device, the host bridge and any device in between them. This allows advanced drivers to determine the best interrupt option. There are also new DDI interfaces available to support interrupt priority and resource management for certain interrupt types. This new DDI interrupt framework implementation will be platform neutral supporting SPARC and X86 platforms. Also it will not introduce any device performance regression. Please refer to [1] for more details on the new DDI interfaces in the proposed framework. 5. Reference Documents: [1] Advanced DDI Interrupt Framework proposal http://dtsw.sfbay/dtos/proj/fire/docs/ddi-intr.txt [2] Dynamic Interrupt Redistribution (intrd) - PSARC/2004/199 http://sac.sfbay/Archives/CaseLog/arc/PSARC/2004/199/mail [3] DDI Interrupt Re-architecture - PSARC/1997/127 http://sac.sfbay/Archives/CaseLog/arc/PSARC/1999/127/mail [4] PCI Local Bus Specification, Revision 2.2, Dec 18, 1998 [5] PCI SIG Engineering Change Notice - MSI-X, June 10, 2003 MSI-X addition and MSI Per Vector Masking addition to PCI 2.3 and PCI 3.0. [6] PCI-X Specification, Revision 2.0a, July 22, 2003 [7] PCI Express Base Specification, Revision 1.0a, April 15, 2003 [8] PCI Express Engineering Change Notice - MSI-X, Oct. 31, 2003 Note: All PCI documents and ECNs are available in the members-only download sections at http://www.pcisig.com. Since Sun is a PCI SIG member company, all Sun employees have access to the PCI SIG members-only areas by registering at the PCI SIG website. 6. Resources and Schedule: 6.1. Projected Availability: Q2CY05 6.2. Cost of Effort: Development 4.0 Engineers - 2 Months Testing 2.0 Engineers - 1 Month 6.3. Cost of Capital Resources: None. 6.4. Product Approval Committee requested information: 6.4.1. Consolidation Name: ON 6.4.2. Contributing OpCo/BU/Division Name: VSP OPG 6.4.3. Type of PAC Review and Approval expected: Standard 6.4.4. Project Boundary Conditions: N/A 6.4.5. Is this a necessary project for OEM agreements: No 6.4.6. Notes/Dependencies: o Availability of MSI/X enabled platforms. o MSI/X enabled devices and corresponding Solaris drivers to verify these new DDI interfaces. 6.4.7. Target RTI Date/Release: TBD 6.4.8. Target Code Design Review Date: TBD 6.4.9. Update approval addition: N/A 6.5. ARC review type: Standard 7. Prototype Availability: 7.1. Prototype Availability: June 2004 7.2. Prototype Cost: 4.0 Engineers - 2 Months