From sacadmin Fri Apr 21 15:23:28 2006 Received: from sac.sfbay.sun.com (localhost [127.0.0.1]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k3LMNSIQ020184; Fri, 21 Apr 2006 15:23:28 -0700 (PDT) Received: (from kais@localhost) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9/Submit) id k3LMNSvC020171; Fri, 21 Apr 2006 15:23:28 -0700 (PDT) Date: Fri, 21 Apr 2006 15:23:28 -0700 (PDT) From: Kais Belgaied Message-Id: <200604212223.k3LMNSvC020171@sac.sfbay.sun.com> To: PSARC@sac.sfbay.sun.com Cc: macrings-interest@sun.com Subject: Multiple MAC address support [PSARC/2006/265 Timeout: 04/28/2006] Status: RO Content-Length: 11587 Subject: PSARC FastTrack [04/28/2006]: Multiple MAC address support Template Version: @(#)sac_nextcase 1.56 10/26/05 SMI 1. Introduction 1.1. Project/Component Working Name: Multiple MAC address support 1.2. Name of Document Author/Supplier: Author: Rajagopal Kunhappan 1.3 Date of This Document: 21 April, 2006 4. Technical Description Multiple MAC address support ============================ release binding:patch 1 Introduction ============== There are NICs available like bge, e1000g, etc that are capable of supporting multiple MAC addresses. This proposal provides a mechanism by which mac clients can query the underlying NIC to see if it supports multiple MAC addresses and provides a way to use this feature. The feature is needed by the LDoms project (Bug 6339368: Ability to set multiple MAC addresses to a network interface) and will be used by Crossbow (which has not yet ARC'd). The proposal treats multiple MAC addresses on a NIC like any other resource, for eg. Rx/Tx rings. An entity that wants to use it should first see if it is available, reserve it and use it, and then release it. This would mean a way to add and remove MAC addresses along with modify and get should be provided by the drivers. Drivers should advertise the support of this feature through a new capability (mac_capab_t type): MAC_CAPAB_MULTIADDRESS 2 Overview ========== Mac clients wanting to use MAC_CAPAB_MULTIADDRESS capability would first use mac_capab_get() to determine if the underlying NIC supports this capability. This call will return the total number of MAC addresses supported, free address slots available and whether factory MAC addresses are available or not. The call will also return the function entry points in the driver to add, reserve, modify, get and remove a unicast MAC address. Note: Address slots are used to identify the addresses. A call to reserve or add will return this as one of the values. This value is used as the handle by modify and remove routines to identify an address to modify or remove. Address slots can take values from 0 to (total addresses - 1). If a mac client wants to use the factory MAC address, it should use maddr_factory_reserve() call to reserve a factory address. maddr_factory_reserve() will return the address, the slot where the address is reserved and enable the NIC to start filtering on the MAC address. If mac client wants to provide its own address, it should use maddr_add() instead. It will add the MAC address, enable the NIC to start filtering on the MAC address and return the slot number where the address was added. The returned slot number should be saved for it needs to be passed in the calls to modify or remove the mac address. After the mac client is done using the mac address, it should be released with a call to maddr_remove(). Since the facility provides for new MAC addresses to be added and modified, the project proposes to add a new mac layer function, mac_unicst_verify() that mac clients should use to verify the address added/modified before calling the driver entry points. 3 MAC_CAPAB_MULTIADDRESS ======================== typedef enum { MAC_CAPAB_HCKSUM, MAC_CAPAB_POLL, /* new capabilities are defined here */ MAC_CAPAB_MULTIADDRESS } mac_capab_t; mac_capab_get() is used to query for this capability and to get other information related to the capability. 4 mac_capab_get() ================ boolean_t mac_capab_get(mac_handle_t mh, mac_capab_t cap, void *cap_data); cap is MAC_CAPAB_MULTIADDRESS. cap_data is multiaddress_capab_t. typedef struct multiaddress_capab_s { int maddr_naddr; /* total addresses */ int maddr_naddrfree;/* free address slots */ /* driver entry points */ maddr_handle_t maddr_handle; /* cookie to be used for the calls */ maddr_factory_reserve_t maddr_factory_reserve; /* reserve a factory address */ maddr_add_t maddr_add; /* add a new unicst address */ maddr_remove_t maddr_remove; /* remove an added address */ maddr_modify_t maddr_modify; /* modify an added address */ maddr_get_t maddr_get; /* get address from specified slot */ } multiaddress_capab_t; Success (B_TRUE) would mean the capability is supported and cap_data will contain the toal number of MAC addresses supported (maddr_naddr) and the number of free address slots available (maddr_naddrfree) for setting new MAC addresses. Drivers supporting multiple mac addresses should provide entry points to add (maddr_add()), remove (maddr_remove()), modify (maddr_modify()) and to get (maddr_get) the addresses. Drivers that have multiple pre-defined factory MAC addresses should provide an entry point to reserve (madd_reserve()) a MAC address. If maddr_factory_reserve is non-NULL, it means factory defined multiple MAC addresses are available. Drivers should fill maddr_handle which the clients will use to access the driver entry points directly. 5 Driver entry points ===================== typedef int (*maddr_factory_reserve_t)(maddr_handle_t, mac_multi_addr_t *); typedef int (*maddr_add_t)(maddr_handle_t, mac_multi_addr_t *); typedef int (*maddr_remove_t)(maddr_handle_t, mac_addr_slot_t); typedef int (*maddr_modify_t)(maddr_handle_t, mac_multi_addr_t *); typedef int (*maddr_get_t)(maddr_handle_t, mac_multi_addr_t *); where maddr_handle_t, mac_addr_slot_t and mac_multi_addr_t are: typedef void * maddr_handle_t; typedef int mac_addr_slot_t; typedef struct mac_multi_address_s { mac_addr_slot_t mma_slot; /* slot for add/remove/get/set */ uint_t mma_addrlen; uint8_t mma_addr[MAXMACADDRLEN]; uint_t mma_flags; } mac_multi_addr_t; 5.1 maddr_factory_reserve ------------------------- typedef int (*maddr_factory_reserve)(maddr_handle_t, mac_multi_addr_t *); Some NICs come with multiple factory defined unicast addresses. In this case, the user would want to get a factory defined address. Such drivers should provide an entry point to reserve a factory defined MAC address. mac_multi_addr_t is passed in the call. All other NICs that do not have factory defined multiple MAC addresses will set this entry point to NULL. On success, the driver will fill mac_multi_addr_t with the slot number of the vendor address reserved, the reserved address and the address length. The driver will also enable the NIC to start filtering on the new reserved MAC address. The returned slot number should be saved as it needs to be passed in the call to modify or remove the address. On success, maddr_factory_reserve returns 0. If unable to reserve an address, ENOSPC is returned. To unreserve an address, maddr_remove() should be called passing the slot number that was returned by maddr_factory_reserve(). 5.2 maddr_add ------------- typedef int (*maddr_add_t)(maddr_handle_t, mac_multi_addr_t *); If multiple factory addresses are not present, or even if present but the user desires to add his own MAC address, maddr_add() entry point should be called passing the address in mac_multi_addr_t. mma_addrlen should be set to the size of the address passed. mma_flags should be set to 0. maddr_add() will find an unused address slot, set the address value to the one specified, reserve that slot and enable the NIC to start filtering on the new MAC address (along with other MAC addresses that it already is filtering on). On success, 0 is returned and mma_slot will indicate the slot to where it added the address. The returned slot number should be saved as it needs to be passed in the call to modify or remove the address. Otherwise it will return an error number indicating failure. The call can fail due to no slots available (ENOSPC), invalid address passed (EINVAL) or addrlen is invalid (EINVAL). 5.3 maddr_remove ---------------- typedef int (*maddr_remove_t)(maddr_handle_t, mac_addr_slot_t); maddr_remove() entry point should be called by the client to remove a MAC address that it added or reserved. The slot number that was returned in maddr_add() or maddr_factory_reserve() is passed in the call to remove the address. This will also cause the NIC to stop filtering on that MAC address. Returns 0 on success or EINVAL if a wrong slot is passed. 5.4 maddr_modify ---------------- typedef int (*maddr_modify_t)(maddr_handle_t, mac_multi_addr_t *); This is the routine that is used to modify the value of an address that has been added by maddr_add() or reserved by maddr_factory_reserve(). The new address, address length and the slot number that was returned in the call to add/reserve should be passed to maddr_modify(). mma_flags should be set to 0. Returns 0 on success. The request can fail with EINVAL if an invalid slot is passed. A slot is invalid if it is out of range or is not reserved or added by the client. 5.5 maddr_get ------------- typedef int (*maddr_get_t)(maddr_handle_t, mac_multi_addr_t *); Will get all the information related to the address slot passed in mac_multi_addr_t as well as the address. mma_flags should be set to 0 in the call. /* mma_flags values */ #define MMAC_SLOT_USED 0x1 /* address slot used */ #define MMAC_VENDOR_ADDR 0x2 /* address returned is vendor supplied */ On success, mma_flags can take the following values: 1) MMAC_SLOT_USED 2) MMAC_VENDOR_ADDR 3) MMAC_SLOT_USED | MMAC_VENDOR_ADDR 4) 0 (MMAC_SLOT_USED) indicates that the slot is being used and mma_addr contains the address. (MMAC_VENDOR_ADDR) indicates that mma_addr contains factory address and the address is unused. (MMAC_SLOT_USED|MMAC_VENDOR_ADDR) indicates that the slot is used and there is factory defined address for that slot and that address is returned in mma_addr. If none of the bits are set on return, it means that the slot is free and there is no address in the slot. maddr_get() return 0 on success and fails with EINVAL if the slot is invalid. 6 mac_unicst_verify =================== boolean_t mac_unicst_verify(mac_handle_t mh, const uint8_t *addr); mac_unicst_verify() will test if the passed address is a group address or not. It returns B_TRUE if address is not a group address. mac_unicst_verify() should be called by the clients before they pass the MAC address to the driver's add (maddr_add) or modify (maddr_modify) entry points. 7 Interface Table ================= _____________________________________________________________________________ | Interfaces Exported | |_______________________|_______________________|___________________________| |Interface | Classification | Comments | |_______________________|_______________________|___________________________| | | | | |MAC_CAPAB_MULTIADDRESS | Consolidation Private | | |mac_unicst_verify() | Consolidation Private | | |multiaddress_capab_t | Consolidation Private | | |mac_multi_addr_t | Consolidation Private | | |_______________________|_______________________|___________________________| 8 References: ============= PSARC/2006/248 Nemo MAC-Type Plugin Architecture PSARC/2006/249 Nemo Changes for Binary Compatibility 6. Resources and Schedule 6.4. Steering Committee requested information 6.4.1. Consolidation C-team Name: ON 6.5. ARC review type: FastTrack From sacadmin Mon Apr 24 02:44:00 2006 Received: from nis-uk.uk.sun.com (nis-uk.UK.Sun.COM [129.156.85.41]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k3O9hxIQ008177; Mon, 24 Apr 2006 02:44:00 -0700 (PDT) Received: from enospc.uk.sun.com (enospc [129.156.173.14]) by nis-uk.uk.sun.com (8.12.10+Sun/8.12.10/ENSMAIL,v2.2) with ESMTP id k3O9hwoR019448; Mon, 24 Apr 2006 10:43:58 +0100 (BST) Received: from [129.150.120.102] (vpn-129-150-120-102.UK.Sun.COM [129.150.120.102]) by enospc.uk.sun.com (8.13.4+Sun/8.13.3/CTE 3.0) with ESMTP id k3O9hve8023388; Mon, 24 Apr 2006 10:43:58 +0100 (BST) Message-ID: <444C9DDF.7060805@Sun.COM> Date: Mon, 24 Apr 2006 05:43:59 -0400 From: Darren J Moffat User-Agent: Thunderbird 1.5 (X11/20060113) MIME-Version: 1.0 To: Kais Belgaied CC: PSARC@sac.sfbay.sun.com, macrings-interest@Sun.COM Subject: Re: Multiple MAC address support [PSARC/2006/265 Timeout: 04/28/2006] References: <200604212223.k3LMNSvC020171@sac.sfbay.sun.com> In-Reply-To: <200604212223.k3LMNSvC020171@sac.sfbay.sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 185 All very cool but.... how does the administrator use this ? Or is that something we expect to see in the LDOM and Crossbow cases ? If so thats an acceptable answer for me. Darren From sacadmin Mon Apr 24 08:12:55 2006 Received: from jurassic.eng.sun.com (jurassic.SFBay.Sun.COM [129.146.224.130]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k3OFCsIQ027293; Mon, 24 Apr 2006 08:12:54 -0700 (PDT) Received: from [129.146.108.78] (yagachi.SFBay.Sun.COM [129.146.108.78]) by jurassic.eng.sun.com (8.13.6+Sun/8.13.6) with ESMTP id k3OFCs69515365; Mon, 24 Apr 2006 08:12:54 -0700 (PDT) Message-ID: <444CEA8D.1050508@sun.com> Date: Mon, 24 Apr 2006 08:11:09 -0700 From: Rajagopal Kunhappan Reply-To: rajagopal.kunhappan@sun.com User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050322) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Darren J Moffat CC: Kais Belgaied , PSARC@sac.sfbay.sun.com, macrings-interest@sun.com Subject: Re: Multiple MAC address support [PSARC/2006/265 Timeout: 04/28/2006] References: <200604212223.k3LMNSvC020171@sac.sfbay.sun.com> <444C9DDF.7060805@Sun.COM> In-Reply-To: <444C9DDF.7060805@Sun.COM> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 1198 Darren J Moffat wrote: > All very cool but.... > > how does the administrator use this ? Or is that something we expect > to see in the LDOM and Crossbow cases ? If so thats an acceptable > answer for me. In the case of crossbow, you can have many vnic's (virtual nics on top of real NICs) that can have their own MAC address. The user would use dladm command to create vnic with its own MAC address: "dladm create-vnic -d bge0 -m 0" This would create a vnic0 with its own MAC address over which you can plumb IP. More on this in the crossbow case. For the LDOM project, they have vswitch that directs traffic to the vnet's in various DOMs. Each of this vnet can have their own MAC addresses. The vswitch driver currently does a mac_open of the underlying NIC device driver (e.g. bge) and then sets the interface into promiscuous mode. The reason for this is that currently they can't program multiple unicast MAC addresses into the NIC so the only way the switch can see and route packets to its client logical domains is to see all the pkts coming over the interface, i.e. to set it into promiscuous mode. This proposal will address their issue. Thanks, -krgopi From sacadmin Wed Apr 26 10:12:29 2006 Received: from jurassic.eng.sun.com (jurassic.SFBay.Sun.COM [129.146.58.166]) by sac.sfbay.sun.com (8.12.9+Sun/8.12.9) with ESMTP id k3QHCTIQ011852 for ; Wed, 26 Apr 2006 10:12:29 -0700 (PDT) Received: from [129.146.225.164] (d-mpk17-225-164.SFBay.Sun.COM [129.146.225.164]) by jurassic.eng.sun.com (8.13.6+Sun/8.13.6) with ESMTP id k3QHCTRw899598; Wed, 26 Apr 2006 10:12:29 -0700 (PDT) Message-ID: <444FB80C.6070903@Sun.COM> Date: Wed, 26 Apr 2006 11:12:28 -0700 From: Kais Belgaied User-Agent: Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.7) Gecko/20060120 X-Accept-Language: ar-eg, en-us, en, ar, ar-dz, ar-bh, ar-iq, ar-jo, ar-kw, ar-lb, ar-ly, ar-ma, ar-om, ar-qa, ar-sa, ar-sy, ar-tn, ar-ae, ar-ye MIME-Version: 1.0 To: rajagopal.kunhappan@Sun.COM CC: PSARC@sac.sfbay.sun.com, macrings-interest@Sun.COM Subject: Re: Multiple MAC address support [PSARC/2006/265 Timeout: 04/28/2006] References: <200604212223.k3LMNSvC020171@sac.sfbay.sun.com> <444C9DDF.7060805@Sun.COM> <444CEA8D.1050508@sun.com> In-Reply-To: <444CEA8D.1050508@sun.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Status: RO Content-Length: 59 This case was approved in today's PSARC meeting Kais. From sacadmin Thu Feb 22 10:03:05 2007 Received: from sunmail4.Singapore.Sun.COM (sunmail4.Singapore.Sun.COM [129.158.71.19]) by sac.sfbay.sun.com (8.13.6+Sun/8.13.6) with ESMTP id l1MI331r025433 for ; Thu, 22 Feb 2007 10:03:04 -0800 (PST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by sunmail4.Singapore.Sun.COM (8.13.4+Sun/8.13.3/ENSMAIL,v2.2) with ESMTP id l1MI2gGZ000976; Fri, 23 Feb 2007 02:02:58 +0800 (SGT) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0JDV00G05M4VHM00@nwk-avmta-1.sfbay.Sun.COM>; Thu, 22 Feb 2007 10:02:55 -0800 (PST) Received: from jurassic.eng.sun.com ([129.146.226.130]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0JDV002VCM4VX790@nwk-avmta-1.sfbay.Sun.COM>; Thu, 22 Feb 2007 10:02:55 -0800 (PST) Received: from [129.145.154.105] (sr1-umpk-55.SFBay.Sun.COM [129.145.154.105]) by jurassic.eng.sun.com (8.13.8+Sun/8.13.8) with ESMTP id l1MI2sVR200154; Thu, 22 Feb 2007 10:02:54 -0800 (PST) Date: Thu, 22 Feb 2007 10:02:53 -0800 From: Kais Belgaied Subject: Clarifications and changes to PSARC/2006/265 - Multiple MAC address support To: PSARC@sun.com Cc: gld-iteam , Yuming.Chen@sun.com Message-id: <45DDDACD.7080006@Sun.COM> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7BIT X-Accept-Language: ar-eg, en-us, en, ar, ar-dz, ar-bh, ar-iq, ar-jo, ar-kw, ar-lb, ar-ly, ar-ma, ar-om, ar-qa, ar-sa, ar-sy, ar-tn, ar-ae, ar-ye X-PMX-Version: 5.2.0.264296 User-Agent: Mozilla/5.0 (X11; U; SunOS sun4v; en-US; rv:1.7) Gecko/20060120 Status: RO Content-Length: 5349 These are clarifications and minor changes to the case, requested by the project team (author: Nicolas Droux). The changes did not seem substancial enough to warrant a new case, or changing the approal state of this one. During the implementation of a consumer of the interfaces defined by this case, it was found that some of the semantics associated with the interface were not clearly defined by the original case. It was also found that the functionality provided by the factory MAC address reservation entry point was not always sufficient. 1. renaming maddr_factory_reserve to maddr_reserve It was found during the implementation of the multiple MAC address interface that the structure and field names maddr_factory_reserve_t and maddr_factory_reserve should be instead maddr_reserve_t and maddr_reserve, respectively. The remaining of this document uses these new names. 2. Clarification on maddr_get The behavior of the maddr_get() entry point was under specified. The following behavior must be implemented by the driver: a) The mma_addrlen field must always be set by maddr_get() when it returns successfully. b) The mma_flag of the structure passed as argument must be initialized by the driver maddr_get() entry point. c) The maddr_get() entry must return the MAC address associated with a factory MAC address slot, even when that slot is not in use. 3. Clarification on slot 0 The original case implied the presence of a special slot 0 which always corresponds to the primary MAC address of the NIC implementing the multiple MAC address capability. This special slot 0 is reserved by the NIC, and is accessed using the mac_unicast_* routines already provided by GLDv3. The maddr_naddr field should contain the number of slots that are accessible by the maddr_* routines, i.e. it should not include slot 0. The valid range of slot numbers is therefore [1..maddr_naddr]. 4. Clarification on maddr_naddr with factory addresses The maddr_naddr field reflects both the factory and non-factory MAC address slots. E.g. with N factory MAC address slots and M non-factory MAC address slots, maddr_naddr should be set to N+M by the driver. Note that if there is a combination of factory and non-factory slots, it is possible for maddr_reserve() to return EBUSY when the factory slots are exhausted, even though there may be non-factory slots available. maddr_naddrfree would be set to non-zero in this case. 5. Allow maddr_reserve() to take a specified slot number The multiple MAC address feature allows factory MAC addresses provided by the hardware to be assigned to virtual NICs assigned to multiple virtual machines running on the same host. Once a factory MAC address has been assigned to one of these virtual machines, the same MAC address should be reassigned to that virtual on reboot or the virtual machine or on reboot of the host. In order to preserve the MAC address, the client should be able to specify the factory MAC address slot to be reserved. Unfortunately the original maddr_reserve() function always picked any available slot. So we are proposing to change the semantics of the mma_slot field of the structure passed to the the maddr_reserve() entry point as follows: - If mma_slot is set to -1, the maddr_reserve() entry point will find the next available factory MAC address slot. If such slot is available, mma_slot on exit will be set to the reserved slot. If no more slots are available, maddr_reserve() will fail with ENOSPC. - If mma_slot is greater than 0, mma_slot specifies the slot number to be reserved by maddr_reserve(). If that slot is already reserved, maddr_reserve() fails with EBUSY. If the slot number specific is not valid, maddr_reserve() fails with EINVAL. Note that this fasttrack assumes that the factory MAC address values associated with slots does not change across reboots or hardware reset. Changing these MAC address values would be similar to changing the primary MAC address of a physical NIC, and the mechanisms used to perform such changes are outside the scope of this case. 6. Clarifications on reserving vendor address slots from maddr_add() In some cases it is possible for a MAC client to want to use the MAC addresses of its chosing instead of the factory MAC addresses. In this scenario, it should be possible for best peformance to use MAC address slots instead of putting the NIC in promiscous mode. The maddr_add() entry should first attempt to reserve unused slots which do not contain vendor MAC addresses. If these slots are exhausted, it should then use unused factory MAC address slots, if any are available, and the hardware allows that operation. If a vendor MAC address slot is reserved by the driver during a maddr_add() call, it is the responsibility of the driver to save the factory address that was associated with that slot, and restore that factory MAC address when the slot is released through the maddr_remove() entry point. 7. State of the NIC when multiple MAC address calls are invoked It should be noted that the multiple MAC addresses calls defined by this case can be called before the NIC is started through its start entry point. It is the responsibility of the driver to perform all required hardware initialization needed by the entry points of the multiple MAC address capability. Kais.