2 .\" Copyright (c) 2004, Sun Microsystems, Inc. All Rights Reserved.
3 .\" The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License.
4 .\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License.
5 .\" When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner]
6 .TH CFGADM_IB 1M "Jul 2, 2008"
8 cfgadm_ib \- InfiniBand hardware specific commands for cfgadm
12 \fB/usr/sbin/cfgadm\fR \fB-f\fR [\fB-y\fR | \fB-n\fR] [\fB-v\fR] \fB-c\fR \fIfunction\fR \fIap_id\fR...
17 \fB/usr/sbin/cfgadm\fR [\fB-f\fR] [\fB-y\fR | \fB-n\fR] [\fB-v\fR] [\fB-o\fR \fIhardware_options\fR]
18 \fB-x\fR \fIhardware_function\fR \fIap_id\fR...
23 \fB/usr/sbin/cfgadm\fR \fB-v\fR [\fB-a\fR] [\fB-s\fR \fIlisting_option\fR] [-] [\fIap_id\fR | \fIap_type...\fR]
28 \fB/usr/sbin/cfgadm\fR \fB-v\fR \fB-h\fR [\fIap_id\fR]...
34 The InfiniBand hardware specific library \fB/usr/lib/cfgadm/ib.so.1\fR provides
35 the functionality for administering its fabric through the \fBcfgadm\fR(1M)
36 utility. \fBcfgadm\fR operates on attachment points. See \fBcfgadm\fR(1M).
39 An InfiniBand (\fBIB\fR) device is enumerated by the IB nexus driver,
40 \fBib\fR(7D), based on the services from the IB Device Manager (IBDM).
43 The IB nexus driver creates and initializes five types of child device nodes:
54 IB HCA service (HCA_SVC) devices
60 IB Virtual Physical Point of Attachment (VPPA) devices
76 See \fBib\fR(7D) for details on enumeration of IB Port, IB VPPA, and IB HCA_SVC
77 devices. For additional information on IBDM, see \fBibdm\fR(7D). See
78 \fBib\fR(4) for details on IB Pseudo devices.
81 For IB administration, two types of static attachment point are created for the
82 fabric administration as seen by the given host. There is one static attachment
83 point \fBib\fR and all IB devices (either an IOC, Port, VPPA, HCA_SVC, or a
84 Pseudo device) in the fabric are represented as dynamic attachment points based
85 off of it. There is another static attachment point for each Host Channel
86 Adapter (HCA) in the host based on its node Globally Unique Identifier (GUID)
90 Attachment points are named through ap_ids. There are two types of ap_ids:
91 logical and physical. The physical ap_id is based on the physical path name.
92 For the IB fabric it is \fB/devices/ib:fabric\fR. The logical ap_id is a
93 shorter, and has a more user friendly name.
96 The static ap_id for the IB fabric is \fBib\fR. The IB devices are dynamic
97 attachment points and have no physical ap_id. The logical ap_id of an IOC
98 contains its GUID, \fBib::\fR\fIIOC-GUID\fR. An example of an IOC ap_id is
99 \fBib::80020123456789a\fR. The logical ap_id of a Pseudo device, see ib(4) for
100 details, is of the format \fBib::\fR\fIdriver_name\fR\fB,\fR\fIunit-address\fR.
101 An example of a pseudo ap_id would be \fBib::sdp,0\fR where "\fBsdp\fR" is the
102 driver name and "\fB0\fR" is its \fIunit-address\fR property. The logical ap_id
103 of Port, VPPA and HCA_SVC device contains its Partition Key (\fIP_Key\fR),
104 \fIPort GUID\fR / \fINode GUID\fR and a communication service-name. The format
105 of ap_id is as below:
113 \fBib::\fR\fIPORT_GUID\fR\fB,\fR\fI0\fR\fB,\fR\fIservice-name\fR
123 \fBib::\fR\fIPORT_GUID\fR\fB,\fR\fIP_Key\fR\fB,\fR\fIservice-name\fR
133 \fBib::\fR\fIHCA_GUID\fR\fB,\fR0\fB,\fR\fIservicename\fR
138 The Partition Key (\fIP_Key\fR) is \fB0\fR for Port and HCA_SVC devices. The
139 \fIP_Key\fR helps determine the partition to which this port belongs for a VPPA
140 device node. A port might have more than one \fIP_Key\fR. An example of a VPPA
141 device logical ap_id point is \fBib::80245678,ffff,ipib\fR. The
142 \fIport-GUID\fR is \fB80245678\fR, the \fIP_Key\fR is \fB0xffff\fR, and the
143 service name is \fBipib\fR. The service-name information is obtained from the
144 file \fB/kernel/drv/ib.conf\fR which contains service-name strings. The HCA's
145 logical ap_id contains its node GUID value, hca:HCA-GUID. An example is
146 \fBhca:21346543210a987\fR.
149 A listing of the IB attachment points includes information on all IB devices
150 (IOC, VPPA, HCA_SVC, Pseudo, and Port devices seen by the IBDM and the IB nexus
151 driver) in the fabric even if they are not seen by the host and configured for
155 The following shows a listing of five IB devices (two IOC, one VPPA, one Port,
156 one HCA_SVC) and one HCA:
161 Ap_Id Type Receptacle Occupant Condition
162 hca:21346543210a987 IB-HCA connected configured ok
163 ib IB-FABRIC connected configured ok
164 ib::80020123456789a IB-IOC connected configured ok
165 ib::802abc9876543 IB-IOC connected unconfigured unknown
166 ib::80245678,ffff,ipib IB-VPPA connected configured ok
167 ib::12245678,0,nfs IB-PORT connected configured ok
168 ib::21346543,0,hnfs IB-HCA_SVC connected configured ok
169 ib::sdp,0 IB-PSEUDO connected configured ok
176 The ap_id \fBib::802abc9876543\fR shows an IOC device that is not yet
177 configured by the host for use or had been previously offlined by an explicit
181 cfgadm \fB-c\fR unconfigure
188 operation. The distinction was made by the information displayed under the
189 \fBCondition\fR column. The IB device with a zero P_Key and HCA GUID is a
190 HCA_SVC device. Refer to \fBcfgadm\fR(1M) for more information regarding
191 listing attachment points.
194 The receptacle state for attachment points have the following meanings:
202 For an IOC/VPPA/Port/Pseudo/HCA_SVC device, \fBconnected\fR implies that it has
203 been seen by the host. The device might not have been configured for use by
206 For a HCA attachment point, \fBconnected\fR implies that it has been configured
209 All IB ap_ids are always shown as connected.
214 The occupant state for attachment points have the following meanings:
222 The IB device, and the HCA ap_id, are configured and usable by Solaris.
232 The IB device at the ap_id was explicitly offlined using \fBcfgadm\fR \fB-c\fR
233 \fBunconfigure\fR, was not successfully configured. This could be because it
234 wasn not successfully configuref for use with Solaris (no driver, or a device
235 problem), or because it was never configured for use by the IB nexus driver.
237 The \fBunconfigured\fR operation is not supported for the HCA attachment point.
238 The IB static apid, \fBib\fR, is shown unconfigured if the system has no IB
244 The attachment point conditions are:
272 Normal state. Ready for use.
282 This state is only valid for IB device that have been probed by IBDM but not
283 yet configured for use by Solaris. It is also shown for devices that have been
284 explicitly offlined by a \fBcfgadm\fR \fB-c\fR \fBunconfigure\fR operation.
285 This condition does not apply to a HCA attachment point.
301 The following options are supported:
305 \fB\fB-c\fR \fIfunction\fR\fR
309 The IB hardware specific library supports two generic commands
310 (\fIfunction\fRs). These commands are not supported on the static attachment
311 points (that is, the HCA ap_ids and the IB static \fBib\fR ap_id).
313 The following generic commands are supported:
321 Configure the IB device to be used by Solaris.
331 Unconfigure the IB device. If successful, \fBcfgadm\fR reports the
332 \fBcondition\fR of this \fBap_id\fR as \fBunknown\fR.
350 \fB\fB-h\fR \fIap_id\fR\fR
354 Obtain IB specific help for an IB attachment point.
364 List the state and condition of IB attachment points. The \fB-l\fR option works
365 as described in \fBcfgadm\fR(1M).
367 When paired with the \fB-a\fR option, displays the dynamic attachment points as
368 well (IOC, VPPA, Port, Pseudo, and HCA_SVC devices).
370 When paired with \fB-v\fR option, displays verbose data about the ap_ids. For
371 an IOC, the \fBInfo\fR field in the
380 output displays the following information: VendorID, IOCDeviceID,
381 DeviceVersion, SubsystemVendorID, SubsystemID, Class, Subclass, Protocol,
382 ProtocolVersion and IDString from the IOCControllerProfile. If the ID string
383 isn't provided then nothing is displayed in its place. These fields are defined
384 in the InfiniBand Specification Volume 1 (http://www.infinibandta.org).
386 For a VPPA, Port, or HCA_SVC device the \fBInfo\fR field in the \fBcfgadm
387 -lav\fR display shows the service name information to which this device is
388 bound. If no such information exists, nothing is displayed.
390 For a Pseudo device \fBcfgadm -alv\fR displays the driver name and its
391 \fBunit-address\fR information. For a HCA the verbose listing displays the
392 VendorID, ProductID of the HCA, number of ports it has, and the PortGUID value
393 of its ports. See \fBEXAMPLES\fR.
399 \fB\fB-o\fR \fIhardware_option\fR\fR
403 This option is not currently defined.
409 \fB\fB-s\fR \fIlisting_option\fR\fR
413 Attachment points of class ib can be listed by using the select sub-option.
414 Refer to the \fBcfgadm\fR(1M) man page for more information.
420 \fB\fB-x\fR \fIhardware_function\fR\fR
424 Perform a hardware specific function. Note that the \fIname\fR can not be more
425 than 4 characters long.
427 The following hardware specific functions are supported:
432 \fB-ocomm\fR\fB=[port|vppa|hca_svc],service=\fR\fIname\fR\fR
436 This hardware specific function is supported on the static \fBIB\fR attachment
437 point. It can be used to add a new service to \fB/kernel/drv/ib.conf\fR file
438 and to update the \fBib\fR(7D) driver.
440 You must use the \fBservice=\fR\fIname\fR option to indicate the new service to
441 be added. You must use the option \fBcomm=[port|vppa|hca_svc]\fR option to add
442 the \fBname\fR service to either \fBport-svc-list\fR or to the
443 \fBhca-svc-list\fR in the \fB/kernel/drv/ib.conf\fR file. See \fBEXAMPLES\fR.
449 \fB\fBdelete_service\fR
450 \fB-ocomm\fR\fB=[port|vppa|hca_svc],service=\fR\fIname\fR\fR
454 This hardware specific function is supported on the static \fBIB\fR attachment
455 point only. It can be used to delete an existing service from the
456 \fB/kernel/drv/ib.conf\fR file and also from the \fBib\fR(7D) driver's data
457 base. You must use the \fBservice=name\fR option to indicate which service to
458 delete. You must use the \fBcomm=[port|vppa|hca_svc]\fR option to delete this
459 service from the \fBport-svc-list\fR, \fBvppa-svc-list\fR, or
460 \fBvppa-svc-list\fR of the \fB/kernel/drv/ib.conf\fR file. See \fBEXAMPLES\fR.
466 \fB\fBlist_clients\fR\fR
470 Supported on \fBHCA\fR attachment points. Displays all the kernel IB clients
471 using this \fBHCA\fR. It also displays the respective ap_ids of these kernel IB
472 clients and if they have opened an alternate \fBHCA\fR device. See
477 If a given kernel IB client does not have a valid ap_id then a \fB-\fR is
478 displayed in that column.
484 \fB\fBlist_services\fR\fR
488 This hardware specific function is supported on the static \fBIB\fR attachment
489 point only. It lists all the Port and VPPA services as read from the
490 \fB/kernel/drv/ib.conf\fR file. See \fBEXAMPLES\fR.
496 \fB\fBunconfig_clients\fR\fR
500 This hardware specific function is supported on the static HCA attachment point
501 only. It can be used to unconfigure all IB kernel clients of this given HCA.
502 Only IB kernel clients that do not have an alternate HCA are unconfigured. See
509 \fB\fBupdate_ioc_config\fR\fR
513 This hardware specific function is supported on static ib attachment point and
514 the IOC attachment points. For the \fBib\fR APID, this function updates
515 properties of all the IOC device nodes. For the IOC APID, this function updates
516 the properties of specified IOC device node. This command updates the
517 \fBport-list\fR, \fBport-entries\fR, \fBservice-id\fR, and \fBservice-name\fR
518 IOC node properties .
526 \fB\fBupdate_pkey_tbls\fR\fR
530 Supported on the static \fBib\fR attachment point. Updates the \fBPKEY\fR
531 information inside \fBIBTL\fR. \fBIBTL\fR re-reads the \fBP_Key\fR tables for
532 all the ports on each \fBHCA\fR present on the host.
541 \fBExample 1 \fRListing the State and Condition of IB Devices
544 The following command lists the state and condition of IB devices on the
545 system. It only shows the static attachment points.
551 hca:21346543210a987 IB-HCA connected configured ok
552 ib IB-FABRIC connected configured ok
559 The \fB-a\fR option lists all attachment points. The following example uses the
560 \fB-a\fR option and lists all attachment points:
566 hca:21346543210a987 IB-HCA connected configured ok
567 ib IB-FABRIC connected configured ok
568 ib::80020123456789a IB-IOC connected unconfigured ok
569 ib::80245678,ffff,ipib IB-VPPA connected configured ok
570 ib::21346543,0,hnfs IB-HCA_SVC connected configured ok
571 ib::12245678,0,nfs IB-PORT connected configured ok
572 ib::sdp,0 IB-PSEUDO connected configured ok
578 \fBExample 2 \fRListing the Verbose Status of a IB VPPA Device
581 The following command lists the verbose status of a IB VPPA device:
586 example# cfgadm -alv ib::80245678,ffff,ipib
587 Ap_Id Receptacle Occupant Condition Information
588 When Type Busy Phys_Id
589 ib::80245678,ffff,ipib connected configured ok ipib
590 unavailable IB-VPPA n /devices/ib:fabric::80245678,ffff,ipib
597 A verbose listing of an IOC shows additional information. The following command
598 shows a verbose listing:
603 example# cfgadm -alv ib::80020123456789a
604 Ap_Id Receptacle Occupant Condition Information
605 When Type Busy Phys_Id
606 ib::80020123456789a connected configured ok VID: 0xeaea
607 DEVID: 0xeaea VER: 0x5 SUBSYS_VID: 0x0 SUBSYS_ID: 0x0 CLASS: 0xffff
608 SUBCLASS: 0xff PROTO: 0xff PROTOVER: 0x1 ID_STRING: Sample Host Adapter
609 unavailable IB-IOC n /devices/ib:fabric::80020123456789a
616 A verbose listing of a Pseudo device shows:
621 example# cfgadm -alv ib::sdp,0
622 Ap_Id Receptacle Occupant Condition Information
623 When Type Busy Phys_Id
624 ib::sdp,0 connected configured ok Driver = "sd
625 p" Unit-address = "0"
626 unavailable IB-PSEUDO n /devices/ib:fabric::sdp,0
633 A verbose listing of a HCA shows:
638 example# cfgadm -alv hca:21346543210a987
639 Ap_Id Receptacle Occupant Condition Information
640 When Type Busy Phys_Id
641 hca:21346543210a987 connected configured ok VID: 0x15b3,
642 PID: 0x5a44, #ports: 0x2, port1 GUID: 0x80245678, port2 GUID: 0x80245679
643 unavailable IB-HCA n /devices/ib:21346543210a987
650 You can obtain more user-friendly output if you specify these following
651 \fBcfgadm\fR class and field selection options: \fB\fR\fB-s\fR
652 \fB"select=class(ib),cols=ap_id:info"\fR
656 The following command displays only IB ap_ids. The output only includes the
657 \fBap_id\fR and \fBInformation\fR fields.
663 # cfgadm -al -s "cols=ap_id:info" ib::80245678,ffff,ipib
665 ib::80245678,ffff,ipib ipib
671 \fBExample 3 \fRUnconfiguring an Existing IB IOC
674 The following command unconfigures the IB IOC attached to
675 \fBib::80020123456789a\fR, then displays the status of the \fBap_id\fR:
680 # cfgadm -c unconfigure ib::80020123456789a
681 Unconfigure the device: /devices/ib:fabric::80020123456789a
682 This operation will suspend activity on the IB device
695 IB device unconfigured successfully.
696 # cfgadm -al ib::80020123456789a
697 Ap_Id Type Receptacle Occupant Condition
698 ib::80020123456789 IB-IOC connected unconfigured unknown
706 The condition unknown implies that the device node doesn't exist anymore and
707 this IB device's existence is known only to the IB Device Manager.
710 \fBExample 4 \fRConfiguring an IB IOC
713 The following series of commands configures an IB device attached to
714 \fBib::80020123456789a\fR:
719 # cfgadm -yc configure ib::80020123456789a
720 # cfgadm -al ib::80020123456789a
721 Ap_Id Type Receptacle Occupant Condition
722 ib::80020123456789a IB-IOC connected configured ok
728 \fBExample 5 \fRListing All Kernel IB Clients of a HCA
731 The following command lists all kernel IB clients of an HCA attached to
732 \fBhca:21346543210a987\fR:
737 # cfgadm -x list_clients hca:21346543210a987
738 Attachment Point Clients Alternate HCA
739 ib::80020123456789a ioc1 Yes
740 ib::80245678,ffff,ipib ipib No
741 ib::21346543,0,hnfs hnfs No
749 \fBExample 6 \fRAdding a Port Service
752 The following command adds a new Port service called \fBsrp\fR:
757 # cfgadm -o comm=port,service=srp -x add_service ib
763 \fBExample 7 \fRDeleting a VPPA Service
766 The following command deletes the \fBibd\fR VPPA service \fBibd\fR:
771 # cfgadm -o comm=vppa,service=ipib -x delete_service ib
777 \fBExample 8 \fRListing Port, VPPA, HCA_SVC Services
780 The following command lists all Port, VPPA, and HCA_SVC services:
785 # cfgadm -x list_services ib
786 Port communication services:
789 VPPA communication services:
793 HCA_SVC communication services:
800 \fBExample 9 \fRReprobing IOC Devices
803 The following command reprobes all IOC device nodes.
808 # cfgadm -x update_ioc_config ib
809 This operation can update properties of IOC devices.
820 \fBExample 10 \fRUnconfiguring All Kernel Clients of a HCA
823 The following command unconfigures all kernel clients of a HCA
828 # cfgadm -x unconfig_clients hca:21346543
829 This operation will unconfigure clients of this HCA.
841 \fB\fB/usr/lib/cfgadm/ib.so.1\fR\fR
845 Hardware-specific library for generic InfiniBand device administration
851 \fBcfgadm\fR(1M), \fBconfig_admin\fR(3CFGADM), \fBlibcfgadm\fR(3LIB),
852 \fBib\fR(4), \fBattributes\fR(5), \fBib\fR(7D), \fBibdm\fR(7D), \fBibtl\fR(7D)
855 InfiniBand Specification Volume 1 (http://www.infinibandta.org)
859 Apart from the listing (\fBcfgadm\fR \fB-l\fR or \fBcfgadm\fR \fB-x\fR
860 \fBlist_clients\fR), only the superuser can execute any functions on an